13.3 Measurement Process Improvement

Quite clearly, we must develop a software measurement process. We must build a measurement system capable of measuring software products, process, people, and environments. That is a given. Once this measurement process is in place, we must take the next critical step. We must institutionalize the measurement process improvement process. The first rulers that we build will be crude instruments. The first software metric tool will be very simple. It will probably only measure a subset of the potential metrics that we think might be important. The first telescope ever made was a very primitive instrument by today's standards. However simple it might have been, it revolutionized the world of science. It opened the world of science to the notion that we are members of a vast universe. From that first very primitive instrument, incremental gains were made in the resolving power of the telescopes. With each improvement in resolution, our understanding of our physical universe also grew. We must be very careful just how complex we make our measurement tool. It can easily provide much more information than we are capable of digesting in our growth of understanding in software measurement. Imagine, if you will, the improbabilities of sharing the latest photographs from the Hubble space telescope with Galileo. He simply would not have been able to grasp the notion of the photograph itself, much less the image that it represented. Science evolves in very slow evolutionary steps. Each time a new telescope technology was introduced into the astronomical world, it took substantial time to assimilate the basic concepts of the new universe revealed by these new, more powerful instruments. It was suggested earlier that the initial metric tool should probably only measure a handful of size metrics. If this measurement is done accurately and precisely, the data that this metric tool will yield, in conjunction with the other data that we are also accumulating, will result in a quantum leap forward in the software development process.

13.3.1 Tools Refinement

In the case of astronomy, each new advance in telescope resolving power permitted astronomers to look farther back in time. This, in turn, allowed them to make new discoveries and expand their knowledge of the universe. Each incremental improvement of the telescope occurred at the right time for the knowledge base of that time. Each new technological advance in telescope construction represented an evolutionary enhancement in the observational technology. The images from the Hubble telescope would probably have been incomprehensible to astronomers as recently as 75 years ago. The same path must be followed in the evolution of software measurement tools. There is real value, for example, in a source code measurement tool that measures a few simple source code attributes such as those presented in Chapter 5.

After the initial tool has been inserted in the measurement process, we will seek to learn that which we do not yet know. The modeling techniques discussed in Chapter 7 will serve as a guide for identifying the directions that this new research should take. We will always know exactly what sources of variation we are measuring, and we will know how well we are measuring these attributes. Once we have clearly established what we are measuring, it will become increasingly clear what we are not measuring. It is clear, for example, that there are no metrics in Chapter 5 that deal directly with the object-oriented (O-O) design metaphor. This is a very complicated environment to measure. Great caution and considerable research will have to be performed before this metaphor can be well understood, let alone measured effectively. One of the major problems confronting us with this particular metaphor is that we must parse our measurement activity carefully into the O-O metaphor constructed for the user precompilation and the part of that metaphor that persists at runtime. From the standpoint of testing and dynamic measurement, a great deal of the O-O metaphor vanishes during compilation. Therefore, measurement of these attributes would be pointless from the standpoint of testing.

In essence, our measurement tools will be refined by an orderly sequence of experiments. Each experiment will be designed to identify new sources of variation in program attributes that contribute to our understanding of one or more criterion attributes. One of the major thrusts of our own research has been in the area of predicting the fault liability in source code constructs. An improvement in our measurement tool, therefore, would be one that would significantly increase the amount of explained variation in software faults. Exactly the same process would be followed for different criterion measures such as maintainability or availability. A better tool is one that explains more variation in one or more criterion measures. It is just that simple. Our initial objective is to build a simple, good tool to prime the measurement process. We will then institute experimental processes that will lead to subsequent improvements in this tool.

13.3.2 Measurement Process Enhancement

Part of the measurement process enhancement will occur directly from the simple and consistent improvement of the measurement tools around which we have initially built the measurement system. Clearly, our initial system has incorporated only a very small set of metrics with a small number of tools. The tools produced data in that they were part of a measurement process that drove the data capture. There is plenty of room for enhancement from our initial system in how these data will be captured, analyzed, and reported.

Even the most primitive kind of measurement system, as sketched above, has a tremendous amount of information in it. It would be pointless to take the next step in the enhancement of the measurement process if we did not take the requisite time to educate the development, test, requirements analysis, and management staff in the effective use of the information now at their disposal. Therefore, a vital step in the enhancement of the measurement system is the establishment of a training program in the correct uses of the information now available from the measurement and analysis system.

The next logical step in the measurement process enhancement is to expand on the number of attribute domains being measured for each of the people, process, product, and environment measurement domains. Initially, for example, we shied away from direct measurement of the people in the software process. This was principally because of the potential land mines in the misuse that this information might bring to bear. It is far too easy to draw exactly the wrong conclusions from data. Only after there has been sufficient training of the personnel who will have access to the measurement information should these potentially volatile sources be measured in the first place.

The most important consideration in the measurement enhancement process is that more is not better. Our objective is not to mine new sources of data just because they exist; it is far more important to have quality information than quantity. Also, no data should be added to the measurement system until there is a complete understanding of how these new data will be converted to useful information.