8.4 Summary

The central theme of this chapter has been to focus on the problem of measuring and understanding evolving software systems. Clearly, as systems change over time, the numerical characteristics of their many measurable attributes will change as well. We have chosen to address two central problems in the measurement of systems as they evolve. First, there is the problem of establishing a measurement baseline. A single system in a chain of evolving systems will be chosen as a baseline system against which all others will be measured. Second, we have identified a mechanism wherein the precise manner in which they differ from each other can be measured. With this methodology, we can determine the changes that have occurred in evolving systems and the relative degree of this change.

Change can be measured in a variety of different ways. We can, with limited success, choose to try to use raw complexity metrics to measure change among systems. Alternatively, we see from this chapter that a far better method for measuring programs is to form orthogonal domain metrics for each system. These orthogonal metrics measure the distinct sources of variation in the suite of raw complexity metrics. This permits the direct comparison of each system build with its predecessor and successor builds. Finally, we have shown that it is possible to characterize each module of each system with a single measure of system complexity, the fault index (FI). This concept of system complexity from a software quality perspective permits a more global understanding of how systems evolve over time.

There are two distinct aspects of this measurement process. First, there are the static measures of fault liability. In this regard, we have developed a measure of the overall burden of a program module, the fault index, and also measures of new fault potential in the fault delta and the NFC. Second, we have developed a measure of test profiles to show the activity of test functionalities in individual program modules.

There is a distinct and a strong relationship between software faults and measurable software attributes. This is in itself not a new result or observation. The most interesting result of this current endeavor is that we also found a strong association between the fault introduction process over the evolutionary history of a software system and the degree of change taking place in each of the program modules. We also found that the direction of the change had an effect on the number of faults introduced. Some changes will have the potential of introducing very few faults, while others may have a serious impact on the number of latent faults. Different numbers of faults can be introduced, depending on whether code is being added to or removed from the system.

For the measurement process to be meaningful, the fault data must be very carefully collected and managed. In our continuing investigations of software evolution, the data was extracted ex post facto as a very labor-intensive effort. Because fault data cannot be collected with the same degree of automation as much of the data on software metrics being gathered by development organizations, material changes in the software development and software maintenance processes must be made to capture this fault data. Among other things, a well-defined fault standard and fault taxonomy must be developed and maintained as part of the software development process. Further, all designers and coders should be thoroughly trained in its use. A viable standard is one that can be used to classify any fault unambiguously. A viable fault recording process is one in which any one person will classify a fault exactly the same as any other person.

The whole notion of measuring the fault introduction process is its ultimate value as a measure of software process. The software engineering literature is replete with examples of how software process improvement can be achieved through the use of some new software development technique. What is almost absent from the same literature is a controlled study to validate the fact that the new process is meaningful. The techniques developed in this study can be implemented in a development organization to provide a consistent method of measuring fault content and structural evolution across multiple projects over time. The initial estimates of fault introduction rates can serve as a baseline against which future projects can be compared to determine whether progress is being made in reducing the fault introduction rate, and to identify those development techniques that seem to provide the greatest reduction.

A constant theme of this chapter has been on the act of measurement. Faults were measured. Software source code was measured. Measurement is the foundation of science and engineering. Today's software systems are simply too large to be hand-crafted as they were in the past. They must now be engineered. Every aspect of the evolving system must be carefully designed and tested based on sound engineering principles. These engineering principles have, as their foundation, sound measurement principles.