Chapter 13: Implementing a Software Measurement Plan

13.1 The Software Measurement Process

Measurement is something that we must learn how to do. The very worst thing that we can do in the measurement process is to presume that all of the tools necessary to perform the required measurement are handily available, that all we need to do is just find them. We would like very much for the measurement effort to be someone else's problem. We would like to buy a miracle. That is, we would like to find the ideal measurement toolset that would automatically install on our software development hardware and provide instant feedback on our software process. Unfortunately, there are no miracles. We have watched software developers acquire many different miraculous metric tools in the past. The end game has always been the same. The miracle just did not happen. If we are going to learn how to do measurement-based software engineering, we are going to have to roll up our sleeves and do some serious investment of our own labor. You simply cannot buy good engineering practice. You must work to make it happen.

Measurement tools simply produce numbers. These numbers are simply data. For the numbers to be made meaningful, they must first be converted to information. This information can, in turn, be used to monitor and change the software development information process. There are really three parts, then, to a measurement system. First, we must have tools that generate the measurement data. Hopefully, the data produced will be meaningful and valid. Second, we must have in place a strategy to manage the data. The data must be sent to a data repository where it can be structured in relation to other data for easy access in the future. Finally, we must have tools in place to convert the data to information.

There is also the misconception that simple is best. If we could just find the right snake oil salesman, we could buy an answer to our measurement problems. Measurement is not just a question of finding the right metric. We have clearly seen that measuring the static attributes of a software system is a multivariate problem. Computer programs are extremely complex entities. The people who create them are even more complex. Any attempt to oversimplify the measurement process will certainly contribute to our misunderstanding of the software development process. One of the most egregious misuses of measurement data can be seen in Exhibit 1. Here we see that one metric, cyclomatic complexity, is the unique determinant of program risk (whatever that might be). When this metric is applied to a program module, we can clearly see that a module with cyclomatic complexity greater than 20 will create high risk. The obvious solution is to split the module into two or more modules, each of which will have less complexity than the first. What this does is to shift the complexity from a domain that is being measured, ostensibly control flow complexity, to one that is not being measured, coupling complexity. It might well be that this decision would result in a program whose net complexity across all attribute domains will increase. This is a very good example of how incomplete information can result in exactly the wrong decision.

Exhibit 1: Misuse of Measurement Data

Cyclomatic Complexity	Risk Evaluation
1–10	A simple program, without much risk
11–20	More complex, moderate risk
21–50	Complex, high risk program
>50	Untestable program (very high risk)
From the Software Enginering Institute, Carnegie Mellon University

The most important concept in the establishment of an effective software measurement process is simplicity. Start with a simple process and get it right. The focus should be on the process itself, and not on the tools. It is possible to have the best tools that money can buy and still not have a viable software measurement system. If we have designed an effective software measurement process, it will be possible to insert tool enhancements at will. The process should not depend on the tools. We would expect to build better tools as our understanding of the measurement process matures. If the process is a good one, the tools can be plugged into or unplugged from it without disruption.

The measurement process should be continuous. With the aid of a software measurement database, the measurement data should be timely. That is, the information in the database should reflect exactly the system as it is right now. Measurement is an integral part of the software development process. Just as is the case with other engineering disciplines, all design decisions and development decisions should be based on measurement outcomes. Many times during the initial stages, we will not have the information we need to make the decisions we will need to make. In this case, we will have to design experiments to understand the nature of the problem we are trying to solve. This will naturally lead to the integration into the measurement database of new measurement data as an aid to the development process.

Inherent in the measurement process should be a mechanism for measurement process improvement. We propose that the measurement system should begin with a very simple set of measurement tools and processes. We will then institutionalize a process of measurement process improvement. In this chapter, we will sketch out such a simple system and show some fundamental elements of this system. What we will discover in this design process is that the measurement structure should be woven out of whole cloth. That is, the source code measurement system should integrate with the source code control system, which should also be used to manage the evolving requirements specification. The software testing process should be integrated into the measurement process at a very early stage. Ultimately, all aspects of the software development of a new software system — from cradle to retirement — should operate under the scrutiny of the measurement system. That is what engineering is all about.

The most difficult obstacle to overcome in the transition from the world of software craftsmanship to the engineering discipline will be from the software management staff. They know how to manage software craftsmen. They do not know how to do engineering, much less manage an engineering process. Unless there is a high degree of management buy-in, any attempt to initiate a software measurement is doomed to failure. This may mean that substantial training in the use of measurement data must occur within the management staff. The typical manager has made decisions by seat-of-the-pants methodology for his entire management lifetime. Managers must be trained to use the information that the measurement system yields. That is part of the process of beginning a new measurement-based development system.

Finally, for there to be a successful transition to a software engineering system, the notion of engineering discipline is paramount. All software measurement and reporting systems must be rigorously followed and there must be processes in place to enforce this discipline. Nothing, for example, can undermine a fault tracking system any faster than the pocket program trouble report (PTR). In this case, a developer may well discover faults in a program module that he or she has written. In that the developers are aware that faults are now being tracked to individual developers, there will be an attempt to hide or disguise faults on the part of the developers. One way to do this is the pocket PTR. In this case, a developer discovers a fault in code that he or she has written, and rather than report this problem, the developer will simply note the problem on a piece of paper that is filed in his or her pocket. The developer will then hold the informal PTR until a problem is found in the same module that was introduced by another developer. When the developer applies a fix to the new problem, he or she will simply introduce the fix for the pocket PTR, thus disguising the original fault.

The use of the pocket PTR by developers indicates that there are several things wrong with the development process. First, the developer thinks that he or she owns the code. In fact, the code base belongs to the company. It is in the company's interest that the code is fixed. A system that punishes the developer for faults will only encourage the faults to be hidden. This, in turn, results in real distortions in the fault measurement system. Developers should be rewarded for finding faults — not punished.

Second, the fault got into the code because the software development process failed, not because the developer introduced it. The process that led to the introduction of the fault in the first place should be the focus of interest in the etiology of the fault. If, for example, a developer introduced a fault that resulted in a divide exception, then it is possible that other developers have done the same thing. We might then elect to introduce a check for division operations in all new code as an item on an inspection protocol for future software review processes. Even more important, if a developer introduced a division operation that resulted in a divide exception, it is likely that there are other instances of this problem in the code as well. A code inspection process should be initiated that will cause the code base to be reviewed for other similar problems.

Third, there was no review process to ensure change integrity. Code should only be changed in response to a single PTR or a single change request. Each code update should address exactly one PTR or one change request. A review committee should audit all proposed code changes before they are committed to the source code control system.

We can use metric tools to measure software code with a high degree of precision. Other measurement activities, such as the fault tracking system, depend heavily on human cooperation. It will do little good to have very precise code measurements and very imprecise and inaccurate fault data. Thus, the majority of the emphasis in the establishment of an effective software measurement system should be on those measurement activities that involve humans in the process. These human beings are, in fact, measurement tools. They must be manufactured (trained) with the same precision as the automated measurement tools, such as source code metric tools.