11.2 Static and Dynamic Measurement

It is clear, by now, that static software measurements will give a clear indication as to where the potential faults in a system might be. It is one thing to know where the potential faults in code might be. It is quite another thing to see whether the program actually executes in the region where the faults are. Static software measurement is like one-hand clapping. Dynamic software measurement, then, is the other hand clapping. Static measurement is useful in that it shows where the potential problems are in the source code. Dynamic code measurement shows what code got executed and how intensively. Both static and dynamic measurements come together in the measurement of the software testing enterprise. It is vital to be able to measure program behavior at runtime to determine our exposure to the potential faults in the code. If we are to be able to test a software system, we must know both the location of the faults and whether the software test execution profiles match where the faults are likely to be. The testing process should be monitored and controlled by simple engineering standards. All test activity should be measured as it is occurring.

It seems entirely reasonable that if we know where the faults in the code are likely to be, then our test activity should exploit this information. Unfortunately, the code base is a moving target. Most software systems will evolve very rapidly from the first build to the last. This means that there are essentially two different types of faults that we must deal with during the testing process: (1) faults that were introduced in the initial code as it was first built, and (2) faults that have been introduced in subsequent builds.

It will help clarify our thinking about the testing process if we can establish some reasonable test objectives. Here we will draw from the centuries of test experience from hardware and electronic engineering enterprises. We do not care that our software has faults in it. No matter how careful we are, there will always be faults in our code. As we attempt to remove old faults, we run the risk of adding new faults. It is entirely possible to remove an innocuous fault only to introduce a severe one in the repair process. Removing all of the software faults, then, is neither a viable nor practical goal. We would just like to remove those faults that will impair the way that the system is used when it is deployed.

The first step, then, in the software test process should center on who will be using the system and how will they be using it. When we know how the system will be used when it is deployed, we can then concentrate our efforts on certifying that it will not break when it is used in a fashion specified by a suite of operational profiles.

There are significant flaws in essentially all hardware systems regardless of the level of engineering expertise brought to bear on the design and manufacture of the product. A concrete building column may, for example, have substantial voids in it where the concrete failed to fill the form correctly as it was being poured. If we have done our engineering correctly, the building will not collapse because of this void. If the void occurs in a structurally significant section of the column where either compression or tension forces are concentrated, it may cause the column to fail. We simply cannot build a building without voids in the concrete structures. We can test to be sure that these voids are not structurally significant. The cost of a perfectly cast concrete structure is prohibitive. The cost of a fault-free program is equally prohibitive. We must learn to focus our test energies on those activities that are vital to our customer and learn to ignore those that do not meet this criterion. The worst thing that we could do is to remove a fault that will not affect the functionality of our code and insert a fault in the replacement code that will impact the functionality of the system.