1.2 The Rationale for Effective Measurement

We have long passed the point where craftsmanship can play a significant role in software systems. It would appear that a small group of people could design, build, operate, and maintain a software system of about 100 KLOC. Much beyond this critical value, the process rapidly decays. No one person can completely understand the entire system. The software becomes beset with performance, reliability, and maintenance problems. Just about every industry has encountered this same problem. A team of three or four people could hack together an early Wright or Curtiss aircraft. As the aircraft became more complex, however, the hacking was rapidly replaced with basic engineering discipline. Detailed designs were developed for the new, more complex aircraft. Tight manufacturing controls were exercised to make sure that all of the pieces integrated smoothly into a working aircraft. Experiments were conducted to study the best aerodynamic characteristics of the wings of the vehicle. What is astonishing about the software development enterprise is that it has taken so long to perceive the need for simple engineering discipline in the development of these systems.

1.2.1 Achieving Software Quality

Software quality does not happen by accident. It does not happen by investing tremendous resources in the problem. And, it most certainly does not happen by espousing meaningless metaphors or sloganeering. Software quality happens by design. ^[1] For our purposes, we insist that the first step in any effort at quality improvement begin with the institution of a measurement program. We can institute a host of new quality improvement processes; but until we have the ability to measure with some degree of accuracy certain relevant quality criteria, we cannot hope to improve our existing processes. It is quite possible the new "Quality Improvement Program" that we just bought from the "Software Quality Improvement Consulting Company" might well have had an adverse effect on our overall software development process. If we are not measuring, we will never know. Spending money to solve software quality problems is not the answer. Unfortunately, it is much easier to throw money at the problem than to do the necessary work to achieve engineering discipline in the process.

It is clear that all software development is accomplished by a software development staff. Unfortunately, it is possible to obtain an undergraduate degree in computer science at just about any major university in the world and never once be exposed to the fact that people write software. If there are faults in the code, people put them there. If the design is bad, people made it so. If the requirements are faulty, the requirements analysts created these faults. Processes that people follow permit them to make errors that, in turn, result in faults in the products. People are the real problem, yet we know so little about them. To achieve any real progress in the area of software quality, we are going to have to learn a little about how people tick.

1.2.1.1 Software Quality Objectives.

At the core of a set of software quality objectives should be a research effort focused on the factors that we can control in the software development process. We really do not know a great deal about this process. The field abounds with ad hoc studies, speculative conjectures of software quality, and models of capability and maturity. There is very little science that has been done in the area. We simply do not know why programmers put faults into code, but we can learn. Therefore, our foremost quality objective will be to do science. We must learn to measure people, processes, products, and environments. We must conduct the necessary science to understand how these domains interact to product good software. It will not be necessary to reinvent the wheel. There are excellent role models for the conduct of scientific inquiry in the fields of chemistry, biology, psychology, and the many engineering disciplines. Not much is known about software quality. The path to enlightenment is clear; many other disciplines have found the way.

The real tragedy in software development organizations today is that there is no learning in the system. If a good software system is accidentally designed and built, no one will know why this happened. It will not be possible to leverage this success on future projects unless very accurate measurement records are kept. If a lousy software system is designed and built, and this is a far more likely outcome, it will not be possible to learn from our mistakes unless very accurate measurement records are kept. If an engineering firm consistently designed and built bridges that systematically fell apart when they were placed in service, that firm would rapidly be sued out of existence. A good engineering firm can learn from both its successes and its (limited) failures.

It seems clear, then, that there are five simple software quality objectives. They are as follows:

Learn to measure accurately people, processes, products, and environments.
Learn to do the necessary science to reveal how these domains interact.
Institutionalize the process of learning from past mistakes.
Institutionalize the process of learning from past successes.
Institutionalize the measurement improvement process.

1.2.1.2 Control of the Software Development Environment.

Control is the key to quality software development. The more unconstrained degrees of freedom there are in the software development process, the more opportunity there will be for unexpected outcomes. From a statistical perspective, each unconstrained degree of freedom on a programmer is a source of uncontrolled variation in the development process. If a programmer is simply slavishly mapping a design to a particular programming metaphor, such as Java, then the effect of the programmer will be minimal. Every programmer should produce essentially the same code for the same design. If, on the other hand, the design is vague and ambiguous, there will be plenty of opportunity for a programmer to make interpretations of what was meant by the designer. Each programmer looking at the same design will produce different code that will function differently. The programmer is now a main player in the development process. A good programmer will produce clean, efficient, fault-free code. A bad programmer will produce code that is at once clumsy and fault ridden. This potential source of variation in code development is an anathema to the creation of software quality.

Individuality is the key to software craftsmanship. It has no real place in modern software development. Modern software systems must be engineered; they can no longer be crafted. They are much too complex for any one craftsman to understand. Furthermore, computer software is now playing a vital role in embedded software systems in items used in daily life. Its operation must be guaranteed and its safety assured. Individuality is not a desirable trait in engineered software systems. It will result in uncontrolled sources of variation in the software development process. The development of rigorous control systems for the software development process is a necessary but not sufficient condition for the attainment of quality software. A measurement-based feedback control system is a sufficient condition for the attainment of processes that will produce quality software.

1.2.2 Deming and the Software Development Process

W. Edwards Deming played a very influential role in the recovery of the Japanese manufacturing industry. His message was a very simple one, so simple, in fact, that it still eludes most of the manufacturing industry in America. It is totally unknown in the software development industry. We must learn not to focus on the products that we are making. Rather, we must learn to focus on the processes that are making these products.

As we evaluate these processes, we must learn to look beyond measures of central tendency to evaluate processes. We might learn that the average widget that we are manufacturing arrives at the end of a production line with five defects. That is an interesting fact but it has little value to us as information. Suppose, on the other hand, that we know that all widgets have five defects and that there is no variation in this number. We could probably go back down the assembly line and identify the precise source of these five defects and correct the problem. If there is little or no variation in the defect count, we have a problem that we can solve. If, on the other hand, some widgets have no defects while others have 20 or more defects, now we have a real problem. We are accidentally building good widgets and we do not know why. It will be very difficult to find and fix the manufacturing processes that are introducing the defects. The essence of Deming's message to the manufacturing community was: the real information is in the variance of the measurement.

One of the greatest obstacles to the development of successful engineering programs in software development environments is slogan-based engineering effort. There are numerous opportunists in the field of software. They persistently exploit the simplicity and ignorance of the software development community. Each of the sloganeers has an N-point methodology that will surely lead to the software development nirvana. If you do Practical Software Measurement (PSM), then surely science and engineering discipline must follow. If you join the teaming masses of software developers in the new science of requirements phrenology, function point analysis, then surely this must be engineering. An automobile engineer does not become an engineer because he joined the Society of Automotive Engineers (SAE). He also cannot expect to become an engineer just because he purchased a set of SAE wrenches. The engineering discipline requires many years of education and training. Engineers learned many generations ago that slogans do not build solid bridges. Solid engineering practice builds solid bridges.

If we are going to do software engineering, then we are going to have to listen to such pundits as Deming. ^[2] His message was a simple one, and a very unpopular one. We are going to have to work hard to achieve reasonable quality in manufacturing processes. It is so much easier to search for a miracle than it is to begin the serious work of a measurement-based engineering discipline. Ford Motor Co. did not build better cars because it posted signs throughout its facilities that said "Quality is Job 1." Ford built better cars because it either did so or lost its market share to the Japanese automobile makers who learned the quality engineering discipline from Deming. In the last analysis, it is probably easier and cheaper to build a quality automobile than it is to build one of lesser quality. The inertia of bad manufacturing practice was very great. It took an outside force (the Japanese) to execute the change. The inertia of bad software development practice is also very great. The industry awaits the stimulus of a country dedicated to the principle of quality software to provide the necessary impetus to produce quality software.

1.2.3 The Role of Statistics in Software Engineering

One of the most powerful tools available to a software engineer is that of statistics. Unfortunately, the typical undergraduate or graduate student has had little or no exposure to this discipline. It would appear that the world of computer science and software engineering is strictly a deterministic one. In this model, it is possible to describe in a deterministic manner exactly what code will be executing in an operating system at any moment, given any number of arbitrary inputs. That is probably true. It is theoretically possible to run the Universe backwards. That is, once we establish the precise location and velocity of each atom in this universe at any instant we could work backwards through time where the atom came from through all of its possible encounters with other atoms. The fundamental premise of the deterministic model is that we can know and observe Nature directly.

The fundamental premise of statistics is that we cannot know or understand Nature directly. We can conduct observations, perform experiments, and learn what we need to know at predetermined levels of confidence or certainty. The important notion is that it is not necessary to know exactly what Nature is about, only to have a good idea what she is up to. Sometimes, Nature is not eager to disclose some of her secrets to us. To this end we will formulate hypotheses about the phenomena we investigate. We can then test our hypotheses with carefully controlled experiments. Nature is not devious; she is not duplicitous. Our experimental process will reveal what we wish to know. The very worst thing that we could do is to make assumptions about Nature's secrets. The experimental method is the only way we can learn what we wish to know about Nature. The discipline of statistics will provide very useful tools to aid us in the discovery process. These statistical analysis tools we will find to be vital in the conversion of data to information.

^[1]Juran, J.M., Juran on Quality by Design: The New Steps for Planning Quality into Goods and Services, Free Press, New York, 1992.

^[2]Deming, WE., Out of the Crisis, MIT Press, Cambridge, MA, 2000.