Quality tools

5.6 Quality tools

The representation of measures and metrics may take many forms. Even their collection must be considered. The following tools are most often used for measure collection and measure and metric use:

• Tally sheet;

• Scatter diagram;

• Graph;

• Histogram;

• Pareto diagram;

• Flowchart;

• Cause and effect diagram;

• Statistical control chart.

None of these tools is new. The hardware quality practitioner has had such tools available for many years. Some, such as the tally sheet and the scatter diagram, have been regularly used in software quality activities. Others, like the cause and effect diagram and statistical control charts, are relatively new to software quality applications.

5.6.1 Tally sheet

The tally sheet is the simplest form of data collection. Each occurrence of some event or situation is tallied as it happens or is detected. The example in Figure 5.4 depicts a collection of data for defects detected in several modules of software being reviewed. Note that this is merely a collection of detected-defects data. Taken by itself, the collection gives little or no information about the modules beyond pure defect counts. It is usually beneficial to chart or graph the numbers for comparison.

Figure 5.4: Tally sheet.

5.6.2 Scatter diagram

Figure 5.5 presents the data from the tally sheet in a more mathematical form. The numbers are the same, but some people find this representation more understandable. The scatter diagram gives a visual comparison of the numbers from the tally sheet. Sometimes, it is useful to plot the trend line or least-squares curve that summarizes the scattered points. The dashed line represents an estimate of the data trend.

Figure 5.5: Scatter diagram

5.6.3 Graph

In its simplest form, a graph is just a scatter diagram with the points connected. Continuing the defect count example, Figure 5.6 is a graphical representation of the numbers on the tally sheet. Graphs are often preferred for observing the progress of a process or activity with respect to time. Continuing the example, the modules 1-12 may have been completed in the time sequence shown. More information, such as calendar dates or complexity, can be shown in a graph. We might discover that module 7 was highly complex, while module 12 was completed near year's end.

Figure 5.6: Graph.

5.6.4 Histogram

A histogram is similar to the graph, but instead of connecting the points with a line, they are shown as vertical [or horizontal] bars whose lengths represent the numbers. Figure 5.7 is a histogram of the tally sheet data. Histograms are often precursors to Pareto charts and are sometimes expanded to have the width represent an additional condition, such as size or effort.

Figure 5.7: Histogram.

5.6.5 Pareto diagram

In the nineteenth century, economist Vilfredo Pareto determined that approximately 80% of his country's wealth was controlled by about 20% of the population. Thus was born the 80/20 rule. Although originally directed toward economics, the 80/20 rule has come to be used in many applications, including software quality management. The 80/20 rule in software quality management suggests that we pay attention to the products that account for 80% or so of the defects. Admittedly not mathematically precise, it serves as a good guide to the application of quality effort.

The Pareto diagram is the histogram arranged in (usually) descending order of bar height. Figure 5.8 is the Pareto representation of the tally sheet numbers. Also indicated is the approximate 80% point. Software quality practitioners could use a Pareto diagram to prioritize their examination of the causes for the defect numbers associated with each module.

Figure 5.8: Pareto diagram.

5.6.6 Flowchart

Flowcharts are diagrams that permit users to describe processes. They are not used to represent data, but rather the way in which things are done. Manufacturing, sales, banking, military, software, in fact nearly all processes have been described with flowcharts. The software quality assurance practitioner will use the flowchart to depict the various processes used in software development, maintenance, operation, and improvement. As the metrics begin to suggest flaws in one or another process, the flowchart can help isolate the part of the process that is flawed. Figure 5.9 depicts the format for a typical flowchart.

Figure 5.9: Flowchart.

Continuing our example, the development process for each module might be flowcharted to look for differences that might account for the variations in the defect rates. Alternatively, the defect detection processes might be flowcharted in a search for their variations and the reasons for their differing results.

5.6.7 Cause-and-effect diagram

Another diagram used to locate process flaws is the cause and effect diagram. Defect detection and correction is responsible for eliminating the defect. Cause and effect diagrams are used to determine the actual cause of a defect. In this way, not only is the defect itself eliminated, but the situation that permitted it to be introduced is also eliminated, so it will not occur again.

The cause and effect diagram, an example of which is shown in Figure 5.10, is also called the Ishikawa diagram after Professor Kaoru Ishi-kawa, who introduced it in Japan, and the root cause diagram Note that neither the flowchart nor the cause and effect diagram is used to depict data. Rather, they help describe processes and analyze defect causes.

Figure 5.10: Cause-and-effect diagram.

In our example, based on the high defect rate for module 7, the software quality practitioner might use the cause and effect diagram to seek out the specific causes for the high defect rate, rather than just guessing.

Coupled with the knowledge of the module's complexity, the time of year of its creation, the type(s) of defect detection methods applied, flowcharts of the development and defect detection processes, and, perhaps, its cost and schedule variations from expectations, the cause and effect diagram can assist the quality practitioner in the analysis. One arm might represent all the potential effects of calendar dates, another the effects of schedule or budget changes, still another the effects of requirements changes, and so on. The user of the cause and effect diagram tries to examine each potential cause until the actual root cause of the situation is discovered. The diagram serves to document each potential cause examined and its relationship to other potential causes.

5.6.8 Process control charts

In 1931, Walter Shewhart applied the laws of statistics to production processes and discovered that the process behavior can be plotted. He then showed how the process can be controlled and how that control can be monitored.

5.6.8.1 Run charts

Run charts depict the statistical behavior of a process. Shewhart intended that these be used to show when a process was stable or in control. It is not based on the intended or target behavior of the process but its actual behavior. Figure 5.11 shows a basic process control chart. The center line is the mean level of the described characteristic (perhaps the error rate for a given software system). The upper line is the upper control limit (UCL) and indicates the presence of special causes of defects. The lower line is the lower control limit (LCL) and also reflects special causes, this time because the product is better than it has to be. Shewhart set the UCL and LCL as functions of the mean and standard deviation for the population being plotted.

Figure 5.11: Basic run chart.

The idea of evaluating process behavior using a run chart is depicted in Figure 5.12. A process that has an error rate falling consistently between the control limits (as in the first section of Figure 5.12) is said to be in control. An occasional point outside the limits identifies a special case, a target for cause and effect analysis. In general, however, the process is considered sound and repeatable. The second section of Figure 5.12 shows the case in which the points fall randomly around the mean. This process is said to be out of control. Changes to the process are not likely to be identified with changes in the results. The third section of Figure 5.12 shows a process in control but drifting. In the manufacturing world, we might conclude that a tool was getting worn. In software we might suspect that a defect detection technique was improving (the drift implies that we are finding more defects) or that the development process was degenerating (and letting more defects into the products).

Figure 5.12: Process behavior.

Continuous process improvement might be depicted as in Figure 5.13. Based on the Japanese concept of kaisen (continuous improvement through small changes), the process is continually improved. As a result, the control limits, still functions of the mean and standard deviation, tend to move closer to the mean.

Figure 5.13: Kaizen concept.

5.6.8.2 Acceptance control charts

Acceptance control charts are not based on statistical determination of the mean and control limits. These charts use the desired value, called the target, as the centerline. The upper and lower control limits are chosen based on permissible variation, not statistical variation.

In software terms, when depicting defects, we would like all three lines to lie on the zero-defects line. That is also the ultimate long-term goal of the SQS. In the meantime, and in the real world, there are other pressures that slow our attainment of the zero-defect goal. Just like in the hardware world of Shewhart, costs and risks will define the realistic levels for the target and control levels. In economic terms, we might say that the UCL is the level at which the customer will no longer accept the product, and the LCL the level at which the costs of finding further defects exceed the costs incurred if the defect occurs in the use of the software. Stated still another way, the UCL defines "how bad I dare to make it" and the LCL defines "how good I can afford to make it."

Acceptance control charts are not as statistically valid as run charts, but they do not require the large sample and population sizes on which run charts are usually based. Acceptance control charts are more easily adapted to the uses of the software quality assurance practitioner, however. Figure 5.14 combines the acceptance control chart, the kaizen concept, the desire for zero defects, and economic reality into a typical software process control chart. In this chart, the LCL is set as close to zero as is economically feasible. The centerline, the target defect rate, starts at the current experienced value and slopes downward at the goal rate of the SQS. At the same time, the UCL is more sharply sloped to motivate process improvement.

Figure 5.14: Toward zero defects.