Defect Removal Effectiveness and Quality Planning

Phase defect removal effectiveness and related metrics associated with effectiveness analyses (such as defect removal and defect injection rates) are useful for quality planning and quality management. These measurements clearly indicate which phase of the development process we should focus on for improvement (e.g., unit testing in our example in Figure 6.4). Effectiveness analyses can be done for the entire project as well as for local areas, such as at the component level and specific departments in an organization, and the control chart technique can be used to enforce consistent improvement across the board (e.g., Figure 5.14 in Chapter 5). Longitudinal release-to-release monitoring of these metrics can give a good feel for the process capability of the development organization. In addition, experiences from previous releases provide the basis for phase-specific target setting and for quality planning.

6.3.1 Phase-Based Defect Removal Model

The phase-based defect removal model (DRM) summarizes the relationships among three metrics ”defect injection, defect removal, and effectiveness. The DRM takes a set of error-injection rates and a set of phase-effectiveness rates as input, then models the defect removal pattern step by step. It takes a simplified view of Figure 6.3 and works like this:

For example, the metrics derived from data in Figure 6.4 can be modeled step by step as shown in Table 6.3.

Now if we are planning for the quality of a new release, we can modify the values of the parameters based on the set of improvement actions that we are going to take. If we plan to improve the effectiveness of I2 and unit tests by 5%, how much can we expect to gain in the final product quality? What are the new targets for defect rates for each phase (before the development team exits the phase)? If we invest in a defect prevention process and in an intensive program of technical education and plan to reduce the error injection rate by 10%, how much could we gain? Approximate answers to questions like these could be obtained through the DRM, given that the DRM is developed from the organization's experience with similar development processes.

Be aware that the DRM is a quality management tool, not a device for software reliability estimation. Unlike the other parametric models that we will discuss in later chapters, the DRM cannot reliably estimate the product quality level. It cannot do so because the error injection rates may vary from case to case even for the same development team. The rationale behind this model is that if one can ensure that the defect removal pattern by phase is similar to one's experience, one can reasonably expect that the quality of the current project will be similar.

Table 6.3. Example of Phase-Based Def ect Removal Model

Phase	(A) Defect Escaped from Previous Phase (per KLOC)	(B) Defect Injection (per KLOC)	Subtotal (A+B)	Removal Effectiveness	Defect Removal (per KLOC)	Defects at Exit of Phase (per KLOC)
Requirements	”	1.2	1.2	”	”	1.2
High-level design	1.2	8.6	9.82	x 74%	= 7.3	2.5
Low-level design	2.5	9.4	11.9	x 61%	= 7.3	4.6
Code	4.6	15.4	20.0	x 55%	= 11.0	9.0
Unit test	9.0	”	9.0	x 36%	= 3.2	5.8
Component test	5.8	”	5.8	x 67%	= 3.9	1.9
System test	1.9	”	1.9	x 58%	= 1.1	0.8
Field	0.8

6.3.2 Some Characteristics of a Special Case Two-Phase Model

Remus and Zilles (1979) elaborate the mathematical relationships of defect removal effectiveness, the number of defects found during the front end of the development process (before the code is integrated), the number found during testing, and the number remaining when the product is ready to ship to customers. They derived some interesting characteristics of the defect removal model in a special case:

1. There are only two phases of defect removal.
2. The defect removal effectiveness for the two phases is the same.

The percentage of bad fixes is one of the parameters in the Remus and Zilles model; the derivation involves more than twenty formulas. Here we take a simplified approach without taking bad fixes into account. Interestingly, despite taking a different approach, we arrive at the same conclusion as Remus and Zilles did.

Assume there are two broad phases of defect removal activities:

1. Those activities handled directly by the development team (design reviews, code inspections, unit test) for large software projects take place before the code is integrated into the system library.
2. The formal machine tests after code integration.

Further assume that the defect removal effectiveness of the two broad phases is the same. Define:

MP	=Major problems found during reviews/inspections and unit testing (from phase 1); these are the problems that if not fixed, will result in testing defects or defects in the field.
PTR	=Problem tracking report after code integration: errors found during formal machine tests.
m	=MP/PTR, m >1 ( Note: The higher the value of m , the more effective the front end.)
Q	= Number of defects in the released software ”defects found in the field (customer usage).
TD	=Total defects for the life of the software = MP + PTR + Q .

By definition of effectiveness:

Equation 6.1

 

Equation 6.2

 

By the assumption that the two phases have the same effectiveness:

 

Thus,

Equation 6.3

 

Then,

 

 

Therefore,

Equation 6.4

 

By the same token, it can be shown that:

Equation 6.5

 

Furthermore, from the definition of m :

 

Therefore,

Equation 6.6

 

Equations (6.4) through (6.6) can be useful for quality planning. The equations can be applied to absolute numbers as well as to normalized rates (e.g., defects per KLOC). Given the number of MP and m , or PTR and m , one can estimate the number of defects that remained in the product by Equations (6.4) and (6.5). Also, assuming we use the lifetime defect rate (TD) of a predecessor product to approximate the TD of the product being developed, given a target product quality level to shoot for, Equation (6.6) can determine the value of m that we need to achieve in order to reach the target. Choosing a specific value of m determines how much focus a project should have on front-end defect removal. Once the m target is set, the team can determine the defect removal techniques to use (e.g., formal inspection, function verification by owner, team verifications, rigorous unit testing, etc.). For example, if we use the data from the example of Figure 6.4 (TD = 34.6 defects/KLOC, Q = 0.81 defects/KLOC for life of customer use), then the value of m should be:

 

This means that if the effectiveness is the same for the two phases, then the number of defects to be removed by the first phase must be at least 6.5 times of the number to be removed by testing in order to achieve the quality target. Note that the equations described in this section are valid only under the assumptions stated. They cannot be generalized. Although Equations (6.4) and (6.5) can be used to estimate product quality, this special case DRM is still not a projection model. The equal effectiveness assumption cannot be verified until the product defect rate Q is known or estimated via an independent method. If this assumption is violated, the results will not be valid.