21.5 Estimating Defect Production and Removal
21.5 Estimating Defect Production and Removal
Defect production on a software project is a function of effort and project size, so it's possible to estimate defect production. Knowing how many defects are likely to be produced is useful information when you are planning how much effort you'll need to remove the defects.
Capers Jones provides one way of looking at defect production based on a program's size in function points (Jones 2000). As Table 21-10 indicates, Jones's data suggests that a typical project produces a total of about 5 defects per function point. This works out to about 50 defects per 1,000 lines of code (depending on the programming language used).
| Activity | Average Defects Created |
|---|---|
| Requirements | 1 defect per function point |
| Architecture | 1.25 defects per function point |
| Construction | 1.75 defects per function point |
| Documentation | 0.60 defects per function point |
| Bad fixes | 0.40 defects per function point |
| TOTAL | 5.0 defects per function point |
One factor that contributes to software's diseconomy of scale is that larger projects tend to produce more defects per line of code, which requires more defect-correction effort, which in turn drives up project costs. Table 21-11 presents a breakdown of defects based on project size.
| Project Size (in Lines of Code) | Typical Error Density |
|---|---|
| Smaller than 2K | 0–25 errors per KLOC |
| 2K–16K | 0–40 errors per KLOC |
| 16K–64K | 0.5–50 errors per KLOC |
| 64K–512K | 2–70 errors per KLOC |
| 512K or more | 4–100 errors per KLOC |
| Source: "Program Quality and Programmer Productivity" (Jones 1977), Estimating Software Costs (Jones 1998). | |
The industry-average ranges of defect production vary by more than a factor of 10. Historical data about defect rates on your past projects will support more accurate estimates of defect production.
| Tip #100 | Use industry-average data or your historical data to estimate the number of defects your project will produce. |
Estimating Defect Removal
Defect production is only part of the planning equation. Defect removal is the other part. The software industry has accumulated a fair amount of data about the defect-removal efficiency of the most common defect-removal techniques. Table 21-12 lists the defect-removal rates for inspections, reviews, unit testing, system testing, and other techniques.
| Removal Step | Lowest Rate | Modal Rate | Highest Rate |
|---|---|---|---|
| Informal design reviews | 25% | 35% | 40% |
| Formal design inspections | 45% | 55% | 65% |
| Informal code reviews | 20% | 25% | 35% |
| Formal code inspections | 45% | 60% | 70% |
| Modeling or prototyping | 35% | 65% | 80% |
| Personal desk checking of code | 20% | 40% | 60% |
| Unit test | 15% | 30% | 50% |
| New function (component) test | 20% | 30% | 35% |
| Integration test | 25% | 35% | 40% |
| Regression test | 15% | 25% | 30% |
| System test | 25% | 40% | 55% |
| Low-volume beta test (<10 sites) | 25% | 35% | 40% |
| High-volume beta test (>1,000 sites) | 60% | 75% | 85% |
| Source: Adapted from Programming Productivity (Jones 1986a), "Software Defect-Removal Efficiency" (Jones 1996), and "What We Have Learned About Fighting Defects" (Shull et al 2002). | |||
The range from lowest rate to highest rate is significant, and, as usual, historical data from your own organization will support more accurate estimates.
An Example of Estimating Defect-Removal Efficiency
Combining the information from the defect-production and the defect-removal tables allows you to estimate the number of defects that will remain in your software at release time (and of course helps you assess the steps to take to remove more or less of the defects, depending on your quality goals).
Suppose you have a 1,000-function-point system. Using Jones's data from Table 21-10, you would estimate that the project would generate a total of 5,000 defects. Table 21-13 shows how those defects would be removed using a typical defect-removal strategy consisting of personal desk checking of code, unit testing, integration testing, system testing, and low-volume beta testing.
| Activity | Effect on Defects | Total Defects Produced So Far | Defects Still Remaining |
|---|---|---|---|
| Requirements | +1,000 defects | 1,000 | 1,000 |
| Architecture | +1,250 defects | 2,250 | 2,250 |
| Construction | +1,750 defects | 4,000 | 4,000 |
| Personal desk checking of code | –40% | 4,000 | 2,400 |
| Documentation | +600 defects | 4,600 | 3,000 |
| Unit testing | –30% | 4,600 | 2,100 |
| Integration testing | –35% | 4,600 | 1,365 |
| System test | –40% | 4,600 | 819 |
| Bad fixes | +400 defects | 5,000 | 1,219 |
| Low-volume beta testing | –35% | 5,000 | 792 |
| Defects remaining at release | –84% | 5,000 | 792 (16%) |
This typical approach to defect removal is expected to remove only about 84% of defects from the software prior to its release, which is approximately the software industry average (Jones 2000). The specific numbers you obtain using this technique are, as usual, approximate.
Table 21-14 shows how a best-in-class organization might plan to remove defects. This example assumes the project team will produce the same total number of 5,000 defects. But defect-removal practices will include requirements prototyping, formal design inspections, personal desk-checking of code, unit testing, integration testing, system testing, and high-volume beta testing. As the data in the table shows, this combination of techniques is estimated to remove about 95% of defects prior to the software's release.
| Activity | Effect on Defects | Total Defects Produced So Far | Defects Still Remaining |
|---|---|---|---|
| Requirements | +1,000 defects | 1,000 | 1,000 |
| Requirements prototyping | -65% | 1,000 | 350 |
| Architecture | +1,250 defects | 2,250 | 1,600 |
| Formal design inspections | -55% | 2,250 | 720 |
| Construction | +1,750 defects | 4,000 | 2,470 |
| Documentation | +600 defects | 4,600 | 3,070 |
| Personal desk checking of code | -40% | 4,600 | 1,842 |
| Unit testing | -30% | 4,600 | 1,289 |
| Integration testing | -35% | 4,600 | 838 |
| System test | -40% | 4,600 | 503 |
| Bad fixes | +400 defects | 5,000 | 903 |
| High-volume beta testing | -75% | 5,000 | 226 |
| Defects remaining at release | -95% | 5,000 | 226 (5%) |
As with the previous example, the specific estimate of 226 defects is more precise than is supported by the underlying data.
| Tip #101 | Use defect-removal-rate data to estimate the number of defects that your quality assurance practices will remove from your software before it is released. |
Lawrence Putnam provides two additional rules of thumb for defect removal. If you want to move from 95% reliability to 99% reliability, you should plan to add 25% to the "main build" part of your schedule. You should plan to add another 25% to your schedule to improve from 99% to 99.9% reliability (Putnam and Myers 2003). (In Putnam's terminology, "reliability" and "pre-release defect removal" are synonymous.)
Further estimation of quality attributes can be an involved topic that relies heavily on the science of estimation. The "Additional Resources" section at the end of this chapter describes where to find more information.
EAN: 2147483647
Pages: 212