Determining the Effectiveness of Your SRE-Based Test Process

Software reliability engineering promises to help teams work smarter to deliver a reliable product. But how to tell if it is working? This chapter concludes by looking at how to measure the Defect Detection Effectiveness of a testing process, touted by some as the single most important quality measure.

Defect Detection Effectiveness (DDE) is a measure of the effectiveness of a testing process to detect defects. The formula for DDE is very simple: it is the number of bugs detected by the testing process before the product release divided by the total population of bugs (see Equation 4.4).

Equation 4.4 Formula for calculating Defect Detection Effectiveness (DDE).

A run chart of defects through time is a good way to visualize DDE. Figure 4.22 is a run chart of cumulative severe defects found for a product release; in fact, it is the same release for which we measured the failure intensity earlier in this chapter (see Figure 4.7).

Figure 4.22. Run chart illustrating defects found in system test before and after release.

In this case, at the time testing was stopped and the product released, the testing effort had found 582 severe defects. Subsequent to release, an additional 50 severe defects were found; therefore, DDE = 582 / (582+50), or about 92%.

Because product testing is typically not directly equivalent to the use a product experiences in the field from customersit's usually more rigorous or less rigorous, but not exactly the samearriving at a failure intensity objective may involve some trial and error. A failure intensity objective that is too low (your reliability goal is too high) keeps a product in testing longer than it needs to be. A failure intensity objective that is too high (your reliability goal is too low) results in unhappy customers inflicted with a buggy product along with the high cost of fixing defects in a deployed product.

In the "But What's the Right Failure Intensity Objective?" section, a low-tech approach was described for determining a ballpark failure intensity objective: Think back to a project that you and your team remember as being successful and that you believe was well received by the customer in terms of reliability, then do some project archaeology to determine the failure intensity of that project. DDE provides a quantitative approach allowing you to correlate failure intensity objectives with successful releases measured by DDE. By tracking failure intensities during testing, followed by DDE analysis after the release, you are able to determine whether you can raise or lower failure intensity objectives.

As an example, the project for which we computed the failure intensity at time of release in Figure 4.7 is also the same project for which we have just calculated the DDE in Figure 4.22. So, as it turned out, a failure intensity of 1.55 at the time of release correlated with a DDE of about 92%, which was considered a success. Future projects could therefore use a failure intensity objective of 1.55 with some measure of confidence that they would be releasing a product the customer would perceive as reliable.

Another way to leverage DDE analysis with software reliability engineering is by comparison of defects found soon after release versus much later. This is illustrated in Figure 4.23; arrows show defects found almost immediately after the release compared with defects found up to seven months later. Doing root cause analysis of such defects and asking why some were caught so quickly and others took so long to be found can provide insight into how to improve your system's use cases, operational profile, and test cases.

Figure 4.23. Analysis of defects found sooner rather than later can provide insights into the operational profile of your product as used by the customer.

Final Notes on DDE

Although DDE is a simple measure, there are caveats of which to be aware. First, DDE is a moving target; because it is impossible to know how many defects there really are in a product, it is computed based on known defects at some point in time, and that, of course, changes through time. Looking at the run chart of Figure 4.22, the measured DDE starts at 100% at the time of release (you don't know what you have missed at this point) and then continues to drop through time as defects are discovered after the release. The trick is to identify a period of time after a release that is long enough for the customer base to have adopted the new release and to have had sufficient opportunity to begin finding defects.

There is also some confusion in the literature around this and similar measures. Defect Removal Effectiveness is a measure of the effectiveness of a development process to both detect, and then remove, defects. Some literature blurs this distinction (detection versus removal). I prefer detection effectiveness rather than removal effectiveness because many development groups don't necessarily fix every defect they findeven severe onesfor valid business reasons.

Also, some literature blurs the distinction between "efficiency" and "effectiveness" by using the term defect detection efficiency as equivalent to defect detection effectiveness. I prefer to keep these two measures distincteffectiveness versus efficiency, as two organizations can have the same effectiveness to detect/remove defects, with varying efficiency at accomplishing the task. Yet other authors avoid the use of "effectiveness" or "efficiency" altogether, using the term defect detection percentage.

So, just be aware that you may come across this measure called out by a different name; your best bet is to simply look at how the measure is calculated.