Chapter 10: Applied Risk Analysis

Overview

The first time I told my management that I would not be testing 100 percent of the project, they were shocked and outraged. There ensued a lively discussion. I found myself with a lot of explaining to do. But once I had showed them my inventory with its hundreds of test scenarios, they were ready to hear about my process for ranking the requirements and tests so that I would only run the most important tests, even if that represented only 67 percent of the inventory, rather than 100 percent.

The managers were much more amenable to the 67 percent test coverage once I had walked them through my test sizing worksheet and they became acquainted with how long it would take the test team to actually perform those tests. Finally, my management was particularly pleased when, many months later, I was able to report to them that these most important tests had found and caused to be removed 90 percent of all the bugs ever found in the software, before it went to production.

I used my inventory ranking worksheet and my test sizing worksheet to explain the facts of testing to my management all those years ago. It still works today. The biggest difference is that back when I started, my worksheets were written and calculated by hand on a quadrille pad. Today they are automated in an Excel spreadsheet. Back then I spent a lot of effort and time to produce these worksheets; today they take only a small amount of effort, and they give me answers so fast that I can keep up and even get ahead of most RAD/Agile developers. In fact, in many Web projects, my worksheets end up being the only piece of development collateral extant.

The ranking worksheet shows the relative importance and the recommended test coverage for the inventory item. The sizing worksheet takes those quantities and combines them with project level constants to grind out an estimate of the time and resources that will be required to perform the recommended test coverage of the most important tests. Or, from another viewpoint, the worksheet tells you how much remains to be done.

These worksheets are very popular with the bean counters. The ranking worksheet and the sizing worksheet have often proved to be the only record of how big the project is, how extensive the test effort should be, and why. The bean counters understand the worksheet; it turns data into information. This information answers many of their favorite questions, like the ones discussed in Chapter 1, "The State of Software Testing Today." These include the following:

• What does the software do? The inventory contains the requirements, if they exist; the feature list; the environment catalog; and so on.

• What are you going to do to prove that it works? A tester reads this as, "What are you going to test?" and "How are you going to test it?" The risk analysis and MITs ranking identifies exactly what needs to be done to prove that it works and how I plan to test it. It also answers the question, "How big is it?"

• What will it cost? The worksheet allows me to answer this question, both by estimation during planning and in actuality during the test effort as the estimates are replaced with actual measured quantities.

Usually the auditors don't want to know how I got the inventory; they just pick it up and run with it. Oddly, in all my years of testing, the developers in the real-world shipping project, discussed in Chapter 7, are the only ones who have ever seriously challenged my risk analysis.

Again, every project is different, and each sizing worksheet is likely to be different as well. I have developed a new flavor of sizing worksheet in almost every project I have ever done. But I almost always start with the same Excel spreadsheet template. The point is, I could not give management the quality answers that I do without an inventory and a sizing worksheet.

Note 

The worksheet is the place where all those numbers, those measurements I have been recording, become valuable information. This information can be used to make decisions.

This chapter talks about how to use risk analysis on the inventory to produce answers to the questions in the preceding list. You will find ways to build a preliminary sizing worksheet and use it in test estimation. Then you will learn how to continue to use the inventory and sizing worksheet as you refine your estimates and add the most important tests to the inventory.

As in previous chapters, these techniques are discussed in terms of how they apply to various development approaches.