Chapter 9: Risk Analysis

Overview

"What are you going to test?" asked the vice president.

"The Most Important things," the tester replied.

"And how do you know what the most important things are?" asked the vice president.

Therein lies the tale.... Again, I was the tester in this conversation, and in truth the process of answering this question continues even today.

Engineers have been performing risk analysis for thousands of years. (It's been done ever since the first building fell down and the engineer's head rolled shortly thereafter.) It's interesting that some entrepreneurs in software testing houses are reinventing it as a marketing concept just when the heat is coming up under the software engineers.

To mitigate the risk of some event causing damage, you must first estimate the probability that the event will occur. This probability has to be translated into some quantity, usually represented as a percentage-for example, "There is a 50 percent chance that this will happen." Next, you need to determine the severity of such a failure. Severity is usually measured in currency, such as dollars, and loss of life. If the severity is minor, then even a high probability of occurrence may still be judged to cause a trivial problem.

If the severity of the failure and its probability of occurrence rise above a certain threshold, then it warrants preventative action. In engineering, a standard is put in place to ensure that the correct preventative actions are taken during construction so that if the event occurs, it will not cause the structure to fail. This standard is a rule.

In structures, the need for standards has long been understood, so we have building codes, which are sets of rules. Building codes ensure that all buildings are built to have enough structural integrity to withstand the demands that will probably be placed on them.

The probability that a thing will or won't occur can be calculated under certain circumstances-especially if you can answer a question like, "What was the outcome last time?" or "Do we know if the platform can really do what the maker claims it can?" If you can't provide a good measured answer to these questions up front, then you will need a strategy for dealing with the events that will occur later in the process. If the probability and severity cannot be measured, then they must be estimated. MITs risk analysis provides a formal method for both estimating up front and dealing with events as they unfold. In this chapter, we look at this formal approach to establishing risk and prioritizing the items on the test inventory.

MITs risk analysis uses both quantitative and qualitative analysis to establish a numeric value for risk based on a number of specific criteria. In the early planning phases of a test effort, this risk number is used to focus test resources to size the test effort. As the inventory evolves, the risk ranking plays an important part in actual test selection and optimal test coverage determination.