Chapter 3: Approaches to Managing Software Testing

Many major hotels in Las Vegas, Nevada, offer gaming schools as a free service to their patrons. The first lesson that they teach in gaming school goes something like this: The instructor takes great pains to point out the red velvet wall coverings, the crystal chandeliers, and all the rich appointments that surround you in the casino. Then they tell you, "Look around. See all the money that built Las Vegas? Always remember, the losers built Las Vegas. They played their hunches. They are the ones that felt lucky."

The I-Feel-Lucky Approach to Software Development

The gaming school teaches students to resist the "I feel lucky" impulse and use methods instead. They show you how to figure out if your hand of cards is more likely to win than someone else's and how to calculate odds of certain combinations of numbers showing up on a pair of dice. The methods taught in gaming school are based on simple counting techniques and probability. They are very similar to the methods taught in this book.

If you have any doubts about the results of using methods versus the I-feel-lucky approach, consider that today most casinos are populated exclusively by computer-driven games-not cards or dice. This system is in place because there are no counting techniques that can be used to successfully calculate or predict the odds of winning in computer gaming. Only a very knowledgeable hacker or someone with inside information has a chance of winning more than they invest with electronic gaming. Clearly, the financial thinkers at the casinos saw the power of good methods and moved to circumvent them.

If management does not have a method that has demonstrated value, they will take their best guess instead. If the product succeeds, management will believe in their winning streak and continue to play their hunches. Generally, no one is much interested in trying a formal method-until this luck runs out and the winning streak breaks. If you want to prove that in the long run a method works better than luck, you have to measure how well each actually performs.

Note 

Methods and metrics must provide demonstrable value to have credibility and worth.

As I said in Chapter 1, currently few people feel lucky, and we (testers) once again have a good chance to prove our worth. But what should that look like? How does a tester make a case for methods with a manager who does not believe that he or she needs to test? The best way I know is to provide that manager with high-quality information that he or she can use to make decisions that have a positive affect on the bottom line.

To do that you will have to develop an approach to testing that complements (and can succeed with) the development methodology being used. You will have to measure and keep track of what you measure, and finally, you will have to convert your measurement data into that information that management finds truly valuable. And that's what this book is about.

Another way that you can demonstrate the value of methods and metrics is to measure the cost benefit of not using the methods and metrics and compare that to the cost benefit of using them.

For example, in a case study of an early-1990s shrink-wrap RAD project that I reported in 1994, a group of trained testers, using the methods and metrics in this book, found bugs at the rate of two to three per hour, while untrained testers, who were not using formal methods and metrics, found three bugs per day in the same applications. Further, over 90 percent of the bugs reported by the trained testers were fixed, while half of the bugs reported by the untrained testers were returned by developers as unreproducible.

In this study the trained testers were paid a wage that was almost double that of the untrained testers. Even at double the pay it cost an average of $13 for the trained testers to find a bug, while it cost an average of $50 for the untrained testers to find the same bug. Even when the cost of test education is taken into account, the testers using methods and metrics are far more efficient. These are facts that will motivate management to consider trying the methods and metrics used by the trained testers.

The best reason for using methods and metrics is that companies using them have a competitive advantage. They get a much better job done for less.

I firmly believe that a competent journeyman tester (the ones using methods and metrics) can successfully test any development project regardless of the development methodology being employed as long as they have sufficient resources and management support to do so. Developing or tailoring the test approach to suit the needs of the project is a critical piece of acquiring those resources and that support.

Before we examine the current approaches to testing in the field and the approaches you should take, let's examine some myths about testing and the approaches to testing. What are some of the widespread, usually erroneous assumptions about the approaches we should take to testing?