Risk Criteria for Picking Tests

As with requirements and functions (discussed in the previous chapter), where criteria must be applied when ranking them, you also need to apply additional criteria when selecting the most important tests. In general, each of the criteria that I use is related to a type of analysis or some other process that defines tests. In this next section, I talk about the tests, the analysis methods I use to identify them, and the criteria I use to rank them. Ultimately, it is the rank of the tests that determines exactly what will be tested and what will not.

There are an infinite number of things that could be tested even for a small, shrink-wrapped application. There can be a hundred permutations on simply resizing a window. It is not generally cost-effective to try and test everything that can be tested. I want to pick the most important tests, and I do that by applying criteria to each test to measure its relative importance. I want to select the tests that fulfill the following requirements.

Validating and Verifying the System Requirements

In traditional commercial software development endeavors (heavyweight to middleweight projects), the requirements are the basis of communication throughout the entire project. They must be well planned and clearly stated. It is a straightforward process to plan tests that verify system requirements when the requirements are well documented. Testers in a traditional software development effort must verify that the system meets the requirements. They were normally detached from the validation of requirements.

In a RAD/Agile effort, the requirements are changing faster than anything else, and they are rarely well documented. This makes it difficult to plan verification tests. Development delivers a function important to end users, and the RAD test team response is, "Oh, that's great, but can we make it do this, too?" Communication is direct; the requirements are embodied in the product. Testers in a RAD/Agile effort are more concerned with answering the validation question, "Does the system do the right things in an acceptable way?"

Exercising the Most Important Functions in the System

Exercise the least understood and the riskiest internal functions of the system-the tests that exercise the most important code or logic paths and data, along with all the interfaces, including module, system, database, network, and business rules. These tests often come together through path analysis (see Chapters 11 and 12), but they also can come directly from the design and requirements.

Exercising the Most Important Paths and Data

From the perspective of the user, the most used functions and data are the most important paths to be tested. These path and data tests are in addition to those identified previously as system function needs. See Chapter 11, "Path Analysis," Chapter 12, "Applied Path Analysis," and Chapter 13, "Data Analysis Techniques," for details on this type of test.

Giving the Most Bang for the Buck

If a test suite provides 50 percent test coverage and finds 97 percent of the errors ever found in the system, then the test coverage provided by this test suite is probably adequate. That's the kind of quality I want to target.

Tests that delve into the areas where the level of confidence is lowest are always a good bet. You are going to want to look very closely at the newer, riskier things. You will want to get way down in the code and make sure you test absolutely everything. Unfortunately, for RAD/Agile efforts, we usually only have time to make sure the function holds together under the best set of circumstances, and then we have to move on.

The test budget gets more bang from the bucks spent on scripts that pack the largest number of the best tests into the most optimized sequence. This type of test usually evolves over time and is the product of several test cycles. Another possible source for this type of test is an expert tester. In either case, these are usually the tests that will be used again and again. If possible, they should be automated. These test scripts are commonly used to form high-power diagnostic suites or basis suites.

Note 

When testers get into the code, there are always more paths to test than originally estimated.

The number of tests associated with each inventory item will continue to grow as you proceed to analyze them, whether that is before or during testing. The reason that I keep my totals on their own worksheet is that it allows me to add columns for the actual number of tests that I end up running. This allows me to carry both the estimated totals and the actual totals through the project. By keeping tabs on the estimates and the actuals, I can baseline the project, measure bloat, and establish factors of safety for the next project. Even in a RAD/Agile effort, this is important, because if the actual numbers deviate too far from the budgeted estimates, I can alert management early-instead of waiting until we are out of money.

This capability of providing constant status information is very compatible with the MFF approach. Without this type of information, testers must approach the test effort as if it were a pie-eating contest. They simply start testing when the product arrives and keep banging away on it until someone ships it. I hate to say it, but I believe that is a big part of the reason that testing has such a bad reputation today. The testers never know how big it is so that when the management asks, "What did you test?" they can say, "We tested it all." From their point of view, they tested everything that they knew about; the problem is that they never really saw the whole thing. No one talks about the heaps of pies that weren't dealt with before the "ship it" whistle blew.