Complementary Strategies for Selecting Test Cases

There are two possible avenues for selecting test cases for the system. One approach is to think about the types of defects that a product might contain and write test cases to find them. A second approach is to determine how the system will be used and build test cases that take it through its paces. In the following sections we will describe each technique and discuss how we use them together for an effective test strategy. Ultimately, we use the second approach to determine how many test cases to apply to each use case and then use the first approach to guide the selection of the test cases for maximum defect-finding power.

Use Profile

A traditional system test approach is to construct an operational profile. The profile is a listing of the relative frequency with which each end-user function in the system is used. For example, in Brickles, the player uses the mouse and moves it from side to side as the most frequent operation. Selecting a help feature or pausing play by depressing a mouse button are very infrequent operations.

This approach is also used in the computation of the reliability of a piece of software. Reliability is a measure of how long a software system operates without failure in a specific operational environment. The operational profile is one technique for specifying the operational environment. However, it is difficult to specify the operational profile until the system has been deployed for some time.

We defined a use profile in Chapter 4 as an accurate estimate of the operational profile. The use profile uses the priority of an operation rather than the frequency. It is possible to estimate the priority of a use case more accurately than it is to estimate the frequency of specific operations that may be used by several different types of users. We can estimate the priority of a use case by considering estimates of both frequency and criticality for each individual actor and then combining these for individual use cases. The use profile is built from the actor and use case diagrams.

ODC

Orthogonal Defect Classification (ODC) is a technique developed at IBM for capturing information about the type of faults that are present in a software system under development. This technique is useful for collecting and analyzing test information to direct a process improvement effort; however, our intent is to use the standard classifications developed by the creators of ODC as a basis for selecting test cases (see Orthogonal Defect Classification as a Test Case Selector on page 125).

Figure 9.3 lists the six categories of causes of failures identified in the ODC technique. Our interest is to be certain that we have built test cases that will allow each of these triggers to occur. Some of the categories, such as Startup and Normal Mode, are actually hard to avoid. However, the Exception category reminds us to cover every system-level exception just as we tried every exception at the component level. The word Recovery in that category also reminds us that the expected result of catching an exception should be clearly specified. It is not always possible to continue operations in the face of some exceptions; however, others are routinely encountered, such as "File Not Found."Any good program should be able to handle these.

Figure 9.3. ODC system-level failure triggers

The Hardware and Software Configuration categories are less obvious but very important areas for testing. For personal computers, software memory, for example, can be a major issue because there may not be a provision for virtual memory, or at least one that is sufficient. We have had clients who were very proud of the up-to-date development environment that every one of their developers had been given. Unfortunately some of their customers did not have that same environment and the program failed because the developer never encountered the situation in which insufficient memory was available to the program. The code failed to catch and handle out-of-memory exceptions.

The system test plan should include the use of a range of machines that have a variety of hardware devices. The sidebar System Configuration provides just one example of the interactions that can happen among software and hardware drivers. Likewise, software configurations can cause problems. Many programs have been thwarted by the order of libraries in a search path. While this is not a program defect, it is a defect in the installation process or the program documentation.

System Configuration

We developed a version of a tic-tac-toe game using the CORBA object request broker (ORB) that comes in the Java Developer's Kit (JDK). The naming service that comes with the JDK runs as a separate process that servers register with and clients query to locate the servers. We created a release of the game and tested it. The game was installed on a laptop that contained a combination modem/networking card. The game started, but did not appear to accept output. The process was left running while we went away to do something else. When we returned, the game was ready for a player to select a square. Selecting a square froze the game again. When the laptop was plugged into the network and the game was started, it ran at reasonable speed. Subsequent tests showed that the naming service object changed behavior with every state of the modem/networking card and was different on machines that had no network card at all.

There are many elements that go into the configuration of a system. Often these seem unrelated but later defects are traced to interactions between these elements. Different versions of the operating system, including foreign releases that are different from the domestic ones, font and language metrics, and even the environment variables can affect the execution of a system. Orthogonal array test designs can be used to reduce the number of combinations of factors that must be tested.