Code Coverage

As with black-box testing, testing the data is only half the battle. For comprehensive coverage you must also test the program's states and the program's flow among them. You must attempt to enter and exit every module, execute every line of code, and follow every logic and decision path through the software. This type of testing is known as code coverage testing.

Code coverage is dynamic white-box testing because it requires you to have full access to the code to view what parts of the software you pass through when you run your test cases.

The simplest form of code coverage testing is using your compiler's debugger to view the lines of code you visit as you single-step through the program. Figure 7.9 shows an example of the Visual Basic debugger in operation.

Figure 7.9. The debugger allows you to single-step through the software to see what lines of code and modules you execute while running your test cases.

For very small programs or individual modules, using a debugger is often sufficient. However, performing code coverage on most software requires a specialized tool known as a code coverage analyzer. Figure 7.10 shows an example of such a tool.

Figure 7.10. A code coverage analyzer provides detailed information about how effective your test cases are. (This figure is copyright and courtesy of Bullseye Testing Technology.)

Code coverage analyzers hook into the software you're testing and run transparently in the background while you run your test cases. Each time a function, a line of code, or a logic decision is executed, the analyzer records the information. You can then obtain statistics that identify which portions of the software were executed and which portions weren't. With this data you'll know

• What parts of the software your test cases don't cover. If a specific module is never executed, you know that you need to write additional test cases for testing that module's function.

• Which test cases are redundant. If you run a series of test cases and they don't increase the percentage of code covered, they are likely in the same equivalence partition.

• What new test cases need to be created for better coverage. You can look at the code that has low coverage, see how it works and what it does, and create new test cases that will exercise it more thoroughly.

You will also have a general feel for the quality of the software. If your test cases cover 90 percent of the software and don't find any bugs, the software is in pretty good shape. If, on the other hand, your tests cover only 50 percent of the software and you're still finding bugs, you know you still have work to do.

REMINDER

Don't forget about the Pesticide Paradox described in Chapter 3, "The Realities of Software Testing"the more you test the software, the more it becomes immune to your tests. If your test cases cover 90 percent of the software and you're not finding any bugs, it could still be that your software isn't in good shapeit could be that it has become immune. Adding new test cases may reveal that the next 10 percent is very buggy!

Program Statement and Line Coverage

The most straightforward form of code coverage is called statement coverage or line coverage. If you're monitoring statement coverage while you test your software, your goal is to make sure that you execute every statement in the program at least once. In the case of the short program shown in Listing 7.1, 100 percent statement coverage would be the execution of lines 1 through 4.

Listing 7.1. It's Very Easy to Test Every Line of This Simple Program

 1: PRINT "Hello World" 2: PRINT "The date is: "; Date$ 3: PRINT "The time is: "; Time$ 4: END 

You might think this would be the perfect way to make sure that you tested your program completely. You could run your tests and add test cases until every statement in the program is touched. Unfortunately, statement coverage is misleading. It can tell you if every statement is executed, but it can't tell you if you've taken all the paths through the software.

Branch Coverage

Attempting to cover all the paths in the software is called path testing. The simplest form of path testing is called branch coverage testing. Consider the program shown in Listing 7.2.

Listing 7.2. The IF Statement Creates Another Branch Through the Code

 1: PRINT "Hello World" 2: IF Date$ = "01-01-2000" THEN 3:     PRINT "Happy New Year" 4:     END IF 5: PRINT "The date is: "; Date$ 6: PRINT "The time is: "; Time$ 7: END 

If you test this program with the goal of 100 percent statement coverage, you would need to run only a single test case with the Date$ variable set to January 1, 2000. The program would then execute the following path:

Lines 1, 2, 3, 4, 5, 6, 7

Your code coverage analyzer would state that you tested every statement and achieved 100 percent coverage. You could quit testing, right? Wrong! You may have tested every statement, but you didn't test every branch.

Your gut may be telling you that you still need to try a test case for a date that's not January 1, 2000. If you did, the program would execute the other path through the program:

Lines 1, 2, 5, 6, 7

Most code coverage analyzers will account for code branches and report both statement coverage and branch coverage results separately, giving you a much better idea of your test's effectiveness.

Condition Coverage

Just when you thought you had it all figured out, there's yet another complication to path testing. Listing 7.3 shows a slight variation to Listing 7.2. An extra condition is added to the IF statement in line 2 that checks the time as well as the date. Condition coverage testing takes the extra conditions on the branch statements into account.

Listing 7.3. The Multiple Conditions in the IF Statement Create More Paths Through the Code

 1: PRINT "Hello World" 2: IF Date$ = "01-01-2000" AND Time$ = "00:00:00" THEN 3:     PRINT "Happy New Year" 4:     END IF 5: PRINT "The date is: "; Date$ 6: PRINT "The time is: "; Time$ 7: END 

In this sample program, to have full condition coverage testing, you need to have the four sets of test cases shown in Table 7.2. These cases assure that each possibility in the IF statement are covered.

Table 7.2. Test Cases to Achieve Full Condition Coverage

Date$	Time$	Line # Execution
01-01-1999	11:11:11	1,2,5,6,7
01-01-1999	00:00:00	1,2,5,6,7
01-01-2000	11:11:11	1,2,5,6,7
01-01-2000	00:00:00	1,2,3,4,5,6,7

If you were concerned only with branch coverage, the first three conditions would be redundant and could be equivalence partitioned into a single test case. But, with condition coverage testing, all four cases are important because they exercise the different conditions of the IF statement in line 2False-False, False-True, True-False, and True-True.

As with branch coverage, code coverage analyzers can be configured to consider conditions when reporting their results. If you test for condition coverage, you will achieve branch coverage and therefore achieve statement coverage.

NOTE

If you manage to test every statement, branch, and condition (and that's impossible except for the smallest of programs), you still haven't tested the program completely. Remember, all the data errors discussed in the first part of this chapter are still possible. The program flow and the data together make up the operation of the software.