Code Coverage
As with black-box testing, testing the data is only half the battle. For comprehensive coverage you must also test the program's states and the program's flow among them. You must attempt to enter and exit every module, execute every line of code, and follow every logic and decision
path
through the software. This type of testing is known as
code coverage
testing.
Code coverage is dynamic white-box testing because it requires you to have full access to the code to view what
parts
of the software you pass through when you run your test cases.
The simplest form of code coverage testing is using your compiler's debugger to view the lines of code you visit as you single-step through the program. Figure 7.9 shows an example of the Visual Basic debugger in operation.
For very small programs or individual modules, using a debugger is often sufficient. However, performing code coverage on most software requires a specialized tool known as a
code coverage analyzer
. Figure 7.10 shows an example of such a tool.
Code coverage analyzers hook into the software you're testing and run
transparently
in the background while you run your test cases. Each time a function, a line of code, or a logic decision is executed, the analyzer records the information. You can then obtain statistics that identify which portions of the software were executed and which portions weren't. With this data you'll know
-
What parts of the software your test cases don't cover. If a specific module is never executed, you know that you need to write additional test cases for testing that module's function.
-
Which test cases are redundant. If you run a series of test cases and they don't increase the percentage of code covered, they are likely in the same equivalence partition.
-
What new test cases need to be created for better coverage. You can look at the code that has low coverage, see how it works and what it does, and create new test cases that will exercise it more thoroughly.
You will also have a general feel for the quality of the software. If your test cases cover 90 percent of the software and don't find any bugs, the software is in pretty good shape. If, on the other hand, your tests cover only 50 percent of the software and you're still finding
bugs
, you know you still have work to do.
REMINDER
Don't forget about the Pesticide Paradox described in Chapter 3, "The Realities of Software Testing"the more you test the software, the more it becomes immune to your tests. If your test cases cover 90 percent of the software and you're not finding any bugs, it could still be that your software isn't in good shapeit could be that it has become immune. Adding new test cases may reveal that the
next
10 percent is very
buggy
!
Program Statement and Line Coverage
The most straightforward form of code coverage is called
statement coverage
or
line coverage
. If you're monitoring statement coverage while you test your software, your goal is to make sure that you execute every statement in the program at least once. In the case of the short program shown in Listing 7.1, 100 percent statement coverage would be the execution of lines 1 through 4.
Listing 7.1. It's Very Easy to Test Every Line of This Simple Program
1: PRINT "Hello World"
2: PRINT "The date is: "; Date$
3: PRINT "The time is: "; Time$
4: END
You might think this would be the perfect way to make sure that you
tested
your program completely. You could run your tests and add test cases until every statement in the program is touched. Unfortunately, statement coverage is misleading. It can tell you if every statement is executed, but it can't tell you if you've taken all the paths through the software.
Branch Coverage
Attempting to cover all the paths in the software is called
path testing
. The simplest form of path testing is called
branch coverage
testing. Consider the program shown in Listing 7.2.
Listing 7.2. The
IF
Statement Creates Another Branch Through the Code
1: PRINT "Hello World"
2: IF Date$ = "01-01-2000" THEN
3: PRINT "Happy New Year"
4: END IF
5: PRINT "The date is: "; Date$
6: PRINT "The time is: "; Time$
7: END
If you test this program with the goal of 100 percent statement coverage, you would need to run only a single test case with the
Date$
variable set to January 1, 2000. The program would then execute the following path:
Lines 1, 2, 3, 4, 5, 6, 7
Your code coverage analyzer would state that you tested every statement and achieved 100 percent coverage. You could quit testing, right? Wrong! You may have tested every
statement
, but you didn't test every
branch
.
Your gut may be telling you that you still need to try a test case for a date that's not January 1, 2000. If you did, the program would execute the other path through the program:
Lines 1, 2, 5, 6, 7
Most code coverage analyzers will account for code branches and report both statement coverage and branch coverage results separately, giving you a much better idea of your test's effectiveness.
Condition Coverage
Just when you thought you had it all figured out, there's yet another complication to path testing. Listing 7.3 shows a
slight
variation to Listing 7.2. An extra condition is added to the
IF
statement in line 2 that checks the time as well as the date.
Condition coverage
testing takes the extra conditions on the branch statements into account.
Listing 7.3. The Multiple Conditions in the
IF
Statement Create More Paths Through the Code
1: PRINT "Hello World"
2: IF Date$ = "01-01-2000" AND Time$ = "00:00:00" THEN
3: PRINT "Happy New Year"
4: END IF
5: PRINT "The date is: "; Date$
6: PRINT "The time is: "; Time$
7: END
In this sample program, to have full condition coverage testing, you need to have the four sets of test cases shown in Table 7.2. These cases assure that each possibility in the
IF
statement are covered.
Table 7.2. Test Cases to Achieve Full Condition Coverage
|
Date$
|
Time$
|
Line # Execution
|
|
01-01-1999
|
11:11:11
|
1,2,5,6,7
|
|
01-01-1999
|
00:00:00
|
1,2,5,6,7
|
|
01-01-2000
|
11:11:11
|
1,2,5,6,7
|
|
01-01-2000
|
00:00:00
|
1,2,3,4,5,6,7
|
If you were
concerned
only with branch coverage, the first three conditions would be redundant and could be equivalence partitioned into a single test case. But, with condition coverage testing, all four cases are important because they exercise the different conditions of the
IF
statement in line 2False-False, False-True, True-False, and True-True.
As with branch coverage, code coverage analyzers can be configured to consider conditions when reporting their results. If you test for condition coverage, you will achieve branch coverage and therefore achieve statement coverage.
NOTE
If you manage to test every statement, branch, and condition (and that's
impossible
except for the smallest of programs), you still haven't tested the program completely. Remember, all the data errors discussed in the first part of this chapter are still possible. The program flow and the data together make up the operation of the software.
|
