| Having some tests is better than having no tests, but having enough tests is better yet. Code coverage is one way to measure how much of the code the tests actually test. Analyzing code coverage by hand is tedious . Fortunately, the Devel ::Cover module from the CPAN automates the analysis and reporting for you. Best of all, it works with the standard Perl test harness. How do I do that? Install Devel::Cover and its dependencies (see "Installing Test Modules" in Chapter 1). You need the ability to build XS modules, unless you install it via ppm or some other binary package.
Note: XS is the Perl extension system. It allows the use of code written in languages other than Perl and requires a working C development environment .
From the top level of a module directory, such as Test::Harness (see "Organizing Tests," earlier in this chapter), build the module, and then run the following commands:
Note: If your module uses Module:: Build, use perl Build testcover instead of make test. Otherwise, install ExtUtils:: MakeMaker:: Coverage and use make testcover .
$ cover -delete Deleting database /home/chromatic/dev/install/Test-Harness-2.46/cover_db $ HARNESS_PERL_SWITCHES=-MDevel::Cover make test PERL_DL_NONLAZY=1 /usr/bin/perl5.8.6 "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/00compile.........ok 1/5# Testing Test::Harness 2.46 t/00compile.........ok t/assert............ok t/base..............ok t/callback..........ok t/harness...........ok t/inc_taint.........ok t/nonumbers.........ok t/ok................ok t/pod...............ok t/prove-globbing....ok t/prove-switches....ok t/strap-analyze.....ok t/strap.............ok t/test-harness......ok 56/208 skipped: various reasons All tests successful, 56 subtests skipped. Files=14, Tests=551, 255 wallclock secs (209.59 cusr + 4.98 csys = 214.57 CPU) $ cover Reading database from /home/chromatic/dev/Test-Harness-2.46/cover_db ---------------------------------- ------ ------ ------ ------ ------ ------ File stmt branch cond sub time total ---------------------------------- ------ ------ ------ ------ ------ ------ blib/lib/Test/Harness.pm 71.6 51.6 61.1 80.8 0.0 65.9 blib/lib/Test/Harness/Assert.pm 100.0 100.0 n/a 100.0 0.0 100.0 blib/lib/Test/Harness/Iterator.pm 70.0 25.0 n/a 80.0 98.9 65.5 blib/lib/Test/Harness/Straps.pm 92.9 82.7 69.0 96.2 1.0 87.6 Total 80.8 66.0 65.4 88.3 100.0 76.0 ---------------------------------- ------ ------ ------ ------ ------ ------ Writing HTML output to /home/chromatic/dev/Test-Harness-2.46/cover_db/coverage.html ... done.
Note: See the documentation for your shell to learn how to set the HARNESS_ PERL_ SWITCHES environment variable .
Note: By default, Devel::Cover ignores the coverage of any file found in @INC, all *.t files, and Devel:: Cover itself. See the +ignore, -ignore, +inc, and -inc options in perldoc Devel:: Cover to learn how to customize this .
This will take a while ”several times as long as it takes your test suite to run normally. Your reward is a nice tabular summary at the end as well as some HTML files in the reported location. What just happened ? When Devel::Cover runs a test suite, it profiles Perl code at the operational level, marking every subroutine, statement, branch, and condition in the code being tested to see if the tests actually exercise them. It writes its output to a database file from which it can produce coverage reports . The important results are in the report shown at the end, where each file being tested has a percentage for subroutine, statement, branch, and condition coverage as well as the percentage of the time spent testing for that file and its overall coverage. What are all of the types of coverage? -
- Statement coverage
-
Asks whether a test exercised a particular statement. Given the statement $flag = 1; , any test that causes that statement to execute will count as having covered it. -
- Branch coverage
-
Tracks whether the tests exercised both parts of a branching statement. Given the code print "True!" if $flag; , the statement must run twice ”once where $flag is true and once where it is false ”to achieve 100 percent branch coverage.
Note: The more complex your conditions, the more difficult they are to test, let alone read .
-
- Condition coverage
-
Considers all of the possibilities of a logical condition. Given the assignment my $ name = shift 'Ben'; within a subroutine, the test must pass in a string with an actual value for $name at least once and pass in no argument or an empty string at least once (receiving the default value) to achieve full coverage for that conditional expression. This is a very simple type of condition coverage, with only one variable and two paths for coverage. More common are conditions with two variables : short-circuiting expressions such as $a = $x $y have three paths for coverage, and fully evaluated expressions such as $a = $x xor $y have four paths for coverage.
Note: Devel::Cover also runs Pod::Coverage (see "Testing Documentation Coverage," in Chapter 4) and reports its results if you have it installed .
-
- Subroutine coverage
-
Checks that a test exercised at least part of a subroutine. If you don't have full coverage for a particular module, start with the subroutine coverage report to see which pieces of code need more tests. Open the reports in your favorite web browser. You'll see a colorful hyperlinked summary generated by the final cover run (Figure 3-1). Figure 3-1. Coverage summary 
Click on the branch, condition, or subroutine coverage links to reach a page of metrics that Devel::Cover gathered for every affected line in each tested module.
Note: Devel::Cover uses B:: Deparse to produce the output for the branch and condition reports. This generates behaviorally ”but not necessarily typographically ”equivalent code .
Consider the links for Test::Harness . The branch coverage is 51.6 percent. Click on that link to see a report showing line numbers for all of branches, the percentage of coverage for each branch, the true or false conditions taken for the branch, and the approximate branch expression. Figure 3-2 shows more details. The T and F columns show whether Devel::Cover believes that the tests exercised the true and false versions of the branch, respectively. A green background means yes and a red background means no. The test run of this example exercised both true and false branches of the condition in line 229, but exercised only the false branch in line 322. Figure 3-2. Branch coverage report 
The condition coverage report page is more complex. For each condition, it reports the line number, the percentage of the condition the tests exercised, and the deparsed code of the conditional expression. However, the important details appear in a truth table that lists all possible boolean combinations for each element of the expression. In Figure 3-3, the tests exercised none of the possible combinations on line 210. Line 229 fared better, with the first expression tested for the two cases: where A is false and then where both A and B are true. The second expression had two tests as well, for the cases where A is false and B is true and for the case where A is true.
Note: A is the first possible outcome of the condition, B is the second, C is the third, and so on .
The final report, which shows subroutine coverage, is very simple. It lists the name and the line number of each subroutine, indicating with a red or green cell background whether the tests covered it. Figure 3-4 shows several BEGIN blocks (mostly use statements), with strap( ) and _all_ok( ) having at least some tests and runtests( ) and _globdir( ) having none. What about... | Q: | How do I improve my test coverage? | | A: | Start with the subroutine coverage report. Any subroutine marked as untested may have lurking bugs , or it may go unused throughout the code. Either way, consider the affected code carefully . Achieving complete test coverage can be difficult; Devel::Cover has a complicated job to do and does it well, but it's not perfect. Running Figure 3-3. Condition coverage report 
Note: 1 means true, 0 means false, and X means that it doesn't matter because of a short-circuited condition .
Figure 3-4. Subroutine coverage report 
a recent version of Perl will help, as will upgrading to newer versions of Devel::Cover as they release. At times, you may have to simplify complex constructs or live with less than 100 percent coverage. As the documentation says, though, reporting potential bugs to the perl-qa@perl.org mailing list is a good way to find enlightenment.
Note: Perl 5.8.2 is the minimum recommended version for using Devel::Cover. Any newer version should work .
It is not always possible to achieve 100 percent coverage for all metrics. Even when it is, trying to reach that goal may not be the best use of your testing efforts. Code coverage can highlight areas in which your test suite is weak and help you reason about your code. Understand what your test suite does not test and why is valuable , even if you decide not to write a test for it. | |
