Section 8.3. Test::Harness

8.3. Test::Harness

When you've installed CPAN modules, you might have noticed two different styles of test output. In the first instance, you run something like perl -Mblib t/1.t and you see a list of results:

     1..25     ok 1 - Loaded module     ok 2 - Can create a new object     ok 3 -  ... of the correct class     ...

And in the second case, you run make test on a MakeMaker-generated install process or prove t/*.t, and you see something like this:

     t/1...............ok     t/2...............ok     t/3...............FAILED tests 2, 5             Failed 2/6 tests, 66.67% okay     ...     Failed Test Stat Wstat Total Fail  Failed  List of Failed     -------------------------------------------------------------------------------     t/3.t                      6    2  33.33%  2 5     Failed 1/20 test scripts, 95.00% okay. 2/349 subtests failed, 99.43% okay.

So, what's the difference? The difference is that, in the second case, something is running each test file in the t/ directorywhether it's one file or many filesand collating the results. The thing that's doing the collating is Test::Harness. Its job is to gather up the test results and make sure everything went OK before the module gets installed.

So, if you're planning on writing tests that don't use the standard Test::Simple or Test::More modules (or indeed any of the other test modules out there), or you want to write your own test module, then you need to know how to produce test output that Test::Harness is going to be happy with. Otherwise, it will think your module is failing its tests. This standard output format is known as TAPthe Test Anything Protocoland credit for the name goes to Joe McMahon and Andy Lester.

The interface to Test::Harness is the runtests function. You give runtests a list of filenames, it runs each one in turn and produces the summary you just saw. That's all. The interesting question is what Test::Harness expects from a test suite.

The first thing it expects to see from a test suite is a plan. A plan is a line of text, of the form 1..N, and it must appear as either the first or the last thing seen on standard output. This ensures that Test::Harness can determine whether your test ended when it was supposed to or died in the middle. If you don't know how many tests you're going to have until you've run them, you can put out a plan right at the end. But you must have one, and only one, either at the very start or the very end.

Each test must output the word ok, or the words not ok, and they must say this at the beginning of a line. They don't have to say what number they are, but it's useful.

Test output can contain comments. Like a comment in Perl, these begin with a hash character. After the ok (or the not ok) and the test number, you can have a description that says what your test is called. Usually these are introducted by a dash, but anything between the test number and either a # character or the end of the line is treated as the description. Here are some valid test results:

     ok     not ok 2     ok 3 - Array in correct order

And here are some things that are not valid test results:

     OK     Checking to see if we can parse the XML again... ok     4 not ok

Test::Harness treats certain test comments specially. These are called directives. If a comment starting with skip immediately follows the test number, then the harness notes that this test has been skipped. Similarly, as we saw when looking at Test::More, the harness marks a test with a TODO directive as non-fatal.

There are other things your test script might produce that Test::Harness knows how to deal with. You can specify that you want to skip the entire test file, by writing out:

     1..0

Or you can abort the current test by outputting the magic words Bail out!. Most other things in your test output will be ignored by the version of Test::Harness current at the time of writing, although that may change. For more details on this format, read Test::Harness::TAP.