Hack 61. Test with Specifications

Let the computer write your tests.

Writing tests is a great way to gain confidence in your code. Each test you write makes a tiny claim about what your code ought to do. When it passes, you have clear evidence to support the claim. If you write enough tests to make a cohesive suite, the tiny claims within the suite combine to imply a general claim that your code works properly.

There are times, however, when writing a suite of tests is the hard way to make a general claim. Sometimes, the claim you want to make seems so simple, yet the tests you have to write seem so voluminous. For these times, it would be nice to be able to turn the process around. Instead of writing tests to make a claim, why not make the claim outright and let the computer write the tests for you? That's the idea behind specification-based testing. The Test::LectroTest family of modules brings this idea to Perl.

The Hack

To make claims about your code, you define properties that say that your code must behave in particular ways for a general spectrum of conditions. Then LectroTest automatically gathers evidence to support or refute your claims by executing large, random test suites that it generates on the fly. When it finishes, it prints the results in standard TAP format, just as the other Perl testing tools do.

Suppose you need to test Perl's sqrt function, which you expect to compute the square root of its argument. The first step is to figure out what sqrt ought to do. From your school days, you recall that the square root of a number x is the number that when multiplied by itself (that is, squared) gives you back x. For example, the square root of 9 is 3 because 3 times 3 is 9.

The square root function undoes the effect of squaring. You could consider this the defining property of the function. Putting it more formally, you might say: "For all numbers x, the square root of xx x should equal x." To test this claim, all you need to do is restate it as a LectroTest property:

# loads and activates LectroTest use Test::LectroTest; Property {     ##[ x <- Float ]##                   # first  part     sqrt( $x * $x ) == $x;               # second part }, name => "sqrt is inverse of square";  # third  part

This is a complete Perl program that you can run. It tests a single property, specified in three parts. The first part (in the funny brackets) specifies the domain over which the property's claim should hold. Read it as saying, "For all floating-point numbers x..."

LectroTest actually offers a tiny language for declaring more complex domains, but it's not necessary here.

The second part is a snippet of Perl code that checks the property's claim for a given value of x. If the claim holds, the code must return a true value; otherwise, it must return a false value. In this property, read the code as saying, "...the square root of xx x should equal x." For convenience, LectroTest makes the variables mentioned in the first part of the property available in the second part as lexical variables$x, here.

Because of the imperfections of floating-point arithmetic, a more robust way of testing this claim would be to check whether the difference between $x and sqrt($x * $x) is within an acceptably small range. For simplicity, however, I've used a straight equality test, which could result in a false negative test result.

The third part gives the property a name. It's optional but adds a lot of documentation value, so give your properties meaningful names.

Running the Hack

To test whether your claims hold, just execute the program that contains your properties. In this case, you have only one property, so the program's output might look like:

1..1 not ok 1 - 'sqrt is inverse of square' falsified in 3 attempts # Counterexample: # $x = "-1.61625080606365";

Here, LectroTest says that, for some value of x, it was able to prove your property's claim false. It provides the troublesome value of x as a counterexample that you can use to figure out what went wrong.

Refining your claims

In this case, what went wrong is that the property made an overly broad claim. The square root function only applies to non-negative numbers (ignore imaginary numbers for this hack), and yet the property made its claim about all floating-point numbers, which includes those less than zero.

This illustrates an important benefit of the specification-based approach to testing: because it forces you to make your claims explicit, it can reveal hidden assumptions and holes in your thinking. Now you must consider what sqrt should do when given a negative number. For Perl, it probably ought to result in an error, so you might revise your property:

# helper:  returns true if calling the given function # results in an error; returns false otherwise sub gives_error(&) {     ! eval { shift->( ) } and $@ ne ""; } Property {     ##[ x <- Float ]##     $x < 0 ? gives_error { sqrt( $x ) }            : sqrt( $x * $x ) = = $x }, name => "sqrt is inverse of square and dies on negatives";

You also could make sqrt's split personality more obvious by writing two properties that together define its behavior:

Property {     ##[ x <- Float ]##     $tcon->retry( ) if $x < 0;      # only test non-negatives     sqrt( $x * $x ) = = $x; }, name => "sqrt is inverse of square"; Property {     ##[ x <- Float ]##     $tcon->retry( ) unless $x < 0;  # only test negatives     gives_error { sqrt( $x ) }; }, name => "sqrt of negative numbers gives error";

Calling $tcon->retry( ) tells LectroTest to retry a test case you don't like, starting over with a brand new, random case. Use this call in your properties to subject only sensible cases to your tests. In the first property, for instance, the conditional call to $tcon->retry( ) ensures that LectroTest subjects only non-negative values of x to the sqrt-is-the-inverse-of-squaring test.

The LectroTest-provided $tcon object lets you talk to the underlying test controller to do all sorts of interesting things. See the LectroTest documentation to learn more.

You now have two properties. The first claims, "For all non-negative floating-point numbers x, the square root of xx x should equal x." The second claims, "For all negative floating-point numbers x, attempting to take the square root of x should result in an error." With practice, it becomes easy to convert LectroTest property specifications into written claims and vice versa. These two claims seem to cover all of the bases, and so it's time to put them to the test.

Interpreting test output

When you run the two-property suite, you get the results:

1..2 ok 1 - 'sqrt is inverse of square' (1000 attempts) ok 2 - 'sqrt of negative numbers gives error' (1000 attempts)

Good news! LectroTest subjected each claim to 1,000 random test cases and was unable to find any problems. Still, this happy outcome doesn't prove that sqrt works as expected. It merely gives you evidence to support that conclusion. Certainly, the evidence is persuasiveyou would think that 2,000 tests ought to be enough to flush out any obvious problemsbut it's important not to lose sight of the limitations of testing.

In light of the evidence, though, there is probably no need to test further. The existing results argue persuasively in favor of sqrt, and there's no reason to think there are special circumstances that might make the current degree of testing inadequate. The only corner case in sight is the negative-number case, and you have that covered. At this point, you can probably rest satisfied that sqrt does the right thing.

Taking advantage of specification-based testing

With the specification-based approach, a little testing code goes a long way. It only took about fifteen lines of code to test sqrt fairly rigorously.

Another strength of the approach is that the claims embodied in your testing code are easy to seejust read the properties. These claims are useful beyond their testing value, serving as formal documentation of what you expect your software to do.

Because of these strengths, specification-based tests make a great complement to hand-written tests. When one approach to testing seems difficult, the other is often easy, and combining the approaches makes many complicated testing problems easier. For this reason, you ought to have both approaches in your toolbox.

Specification-based testing is a deep and interesting topic, and this hack only scratches its surface. To learn more, including how to use it with more-traditional Perl testing tools, the documentation for Test::LectroTest is a good starting point.