Section 18.5. What to Test

18.5. What to Test

Test both the likely and the unlikely.

Having a test suite that fails to fail might not be a major problem, so long as your tests cover the most common ways in which your software will actually be used. The single most important practice here is to run your tests on real-world cases.

That is, if you're building software to handle particular datasets or data streams, test it using actual samples of that data. And make sure those samples are of a similar size to the data on which the software will eventually need to operate.

Play-testing (see Chapter 17) can also come in handy here. If you (or other prospective users) have prototyped the kind of code you expect to write, then you should test the kinds of activities that your exploratory code implements, and the kinds of errors that you made when writing that code. Better yet, just write your hypothetical code as a test suite, using one of the Test:: modules. Then, when you're ready to implement, your test suite will already be in place.

Testing the most likely uses of your software is essential, but it's also vital to write tests that examine both edge-cases (i.e., one parameter with an extreme or unusual value) and corner-cases (i.e., several parameters with an extreme or unusual value).

Good places to hunt for bad behaviour include:

• The minimum and maximum possible values

• Slightly less than the minimum possible value and slightly more than the maximum possible value

• Negative values, positive values, and zero

• Very small positive and negative values

• Empty strings and multiline strings

• Strings with control characters (including "\0")

• Strings with non-ASCII characters (e.g., Latin-1 or Unicode)

• undef, and lists of undef

• '0', '0E0', '0.0', and '0 but true'

• Empty lists, arrays, and hashes

• Lists with duplicated and triplicated values

• Input values that "will never be entered" (but which are)

• Interactions with resources that "will never be missing" (but which are)

• Non-numeric input where a number is expected, and vice versa

• Non-references where a reference is expected, and vice versa

• Missing arguments to a subroutine or method

• Extra arguments to a subroutine or method

• Positional arguments that are out of order

• Key/value arguments that are mislabeled

• Loading the wrong version of a module, where multiple versions are installed on your system

• Every bug you ever actually encounter (see the following guideline, "Debugging and Testing")