Hack 69. Simulate Hostile Environments in Your Tests

Test devastating failures with aplomb.

When you publish a CPAN module that depends on other modules, you list the prerequisite modules in your Makefile.PL or Build.PL script.

Using Build.PL:

my $builder = Module::Build->new(   # ... other Build.PL options ...   requires =>   {       'Test::More'       => 0,       'CGI'              => 2.0,   } );

Using Makefile.PL:

WriteMakefile(     # ... other Makefile.PL options ...     'PREREQ_PM' =>     {         'Test::More'     => 0,         'CGI'            => 2.0,     } );

However, there are a few ways that this standard prerequisite checking can be insufficient. First, you may have optional prerequisites. For instance, your module will use Foo::Bar if it happens to be installed, but should fail gracefully when Foo::Bar is absent.

Second, if the behavior of a module changed between two versions, you may still want to support both versions. For example, CGI changed how it handles PATH_INFO in version 3.11. Your CGI::Super_Path_Info module probably wants to be compatible with both CGI version 3.11 and also with earlier (and later) versions.

Finally, occasionally a user will install your module by hand to bypass the prerequisite check, hoping to use an older version of Foo::Bar than the one you require. Sometimes your module works fine (maybe with some feature limitations), but your test suite breaks because your tests assumed the presence of a new feature.

For each of these cases. you can make your module and tests more robust. For example, you can skip tests that are incompatible with an older version of a module:

use Test::More; use CGI; if ($CGI->VERSION >= 3.11) {     plan skip_all => 'skipping compatibility tests for old CGI.pm'; } else {     plan 'tests' => 17; }

You can also skip tests that require the presence of a particular optional module:

eval 'require URI;'; if ($@) {     plan skip_all => 'optional module URI not installed'; } else {     plan 'tests' => 10; }

Now your tests are (hopefully) more robust, but how do you make sure that they will actually work on a system that is missing some modules and has older versions of others?

Ideally, you want to run your test suite against a few different sets of installed modules. Each set will be different from what you have installed in the main Perl site_lib of your development machine. It's way too much work to uninstall and reinstall ten different modules every time you make a new CPAN release.

The Hack

There are three possibilities: the user has an old version of the module installed, the user does not have the module installed, and the user has some combination of both for multiple modules.

Simulating old versions of modules

Create custom Perl library directories and include these directories when you use prove to run your tests. For instance, to run the tests against an old version of CGI:

$ mkdir t/prereq_lib $ mkdir t/prereq_lib/CGI $ cp CGI-3.10.pm t/prereq_lib/CGI.pm $ prove -Ilib -It/prereq_lib t/                

Including t/prereq_lib on the command line to prove puts at the start of @INC, so Perl will load any modules you put in this directory before modules installed in your system's Perl lib directories.

Simulating missing modules

That works for older versions of modules, but how do you install the absence of a module in a custom library directory so that it takes precedence over a copy already installed on your system?

The solution is to create a zero-length file with the same name as the module. This works because in order for a module to load successfully (via require or use) it has to end in a true value, such as (from actual CPAN modules):^[1]

^[1] The more boring the line of code, the better the opportunity for creativity.

1; 666; "false"; "Steve Peters, Master Of True Value Finding, was here.";

A zero-length file doesn't end in a true value, and consequently require fails. It doesn't fail with the same error message as a missing module fails with, but it still fails.

For example, to run the tests in an environment missing URI:

$ mkdir -p t/skip_lib $ touch t/skip_lib/URI.pm $ prove -Ilib -It/skip_lib t/                

Running multiple scenarios

You can create multiple different library directories, each containing a different combination of missing and/or old modules:

$ mkdir -p t/prereq_scenarios/missing_uri $ touch t/prereq_scenarios/missing_uri/URI.pm $ mkdir -p t/prereq_scenarios/old_cgi $ cp CGI-3.10.pm t/prereq_scenarios/old_cgi/CGI.pm $ mkdir -p t/prereq_scenarios/new_cgi $ cp CGI-3.15.pm t/prereq_scenarios/new_cgi/CGI.pm                

Then run all of these scenarios at once:

$ for lib in t/prereq_scenarios/*; do prove -Ilib -I$lib t/; done                

However this one-liner stops at the first error and doesn't provide any summary information. Here's a more complete version:

#!/usr/bin/perl use strict; use File::Find; if (@ARGV < 2) {     die "Usage: $0 [prereq_scenarios_dir] [args to prove]\\n"; } my $scenarios_dir = shift; my %scenario_modules; my $errors; my @scenarios     = grep { -d } <$scenarios_dir/*>; for my $lib_dir (@scenarios) {     unless (-d $lib_dir)     {         $errors   = 1;         warn "lib dir does not exist: $lib_dir\\n";         next;     }     my @modules;     find(sub     {         return unless -f;         my $dir =  "$File::Find::dir/$_";         $dir    =~ s/^\\Q$lib_dir\\E//;         $dir    =~ s/\\.pm$//;         $dir    =~ s{^/}{ };         $dir    =~ s{/}{::}g;         push @modules, $dir;     }, $lib_dir);     $scenario_modules{$lib_dir} = \\@modules; } die "Terminating." if $errors; for my $lib_dir (@scenarios) {     my $modules   = join ', ', sort @{ $scenario_modules{$lib_dir} };     $modules    ||= 'none';     print "\\n" . '#' x 62 . "\\n";     print "Running tests.  Old (or absent) modules in this scenario:\\n";     print "$modules\\n";     my @prove_command = ('prove', "-I$lib_dir", @ARGV);     system( @prove_command ) && do     {         die <<EOF; ############################################################## One or more tests failed in scenario $lib_dir. The old or absent modules were:     $modules The command was:     @prove_command Terminating. ############################################################## EOF     }; }

Save this as prove-prereqs and run it as:

$ prove-prereqs t/prereq_scenarios -Ilib t/                

Hacking the Hack

PITA, the Perl Image Testing Architecture (http://search.cpan.org/dist/PITA) project, goes much further than this hack does. PITA will be able to test your Perl modules under different versions of Perl and even on different operating systems (possibly running within virtual machines on a single computer). It will allow you to automate the testing process and collect the results generated from several testing environments.