Hack 65. Test Live Code

Verify your code against actual use...with little penalty.

Perl culture widely acknowledges automated testing as one step in verifying quality. Most CPAN modules and many large applications, not to mention Perl distributions themselves, have comprehensive test suites that run before installation to show what works and, occasionally, what doesn't work.

In theory, that's enough. In practice, it can be difficult to predict exactly how your code will react to production systems, live customers, and their actual data. Testing against this scenario would be incredibly valuable, but it's much more complexnot to mention probably much slower. If your automated tests are effective, they'll match the behavior of most customer requests.

Fortunately, it's possible to embed tests in production code that test against live data, non-destructively, but that don't generate too much data nor slow down your system.

The Hack

Imagine you have a web application that allows employees to manage their user data stored in a backend LDAP database. Abstraction is the key to a good system. You've created a User object and you've tested that the system vets and verifies all sorts of names and addresses that you could think of. You don't know how the system will react to the messy real world, though.

Instead of hard-coding the creation of the User object, create a factory object that returns a User object or equivalent.

package UserFactory; use User; use UserProxy; my $count    = 0; sub create {    my $self  = shift;    my $class = $count++ % 100 ? 'User' : 'UserProxy';    return $class->new( id => $count, @_ ); } 1;

Every hundred requests, the factory will create a UserProxy object instead of a User object. As long as UserProxy implements the same interface as User (and actually behaves similarly), the rest of the code should see no difference. UserProxy is a bit more complex:

package UserProxy; use strict; use warnings; use User; use Test::Builder; sub new {     my ($class, %args) = @_;     my $proxied        = User->new( %args );     my $Test           = Test::Builder->create( );     $Test->output( time( ) . '_' . $proxied->id( ) . '.tap' );     $Test->plan( 'no_plan' );     bless { proxied => $proxied, test => $Test }, $class; } sub proxied {     my $self = shift;     return $self->{proxied}; } sub test {     my $self = shift;     return $self->{test}; } sub can {     my ($self, $method) = @_;     my $proxied         = $self->proxied( );     return $proxied->can( $method ); } sub verify_name {     my ($self, $name)   = @_;     my $proxied         = $self->proxied( ):     my $test            = $self->test( );     $test->ok( $proxied->verify_name( $name ), "verify_name( ) for '$name'" )         || $test->diag( $proxied->verification_error( ) ); } # ... 1;

When UserFactory creates a UserProxy, the proxy class creates an actual User object with the same arguments. It also wraps some calls to normal User methods with its own methods to run tests. The example verify_name( ) method wraps the call to User->verify_name( ) in a test case, expecting it to pass but logging it with a test diagnostic containing the error if it does not.

Apart from the factory/proxy design, the other part that makes this hack work is the use of Test::Builder in UserProxy->new( ). Each UserProxy object contains its own Test::Builder object and sends its output to a unique file with a name based on the current time and the number of the request. From there, use a cron job to run prove or some other Test::Harness-related program to analyze the tests and notify the proper people if things fail.

Be sure to use Test::Builder 0.60 or newer to have access to create( ).

Hacking the Hack

The more data you can keep about failing requests, the betteryou can use this data in your automated tests as you add test cases and fix the bugs.

Test::Builder provides only a few methods. You may want to write your own test library atop Test::Builder based on your needs. An alternate approach is to use Test::Class within your proxy. Though you need to find some way to manage the test output and counter on a per-object basis, the module will handle much of the setup code for you. It will also be very valuable if you want to test multiple types of objects that inherit from common ancestors.

Sampling one out of every hundred requests may be the wrong frequency. There's a whole field of statistical analysis devoted to sample and defect rates. This is a decent place to start, and it's likely an improvement over only automated testing, but it's not perfect for every need.

It may be worth serializing the proxied object and storing it somewhere useful in case of failure. Whether you use Storable or YAML or just save the relevant data somewhere else, having the exact information available to recreate the appropriate customer request will aid debugging.