Hack 20. Read Files Backwards

Process the most recent lines of a file first.

Perl's position in system administration is stable and secure, due in no small part to its fast and flexible text-processing abilities. If you need to slice and dice log files, monitor services, and send out messages, you could glue together the perfect combination of shell and command-line utilities, or you could have Perl do it.

Of course, Perl is a general-purpose language and doesn't always provide every tool you might need by default. For example, if you find yourself processing system logs often, you might wish for a way to read files in reverse order, the most recent line first. Sure, you could slurp all of the lines into an array and read the last one, but on a busy system with lots of huge logs, that can be slow and memory-consuming.

Fortunately, there's more than one way to do it.

The Hack

Yes, you could look up perldoc -F -X and find a file's size and read backwards until you find the appropriate newline and then read forward...but just install File::ReadBackwards from the CPAN instead.

Suppose you have a server process that continually writes its status to a file. You only care about its current status (at least for now), not its historical data. If its status is up, everything is happy. If its status is down, you need to panic and notify everyone, especially if it's 3 a.m. on Boxing Day.

Simulate the program that writes its logs with:

#!/usr/bin/perl use strict; use warnings; use Time::HiRes 'sleep'; local $| = 1; for ( 1 .. 10000 ) {     my $status = $_ % 10 ? 'up' : 'down';     print "$status\\n";     sleep( 0.1 ); }

This program writes a status message to STDOUT ten times a second; nine of those are up and the last is down. Run it and redirect it to a file such as status.log. In bash, this is:

$ perl write_fake_log.pl > status.log &             

Running the Hack

With status.log continually growing with newer information, finding the most recent status is easy with File::ReadBackwards:

use FileReadBackwards; my $bw = File::ReadBackwards->new( 'status.log' )     or die "Cannot read 'status.log': $!\\n"; exit( 0 ) if $bw->readline( ) =~ /up/; # panic( ) ...

The program is straightforward. Create a File::ReadBackwards object by passing the name of the file you want to read. Then, every time you call readline( ) on the object, you'll receive the previous line of the file, starting with the last line and working backwards to the first line.

Hacking the Hack

Note that the current version of the module (1.04) does not flock the file it reads, so you may read a partial line. Also, you may get a partial line, depending on how large your filesystem's buffers are and how much the process has written to it. If either is important to you, the source code is short, full of good comments, and easy to modifybut if you just need the last n lines of a file, this is the easy way.