Hack 17 Respecting robots.txt

The robots.txt file is a bastion of fair play, allowing a site to restrict what visiting scrapers are allowed to see and do or, indeed, keep them out entirely. Play fair by respecting their requests .

If you've ever built your own web site, you may have come across something called a robots.txt file (http://www.robotstxt.org)a magical bit of text that you, as web developer and site owner, can create to control the capabilities of third-party robots, agents , scrapers, spiders, or what have you. Here is an example of a robots.txt file that blocks any robot's access to three specific directories:

 User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /private/ 

Applications that understood your robots.txt file will resolutely abstain from indexing those parts of your site, or they'll leave dejectedly if you deny them outright , as per this example:

 User-agent: * Disallow: / 

If you're planning on releasing your scraper or spider into the wild, it's important that you make every possible attempt to support robots.txt . Its power comes solely from the number of clients that choose to respect it. Thankfully, with LWP , we can rise to the occasion quite simply.

If you want to make sure that your LWP -based program respects robots.txt , you can use the LWP::RobotUA class (http://search.cpan.org/author/GAAS/libwww-perl/lib/LWP/RobotUA.pm) instead of LWP::UserAgent . Doing so also ensures that your script doesn't make requests too many times a second, saturating the site's bandwidth unnecessarily. LWP::RobotUA is just like LWP::UserAgent , and you can use it like so:

 use LWP::RobotUA; # Your bot's name and your email address my $browser = LWP::RobotUA->new('SuperBot/1.34', '[e-mail]'); my $response = $browser->get($url); 

If the robots.txt file on $url 's server forbids you from accessing $url , then the $browser object ( assuming it's of the class LWP::RobotUA ) won't actually request it, but instead will give you back (in $response ) a 403 error with a message "Forbidden by robots.txt." Trap such an eventuality like so:

 die "$url -- ", $response->status_line, "\nAborted"     unless $response->is_success; 

Upon encountering such a resource, your script would die with:

 http://whatever.site.int/pith/x.html -- 403 Forbidden by robots.txt 

If this $browser object sees that the last time it talked to $url 's server was too recently, it will pause (via sleep ) to avoid making too many requests too often. By default, it will pause for one minute, but you can control the length of the pause with the $browser->delay( minutes ) attribute.

For example, $browser->delay(7/60) means that this browser will pause when it needs to avoid talking to any given server more than once every seven seconds.

Sean Burke