Recipe 20.11 Creating a Robot

20.11.1 Problem

You want to create a script that navigates the Web on its own (i.e., a robot), and you'd like to respect the remote sites' wishes.

20.11.2 Solution

Instead of writing your robot with LWP::UserAgent, use LWP::RobotUA instead:

use LWP::RobotUA; $ua = LWP::RobotUA->new('websnuffler/0.1', 'me@wherever.com');

20.11.3 Discussion

To avoid marauding robots and web crawlers hammering their servers, sites are encouraged to create a file with access rules called robots.txt. If you're fetching only one document, this is no big deal, but if your script fetches many documents from the same server, you could easily exhaust that site's bandwidth.

When writing scripts to run around the Web, it's important to be a good net citizen: don't request documents from the same server too often, and heed the advisory access rules in their robots.txt file.

The easiest way to handle this is to use the LWP::RobotUA module instead of LWP::UserAgent to create agents. This agent automatically knows to fetch data slowly when calling the same server repeatedly. It also checks each site's robots.txt file to see whether you're trying to grab a file that is off-limits. If you do, you'll get a response like this:

403 (Forbidden) Forbidden by robots.txt

Here's an example robots.txt file, fetched using the GET program that comes with the LWP module suite:

% GET http://www.webtechniques.com/robots.txt  User-agent: *      Disallow: /stats      Disallow: /db      Disallow: /logs      Disallow: /store      Disallow: /forms      Disallow: /gifs      Disallow: /wais-src      Disallow: /scripts      Disallow: /config

A more interesting and extensive example is at http://www.cnn.com/robots.txt. This file is so big, they even keep it under RCS control!

% GET http://www.cnn.com/robots.txt | head # robots, scram # $I d : robots.txt,v 1.2 1998/03/10 18:27:01 mreed Exp $ User-agent: * Disallow: / User-agent:     Mozilla/3.01 (hotwired-test/0.1) Disallow:   /cgi-bin Disallow:   /TRANSCRIPTS Disallow:   /development

20.11.4 See Also

The documentation for the CPAN module LWP::RobotUA(3); http://info.webcrawler.com/mak/projects/robots/robots.html for a description of how well-behaved robots act



Perl Cookbook
Perl Cookbook, Second Edition
ISBN: 0596003137
EAN: 2147483647
Year: 2003
Pages: 501

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net