For More Information

9.7 For More Information

For more information on web clients, refer to:

http://www.robotstxt.org/wc/robots.html

The Web Robots Pagesresources for robot developers, including the registry of Internet Robots.

http://www.searchengineworld.com

Search Engine Worldresources for search engines and robots.

http://www.searchtools.com

Search Tools for Web Sites and Intranetsresources for search tools and robots.

http://search.cpan.org/doc/ILYAZ/perl_ste/WWW/RobotRules.pm

RobotRules Perl source.

http://www.conman.org/people/spc/robots2.html

An Extended Standard for Robot Exclusion.

Managing Gigabytes: Compressing and Indexing Documents and Images

Witten, I., Moffat, A., and Bell, T., Morgan Kaufmann.