Google is working on some interesting tools for getting your site indexed as fully as possible. First, they have developed the sitemap protocol, which is an XML application that lists all the URLs on your site, as well as when they were last updated and how important they are with respect to the site as a whole.
Second, they have created the Sitemap Generator, an Open Source Python program that you can run on your site to create a site map in Google's sitemap format from a list of your site's directories, from your site's access logs, or even from a simple text-format listing of the URLs on your site.
Finally, Google offers some very helpful diagnostic and statistical tools called Google Sitemaps. They can tell you if they're having trouble analyzing your site, the average rank of your twenty highest ranked pages, and much more.
And it's all free.
Figure 24.14. Google's Sitemap Generator creates an XML file with the URL, last modification date and relative importance of each of the pages on your site. It can generate the file from a list of directories, your access logs, or even a text file you've written manually.