Web masters who don't want their sites to be searchable via Google can instruct Google not to index their sites. To do it, they create a text file called robots.txt containing only these two lines and put it in the root directory:
User-agent: * Disallow /
That tells all search engines, not just Googlebot, to stay away. They can also tell Googlebot or other search engines to not search their site by putting this HTML tag into the <head> section of the HTML for their web page:
