How Google Searches the Internet

Web masters who don't want their sites to be searchable via Google can instruct Google not to index their sites. To do it, they create a text file called robots.txt containing only these two lines and put it in the root directory:

 User-agent: * Disallow / 

That tells all search engines, not just Googlebot, to stay away. They can also tell Googlebot or other search engines to not search their site by putting this HTML tag into the <head> section of the HTML for their web page:

 <META name="ROBOTS" content="NOINDEX, NOFOLLOW" /> 

How Personal & Internet Security Work
How Personal & Internet Security Works
ISBN: 0789735539
EAN: 2147483647
Year: 2004
Pages: 161

Similar book on Amazon © 2008-2017.
If you may any questions please contact us: