Site Indexes and Search Engines

After you list your new site on the major directories and maybe a few smaller directories, you need to turn your attention to the indexing and search tools. The following are the most popular of these sites:

Google	www.google.com
Yahoo	www.yahoo.com
MSN Search	search.msn.com
Ask Jeeves	www.ask.com

Unlike web directories, which contain a hierarchical list of websites that have been submitted for inclusion, these indexes have search engines (sometimes called spiders) that prowl the Web and store information about every page and site they find. Users can then go to the sites and search for things using a full text search. In other words, whereas directories store the name and description of sites, search engines index the full contents of sites so that pages containing specific text can be found. After you publish your site on the Web and other people link to it, chances are that at some point a search engine will find it and index it. However, you don't have to wait. You can tell these indexes about your site and get it included on the list of sites to be indexed. Each of these search engines enables you to submit your site for indexing.

In the text that follows, I'm going to describe how to submit your sites to some of the popular search engines. There's a set of interlocking relationships among search engine providers that can make it difficult to keep track of who is providing search functionality for whom. The search engines listed previously maintain their own indexes. Once your site is included in their index, it will be available via all the search engines that use their index as well.

Note

The search engine descriptions are very brief, and due to the rapidly changing nature of the search engine industry, may be out of date by the time you read them. For more information on search engines, I strongly recommend Search Engine Watch at www.searchenginewatch.com.

Google

Google is currently the most popular search engine. Its search results are ranked based not only on how frequently the search terms appear on a listed page, but also on the number of other pages in the index that link to that page. So, a popular page with thousands of incoming links will be ranked higher than a page that has only a few incoming links.

This search algorithm does a remarkably good job of pushing the most relevant sites to the top of the search results. It also rewards people who publish useful, popular sites rather than those who've figured out how to manipulate the algorithms that other search engines use. Many sites that aren't dedicated to providing search functionality use Google's index, so getting into the Google index provides wide exposure. Statistics from November 2005 show that Google searches make up about half of the search engine market.

How Search Results Are Ranked

Every search engine has an algorithm that ranks sites based on their relevance. It might take into account how many times the keyword you enter appears on the page, whether it appears inside heading tags or in the page title, or whether it appears in text inside links. It might also take into account how high on the page your search terms appear. Such algorithms are trade secrets within the search engine industry, but some of them have been unraveled to greater or lesser degrees. Armed with this information, some site authors write their pages in such a way that search engines will give them a higher relevance ranking than they deserve. For example, some sites have really long titles with lots of information in hopes of appearing first in search results.

Yahoo!

Yahoo! has been around since 1994. They provide both a human-edited directory of the Web, which I discussed earlier, and web search. Yahoo! has leased their web index from other companies, including Google, in the past, but they currently maintain their own index. Yahoo!'s index includes not only HTML documents, but also PDF and Microsoft Office documents. Yahoo!'s search engine market share is about 25% as of November 2005.

MSN Search

MSN Search is Microsoft's web search offering. Like Yahoo! and Google, Microsoft maintains its own index of the Web. Like Google, MSN Search includes incoming links to your site to determine how to rank it. As of November 2005, searches at MSN Search make up about 10% of the market.

Ask.com

Ask.com is another search engine that maintains its own index. Ask.com indexes fewer pages than Google or Yahoo!. Ask.com was formerly known as Ask Jeeves and was notable for allowing users to submit their searches in the form of questions. Its basic approach is now similar to other search engines. Ask.com only controls about 2% of the search engine market as of November 2005, but is listed because like the other sites listed previously, it has its own index.