What If Your Site Is Not Indexed?

If your business has been around for a while and has a Web site, your site is overwhelmingly likely to be indexed by all the leading search engines. It is rare for corporate sites to be completely missing from search indexes, although it is quite common for individual pages from the site to be missing. (We show you how to figure that out later in this chapter.)

One way to tell whether your site is indexed is to search for it and see whether it is found. (Yeah, we figured you thought of that one.) If your company has a common name (AAA Plumbing), you might want to search for more than just the name ("aaa plumbing syracuse"). It's common for site owners to panic when their sites are not shown by the search engine for these navigational queries for company names. It's easy to jump to the conclusion that the entire site is not indexed, but that is rarely the case.

You can also use a search toolbar in your browser to check to see whether your pages are indexed. If you use the Google toolbar (or one for another search engine), you can navigate to your home page and take a look at the toolbarmost toolbars indicate that the page is indexed in some way. Figure 10-1 shows how Google's toolbar does it. You can see whether your site is indexed by that toolbar's search engine, although it does not help you figure out whether your page is indexed by other search engines. (We show you how to do that later in the chapter.)

Figure 10-1. Toolbars show indexed pages. Google's toolbar indicates the current page in the browser is indexed with the green PageRank bar.

Nearly all corporate sites have their home pages (and at least a few other pages) in the leading search indexes, but if somehow you do not, read on. It is unlikely but possible that your site is missing in actionit has happened, even to relatively large companies. Or perhaps your search marketing program's scope does not cover your whole company, but all the pages within your scope are missing from search engines. In that case, you need to ask a few questions:

  • Is your site banned by the search engines? Search engines have very specific rules for being included in their index. Sites that violate these rules might find all of their pages removed from the search index.

  • Is the spider visiting your site? Your pages cannot be indexed if the spider never comes. Check to make sure.

  • Are other sites linking to yours? Spiders find your site by following links from other sites, so you must verify that your site is linked into the larger Web.

If you can find at least a few pages of your site when you perform searches in the major search engines, you can skip ahead to the next section to determine how many pages you have indexed. If your site is not found at all, however, you can explore these questions to solve the mystery of the missing site.

Verify Your Site Is Not Banned or Penalized

The most difficult situation occurs when one or more search engines have banned or penalized your site. If your site is well represented in some search engines, but completely missing in others, your site might be banned. Sites are banned when search engines detect that those sites are trying to "fool" the search engine to rank that site's pages more highly than they deserve.

Search engines do not ban sites on a whimthey do so only when your site has persistently violated their rules. As you might expect, some search marketers are unscrupulous, trying to take unfair advantage with tricks that provide an edge. Your company might have unwittingly become involved with an unethical search marketing consultant who uses spam techniques that try to fool the search engine, or you might have unwittingly violated a search engine's guidelines. (You can look at Google's guidelines at http://www.google.com/webmasters/guidelines.html; other search engines have similar rules.)

If your violation of the rules is not severe enough for a search engine to ban your site, you still might suffer from penalizationwhen the search engine starts dropping large numbers of your pages from its index or begins to lower your search rankings. You should make sure that signs of penalization are checked regularly (we explain how in Chapter 15, "Make Search Marketing Operational"), and you should investigate further if you see the following:

  • Your site's search engine referrals have dropped drastically in a short period of time.

  • The number of your site's pages included in the search index rapidly decreases.

  • The search engine shows fewer and fewer links to your site each month, maybe decreasing to zero. A simple way to check this for Google's index is to use the Google toolbaryou will see a gray PageRank bar when you browse your Web pages. In Chapter 13, "Attract Links to Your Site," we show you easy ways to check the number of links to your site that are stored by each search engine.

  • Your home page can be found only by a direct search on the URLinformational queries for words on the page do not seem to work anymore.

If you suspect a problem, you first need to diagnose the cause. In the next section, we discuss a spam technique called cloaking. We cover doorway pages and other stupid content tricks in Chapter 12, "Optimize Your Content," and link farms in Chapter 13, "Attract Links to Your Site." These are the most common spam techniques. If your site has been banned or penalized for using these techniques, you can clean up your site and request reinstatement, which is usually granted (although reinstatement sometimes requires an extended period of explanation and begging).

Make Sure the Spider Is Visiting

If the spiders are not coming to your site, your pages cannot be indexed and your site will not be found by organic searchers. Your Webmaster can help you check your Web servers' log files to see which search spiders have been visiting your site. (Most Web servers are configured to log search spider visits, but some servers might need to be adjusted by your Webmaster to capture this important information.) Figure 10-2 shows an extract of a log file showing that a spider has visited. The log file indicates the name of the user agent that accessed the page noted, which indicates the software that was used to see the page.

Figure 10-2. Spotting spider activity. Carefully examining your log files can prove that spiders are visiting your site.

Most of the visits to your site generate a log listing of something like "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)," which indicates that the Internet Explorer Web browser (version 5.5) was used to access your site. As you might expect, the vast majority of the user agents listed in your log file are from Web browsers. But a small number of log entries might show you that the spiders are crawling. Googlebot, as you might surmise, is Google's user agent. Yahoo! aptly named its spider Slurp. Table 10-1 shows some the names of some of the leading spider's agents, but there are many more, and they change their names frequently.

Table 10-1. Spider User Agent Names (Each search engine sends a spider to your site that you can spot by its agent name in your Web server logs.)

Search Engine

Spider Agent Name



Ask Jeeves








By examining your log, you can tell which spiders are crawling your site, and how frequently they do so. Spammers use these user agent names for more nefarious purposes, however. Variously known as cloaking or IP delivery, they use a high-tech version of the old "bait-and switch" scam. Here's how it works. The spammer sets up a URL served by a program dynamically and waits for someone to request it. When the request is received, the user agent name and its IP address are checked. If a browser is making the request (with Mozilla in the name, for example), the program returns the page that human visitors should see. If it is a search engine's spider, however, the program sends back a page full of keywords designed to attain a high search ranking. Later in this chapter, we discuss situations where you can legitimately use IP delivery techniques, but using this technique to fool a search engine about what visitors see on the page is clearly spam and the search engines will deal with it harshly. Unless you know that what you are doing is acceptable, cloaking is a very dangerous game that can get your site banned. Cloaking can bring quick rankings, but when competitors see what you are doing, they will complain to the search engines and shut you down.

Each search engine spider has its own frequency for returning to your Web site. Spiders return to a typical Web site at least once a month, but popular corporate Web sites might be revisited weekly, or even daily. By analyzing how frequently spiders crawl your pages, and which pages they check the most, you will know how quickly content changes on your site will be reflected in search indexes.

Although it is rare to find a site that the spiders do not visit at all, it can happen for a few reasons:

  • Your site is not linked. If you have a brand new site that is not linked to by any other site in the spider's path, you will not get any spider visits. If the search engine spider does not know your site exists, it obviously cannot visit.

  • Links to your site are not effective. Some links cannot be followed by spiders, for many reasons that we get into later in this chapter. Or the links to your site are from sites that themselves are not crawled by spiders, perhaps because they are also new, or possibly because they are banned for using unethical techniques. You can use the Google toolbar to check the PageRank of sites that link to yoursthat shows you how valuable a link you are getting. If your linker has a zero PageRank, it is not doing you any good in Google because that site is not indexed.

  • The spider has given up. Perhaps the spider was visiting your site at one time, but your site was blocked so that the spider could not index any pages in the search index. After a few months of fruitless visits, spiders sometimes permanently stop visiting.

If your site truly is not being visited by spiders, the remedy depends on which of the above reasons is the cause. If your site is not linked or links to your site are not effective, the best way to get the spider to visit is to make sure other well-respected sites link to yours (explained in more detail later). If the spider has given up, first remove the spider trap (also explained later), and then you should manually submit your site to the search engines.

Search engines vastly prefer to find new sites by following links because analyzing link patterns is one of the ways engines judge relevance; if your site is linked and spiders are not visiting, however, you should manually submit your home page's URL. (If you create a new site but are too impatient to wait until someone links to you, you can also submit, but waiting to be found from a link will help your pages rank higher.)

Every major search engine, except Ask Jeeves, has a way to submit your site, but not all of them are free. Search engines that offer free submission often refer to their "Add URL" page as the place you should goat Google it's at www.google.com/addurl (with others listed in Table 10-2). Remember, if your site is missing from multiple search indexes, you need to submit to each search engine. Typically, pages submitted are included in the search index within a few weeksyou can check your log files to see whether the spider is visiting.

Table 10-2. Submitting URLs (Each search engine allows you to request inclusion of your sitesome charge nothing, whereas others have paid inclusion programs.)

Search Engine

Free or Fee?














Submitting to local or country-specific search engines is no different from worldwide search engines. The spiders can detect the language of the site and will add it into the appropriate-language version of the index, which we cover in detail in Chapter 12.


Before you get excited about submitting all your Web pages to the search engines, a word of caution is in order. Many "experts" will advise you to submit your site early and often, and to submit many pages from your site. Don't. It is more complicated than that.

Submitting your site should be a last resort because you have tried everything else without successyou cannot get anyone to link to your site, or your site has been blocked so long that the spider has given up. If you are forced to submit, submit only your home page. Your site map should help the spider find everything else, so make sure your map is comprehensive. Often, people submit their sites over and over without ever checking to see whether the spider is blocked from their site. Before submitting, follow the advice in this chapter first. If you do, you will probably see that you do not need to submit after all.

Even worse than manually submitting your site is automatic submission. You might have received unsolicited e-mails offering to submit your site to thousands of search engines for $19.95. The e-mail might include some compelling statistics about millions of visitors this service will generate. Avoid "automated submissions" entirely. If you are lucky, these services just set you back $20 and do no harm. But some search engines consider these automated submissions to be spam and many actually have measures in place to block such submissions. At best, these submissions are ignored, but at worst your site could be penalized or banned for excessive submissions.

A surefire way to get your pages included is to pay a search engine to put them in the index. As discussed in Chapter 3, "How Search Marketing Works," paid inclusion not only guarantees to keep your pages in the index, it also promises that they will be revisited by the spider regularly. Only Yahoo! (of the major worldwide engines) is offering paid inclusion, but several search engines offered it a few years ago, so the trend might change yet again. Fees are typically charged for each page included and for every time a searcher clicks your page. Remember that paid inclusion does not guarantee your page will be shown by the search engineonly that it is in the index to be found. Later in this chapter, we look at paid inclusion in detail.

Get Sites to Link to You

As we have emphasized, the best way to get indexed is through a link from another site (one that is already indexed itself). If you have a well-established site, you have probably already attracted many links, but a new site obviously does not have any.

The best kind of link is one from a high-profile site, such as a directory, but almost any link has some value. By creating high-quality content on a subject, you will eventually attract links from other sites, but you can also execute a campaign to attract links. Chapter 13 is devoted to attracting more links to your site, an important subject whether it is new or well known.

    Search Engine Marketing, Inc. Driving Search Traffic to Your Company's Web Site
    Search Engine Marketing, Inc.: Driving Search Traffic to Your Companys Web Site (2nd Edition)
    ISBN: 0136068685
    EAN: 2147483647
    Year: 2005
    Pages: 138

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net