If your business has been around for a while and has a Web site, your site is overwhelmingly likely to be indexed by all the leading search engines. It is rare for corporate sites to be completely missing from search indexes, although it is quite common for individual pages from the site to be missing. (We show you how to figure that out later in this chapter.)
One way to tell whether your site is indexed is to search for it and see whether it is found. (Yeah, we figured you thought of that one.) If your company has a common name (AAA Plumbing), you might want to search for more than just the name ("aaa plumbing syracuse"). It's common for site owners to panic when their sites are not shown by the search engine for these navigational queries for company names. It's easy to jump to the conclusion that the entire site is not indexed, but that is rarely the case.
You can also use a search toolbar in your browser to check to see whether your pages are indexed. If you use the Google toolbar (or one for another search engine), you can navigate to your home page and take a look at the toolbarmost toolbars indicate that the page is indexed in some way. Figure 10-1 shows how Google's toolbar does it. You can see whether your site is indexed by that toolbar's search engine, although it does not help you figure out whether your page is indexed by other search engines. (We show you how to do that later in the chapter.)
Figure 10-1. Toolbars show indexed pages. Google's toolbar indicates the current page in the browser is indexed with the green PageRank bar.
Nearly all corporate sites have their home pages (and at least a few other pages) in the leading search indexes, but if somehow you do not, read on. It is unlikely but possible that your site is missing in actionit has happened, even to relatively large companies. Or perhaps your search marketing program's scope does not cover your whole company, but all the pages within your scope are missing from search engines. In that case, you need to ask a few questions:
If you can find at least a few pages of your site when you perform searches in the major search engines, you can skip ahead to the next section to determine how many pages you have indexed. If your site is not found at all, however, you can explore these questions to solve the mystery of the missing site.
Verify Your Site Is Not Banned or Penalized
Search engines do not ban sites on a whimthey do so only when your site has persistently violated their rules. As you might expect, some search marketers are unscrupulous, trying to take unfair advantage with tricks that provide an edge. Your company might have unwittingly become involved with an unethical search marketing consultant who uses spam techniques that try to fool the search engine, or you might have unwittingly violated a search engine's guidelines. (You can look at Google's guidelines at http://www.google.com/webmasters/guidelines.html; other search engines have similar rules.)
If your violation of the rules is not severe enough for a search engine to ban your site, you still might suffer from penalizationwhen the search engine starts dropping large numbers of your pages from its index or begins to lower your search rankings. You should make sure that signs of penalization are checked regularly (we explain how in Chapter 15, "Make Search Marketing Operational"), and you should investigate further if you see the following:
If you suspect a problem, you first need to diagnose the cause. In the next section, we discuss a spam technique called cloaking. We cover doorway pages and other stupid content tricks in Chapter 12, "Optimize Your Content," and link farms in Chapter 13, "Attract Links to Your Site." These are the most common spam techniques. If your site has been banned or penalized for using these techniques, you can clean up your site and request reinstatement, which is usually granted (although reinstatement sometimes requires an extended period of explanation and begging).
Make Sure the Spider Is Visiting
If the spiders are not coming to your site, your pages cannot be indexed and your site will not be found by organic searchers. Your Webmaster can help you check your Web servers' log files to see which search spiders have been visiting your site. (Most Web servers are configured to log search spider visits, but some servers might need to be adjusted by your Webmaster to capture this important information.) Figure 10-2 shows an extract of a log file showing that a spider has visited. The log file indicates the name of the user agent that accessed the page noted, which indicates the software that was used to see the page.
Figure 10-2. Spotting spider activity. Carefully examining your log files can prove that spiders are visiting your site.
Most of the visits to your site generate a log listing of something like "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0)," which indicates that the Internet Explorer Web browser (version 5.5) was used to access your site. As you might expect, the vast majority of the user agents listed in your log file are from Web browsers. But a small number of log entries might show you that the spiders are crawling. Googlebot, as you might surmise, is Google's user agent. Yahoo! aptly named its spider Slurp. Table 10-1 shows some the names of some of the leading spider's agents, but there are many more, and they change their names frequently.
Each search engine spider has its own frequency for returning to your Web site. Spiders return to a typical Web site at least once a month, but popular corporate Web sites might be revisited weekly, or even daily. By analyzing how frequently spiders crawl your pages, and which pages they check the most, you will know how quickly content changes on your site will be reflected in search indexes.
Although it is rare to find a site that the spiders do not visit at all, it can happen for a few reasons:
If your site truly is not being visited by spiders, the remedy depends on which of the above reasons is the cause. If your site is not linked or links to your site are not effective, the best way to get the spider to visit is to make sure other well-respected sites link to yours (explained in more detail later). If the spider has given up, first remove the spider trap (also explained later), and then you should manually submit your site to the search engines.
Search engines vastly prefer to find new sites by following links because analyzing link patterns is one of the ways engines judge relevance; if your site is linked and spiders are not visiting, however, you should manually submit your home page's URL. (If you create a new site but are too impatient to wait until someone links to you, you can also submit, but waiting to be found from a link will help your pages rank higher.)
Every major search engine, except Ask Jeeves, has a way to submit your site, but not all of them are free. Search engines that offer free submission often refer to their "Add URL" page as the place you should goat Google it's at www.google.com/addurl (with others listed in Table 10-2). Remember, if your site is missing from multiple search indexes, you need to submit to each search engine. Typically, pages submitted are included in the search index within a few weeksyou can check your log files to see whether the spider is visiting.
Submitting to local or country-specific search engines is no different from worldwide search engines. The spiders can detect the language of the site and will add it into the appropriate-language version of the index, which we cover in detail in Chapter 12.
A surefire way to get your pages included is to pay a search engine to put them in the index. As discussed in Chapter 3, "How Search Marketing Works," paid inclusion not only guarantees to keep your pages in the index, it also promises that they will be revisited by the spider regularly. Only Yahoo! (of the major worldwide engines) is offering paid inclusion, but several search engines offered it a few years ago, so the trend might change yet again. Fees are typically charged for each page included and for every time a searcher clicks your page. Remember that paid inclusion does not guarantee your page will be shown by the search engineonly that it is in the index to be found. Later in this chapter, we look at paid inclusion in detail.
Get Sites to Link to You
As we have emphasized, the best way to get indexed is through a link from another site (one that is already indexed itself). If you have a well-established site, you have probably already attracted many links, but a new site obviously does not have any.
The best kind of link is one from a high-profile site, such as a directory, but almost any link has some value. By creating high-quality content on a subject, you will eventually attract links from other sites, but you can also execute a campaign to attract links. Chapter 13 is devoted to attracting more links to your site, an important subject whether it is new or well known.