Getting Found

Web surfers who know about your organization, products, or services but don't have your URL may rely on search engines and directories to find your Web site. Still others, who don't even know your company exists, may use search engines to enter search phrases that should apply to you. How do you increase the odds of having your site listed among those found?

Search Engine

An online utility that enables Web surfers to find Web sites based on search criteria they provide.

This part of the chapter explains how search engines work and tells you what you can do to get your Web site found by the people searching for it.

Tip

Search Engine Watch is an excellent resource for getting more information about search engines ”including lists of search engines. Visit it at http://www.searchenginewatch.com/.

Search Engine Basics

Search engines are one of the most misunderstood features of the Web. Few people know how they work or why different search engines come up with different results for the same search phrases. Even fewer people know how the search engines know about all the Web sites they list on the search results pages.

Here's a quick look at how search engines work.

Tip

To learn more about how to use search engines to find Web sites, check out Search Engines for the World Wide Web: Visual QuickStart Guide , a Peachpit Press book by Alfred and Emily Glossbrenner. You can learn more about it at http://www.peachpit.com/books/catalog/K5842.html.

Robots and Spiders and Crawlers ”Oh My!

Many search engines ”such as HotBot (http://www.hotbot.com/) and WebCrawler (http://www.webcrawler.com/) ”utilize software programs called robots , spiders , or crawlers . These programs venture out onto the Web when activity is low (whenever that is) and search for new pages. When they find a site they've never visited before, they follow all the internal links within the pages to learn about the pages on the site. They also occasionally (but not often enough) revisit sites to update the information they already have.

Newbie Search Confusion

I know how to use a search engine and I assumed that everyone else did. That is, until recently. I began talking to Web newbies and watching them access the Internet. Here are some of the startling things I learned:

• Some people think the only way to access a Web page is by clicking a link on another page. If the site they want to access is not linked through the default Home page that appears in their Web browser ”normally a portal site like Yahoo!, MSN, or NetCenter ”they can't access the site.

• Some people think you search for a site by entering its URL or domain name in a search box and clicking Search or Go or whatever. Unfortunately, if the site has not been indexed or listed in that search engine's database, the site will not appear in the list of search results.

• Some people do not understand that search results seldom fit on one page. They look no further than the first page of results to find sites they seek.

• Some people think that the featured sites that appear in search results lists are featured because they're the best match for their criteria. They don't realize that many of those sites paid to be featured and may not even be good matches for their criteria.

• Some AOL users think that if a Web site does not have an AOL keyword, they cannot access it via AOL. (Other AOL users don't understand that you don't need AOL to access the Web, but that's another sad yet unrelated problem.)

These examples illustrate how difficult it can be to get your site found by newcomers to the Web. But until these people become more search savvy, there isn't much you can do.

What kind of information do they gather? Well, that depends on the program. Some programs gather just Web page titles ”the information that appears in the Web browser title bar. (This is not the same as a file name). Others gather every single word in the page. Still others gather information stored in special meta tags embedded in the page (more on that later).

Directories

While some search engines depend solely on robots, spiders, and crawlers, others ”such as Excite (http://www.excite.com/) and Yahoo! (http://www.yahoo.com/) ”start with a directory that entries can be manually added to. Some of these directories are static, containing only the information manually entered. Others use directory entries to send out robots, crawlers, and spiders to gather information directly from the Web sites entered.

The Database

The information gathered about Web sites ”whether gathered by robots, etc., or by manual entries into a directory ”is stored in a database that can be indexed and searched. Web surfers search the database based on words or phrases. Sites that match the criteria are displayed in a search results list.

When you use a search engine to find a Web site, you're searching that search engine's database of sites, not the Web itself.

Who's On First

Search engine result lists are usually very long ”sometimes with tens of thousands of matches. (Of course, not all of them appear on one page; you'll have to click through hundreds of pages to see them all.) The sites that most closely match the search word or phrase should appear at the top...right? Well, sometimes. It depends on the search engine and what you entered.

Figure 11.1 shows the first page of results for a query on Search.com (http://www.search.com/) for horse camping in arizona . (I'm going camping tomorrow and it's on my mind.) If you can read the tiny print, you can see that not all of the pages seem to relate to the search phrase. (If you can't read the tiny print, please take my word for it.) Yet several pages further into the search results is information about the campground I'll be going to. Why didn't it come up on top?

Figure 11.1. The first search results page for a search.com search.

The answer is: it depends. Most search engines list results based on how well the search word or phrase was matched in its database. (In my opinion, this is the way it should work.) Others ”such as Direct Hit (http://www.directhit.com/) ”list results based on popularity ”how many people click through to that page. Still others put paid-for listings near the top and others below it.

And of course, whether a listing appears at all depends on whether the listing was added to the search engine's database to begin with.

The Bottom Line

The bottom line is this: to be found by a search engine, your site must be included in that search engine's database. Your job is to get included in as many search engine databases and directories as possible.

Tip

Search Engine Watch's Search Engine Submission Tips page (see Figure 11.2; http://www.searchenginewatch.com/webmasters/) is a great place to learn more about how search engines work and how you can increase your odds of being near the top of the search results list.

Figure 11.2. The Search Engine Submission Tips page on Search Engine Watch is a great way to learn more about getting your site found by search engines.

Using Meta Tags

If you were reading carefully , you probably caught my use of the phrase meta tag . I didn't define it then because I was saving it for now.

Meta Tag

An HTML tag for embedding certain types of information in a Web page document.

Meta Tags, Defined

Meta tags are HTML tags that can be inserted in the HEAD part of an HTML document to embed document information. Now, in English: meta tags enable you to include information about a Web page where it won't be seen by site visitors but can be seen by robots, spiders, and crawlers.

There are many uses for meta tags, but the ones you should be most interested in are for storing description and keyword information. This is the information most often picked up by robots, etc.

Tip

Be sure to include the description and keyword meta tags in all the pages of your Web site. Remember, not everyone visits your site starting with the Home page.

Description

The description information should be a short, concise description of the site or page. Don't repeat the page title; it's a waste of words. Since you're usually limited to 256 characters , make the most of them.

The format for the description meta tag is as follows :

 <META name="description" content="  concise description of Web site or page  "> 

What does the description meta tag look like in an HTML document? Figure 11.3 shows an example from the Home page of one of my sites, wickenburg-az.com (http://www.wickenburg-az.com/).

Figure 11.3. Meta tags for keywords and descriptions are inserted in the HEAD part of an HTML document, along with the page's title.

Keywords

The keyword information should be a list of key words or phrases, separated by commas. Again, there's usually a limit of 256 characters for keywords, but not all robots, etc., accept that many. If a robot, etc., is limited to, say, 150 characters, it will take the first 150 characters it finds. For that reason, you should put the most important keywords at the beginning of the meta tag. Also, don't repeat the same words over and over again because most robots, etc., are programmed to ignore repetition ”and a few will actually penalize you for it!

The format for the keyword meta tag is as follows:

 <META name="keywords" content="  list of key words and phrases separated by commas  "> 

Figure 11.3 also shows the keywords tag from the Home page of wickenburg-az.com. Note that it includes the words that best describe the site's contents ”and even some misspellings of those words! (When I analyze my site's logs, I often find that site visitors used an incorrect spelling of Wickenburg and still find my site.)

Meta Tags in Search Results

Figure 11.4 shows a close-up of some search results for the phrase horse camping .

Figure 11.4. A close-up of some search results reveals how meta tags are not always properly used.

Most of the entries include the name of the page, followed by a brief description. The description comes from the description meta tag. This is how you want your pages to appear in search engine results.

But look at the second entry, "Camping at Crazy Horse." The description text doesn't appear to be a description at all. Sure enough, when you click the link you can see that the "description" is really the first handful of words from the Home page (see Figure 11.5).

Figure 11.5. When you omit the description meta tag from an HTML document, the first handful of words on the page appear as a description.

Now look at the last entry in the Figure. This entry is a real mess. It appears that the Web author included keywords in the page's title, as well as the description. Figures 11.6 and 11.7 partially confirm this; the title is entered three times in the HTML document's source code. Although the description meta tag appears okay, the error in the title's code may have caused the robot, etc., to pull up the wrong information for the description.

Figure 11.6. In this example, the page title includes keywords,...

Figure 11.7. ...a fact that's confirmed when you inspect the page's source code. Perhaps the inclusion of the title code in triplicate is what messed up this page's listing in the search results; the description meta tag appears okay.

As you can see, when you set up meta tags correctly, your site's listing in search engines that use meta tags can be exactly what you want it to be. But when you do it wrong, the listing is wrong.

Adding Your Site to Search Engine Databases

Once you've set up meta tags for all the pages on your site, you can help improve the chances of robots, etc. finding your site by manually adding your site's information to search engine directories. Then, if that search engine sends out robots, etc., your site will be found and indexed.

Hiding a Page with Meta Tags

There may be instances when you don't want a particular page on your site indexed by a robot, etc. For example, perhaps a page is private, for the use of a few people who have the exact URL of the page. Having the page listed in search engine results lists won't keep it private for long.

You can use a meta tag to block robots, etc., from indexing it:

 <META name="robots" content="noindex"> 

This really works. My Web server sits on a computer desk beside my main production computer. Sometimes I leave the monitor turned on so I can watch activity. (You think I'd have better things to do with my time, right?) I've noticed that within 10 seconds of adding one of my sites to a directory, a robot sometimes appears on my server to check out the site's Home page. Now that's service!

To add your site to a search engine's directory, begin by visiting that search engine's main page. Now look for a link that says something like "Add a site" or "Add a URL" or "Suggest a Site." This link is usually in very small letters near the top or bottom of the Home page, but sometimes it can be buried on other pages. And sometimes it may not appear at all. Figure 11.8 shows how it appears at the bottom of the Excite Home page (http://www.excite.com/).

Figure 11.8. You can find a tiny link for adding a URL at the bottom of Excite's Home page.

Unfortunately, you can't add your Web site to every search engine. Some search engines let their robots, etc., do all the work on their own. Others are very particular about which sites they include.

When you click the link, a page with instructions for adding your site will appear. Read the information and follow the instructions. Figure 11.9 shows Excite's Add URL page. It offers a pretty straightforward form. Fill it out and click the Send button. That's all there is to it.

Figure 11.9. This page on Excite offers a simple form for adding your site to its database of Web sites.

In some cases, a message will appear with some kind of confirmation that your request is being processed . Sometimes you'll be told that it takes four to eight weeks (or some other silly period) to process your request. You might want to make a note of it, then try the search engine after a reasonable amount of time has passed. If you're not listed, try adding it again.

Submit to All 2,550 Search Engines Here

Sound familiar? It should ”it's on a banner in the screen shot in Figure 11.9. It'll also come to you in junk e-mail and flash at you periodically as you surf the Web.

Beware of services that claim to submit information about your Web site to hundreds or thousands of search engines all at once. I've tried these services and found two things:

• They seldom work as advertised. In fact, I've found that my site was not added to listings that it was supposed to be added to.

• They often require a fee for listing your site with more than a handful of search engines.

My advice: if you have the time to try one of these services, go right ahead. Just don't expect them to work. And don't pay for the service.

Adding Your Site to Directories & Other Sites

You can also add your site as a link to other sites, including directories of sites in your geographical or business area. This is especially useful if the other site is a good match for yours. For example, a state directory of accountants would be a good match for a CPA and a tourism directory would be a good match for a hotel or tour service.

Paying for Links

I do not believe in paying for links. There. I said it. Now let me explain.

There are two ways to make money with a Web site: sell a darn good product or service that people are willing to buy online or sell advertising space on the site. Many Web sites exist solely to make money using the second method ”selling advertising space to other Web sites.

Some of these Web sites are portals for Web surfers to access other sites. They often call themselves directories and they offer categorized or searchable links to other sites. The sites that are listed are the ones that have paid a fee. Or perhaps any site can be listed, but the ones that pay a fee are listed first or with a banner ad. The point is, the site's links exist to generate revenue for the site.

Is the person paying for that listing getting his money's worth? Well, it depends on many things:

• Is the directory site popular? If the site doesn't get many hits, few people will see its links. This is the biggest problem with many of the pay directory sites: they're more interested in collecting link fees than promoting their own site. Would you advertise in a newspaper that didn't have a good circulation?

• Is the directory site related to the site that's advertising on it? If the directory specializes in real estate, for example, it won't do much to promote the sites of unrelated businesses.

• Do people come to the directory site for links or for some other content? If they're coming for other content and they're like me, they'll ignore the ads and most of the links.

• Are the other links for quality sites? When I go to a directory Web site and half the links are worthless, I'll move on to another site before I waste any more of my time.

Here's a true story. I know a certain Chamber of Commerce that charges its members $70 a year to include a link on the Chamber's Web site. (That's paying members , mind you, paying even more for this link.) When asked why there's a fee, the Webmaster claimed that the money collected paid to link the Chamber's site to other sites, most of which were state and tourism related. Can someone explain to me how the local appliance fix-it guy would benefit from having his site indirectly linked to a state tourism site?

I will agree that there are some instances when paying for a link to another site can really get you more hits ”hits that are valuable to your business. But like any other advertising expenditure, you should carefully consider what you're getting for your money before you pay. Then monitor hits from that site to see if you're getting what you paid for.

Finally, I believe that if your site is included in as many free search engine databases and directories as possible, and referenced in all your printed materials, you'll get all the hits you need...and then some.

To add yourself to one of these directories, begin by visiting the site. Make sure it's a good match for you business. Then poke around until you see some way to add a link or ask about adding a link. If all else fails, e-mail the Webmaster.

Be aware that some sites charge a fee for links. Keep your advertising budget in mind when considering whether a link is worth the fee. (And please read the sidebar on the next page for my very strong opinions on the subject.)

Other sites will only ask for a reciprocating link. Before agreeing to that, make sure the site is up to your standards. I discuss reciprocating links in more detail back in Chapter 6.