Monitoring Usage

Once your site is up and running, it should start getting hits. At least you hope so. But how can you tell for sure? And what can you learn about the people who visit your site?

The answer is in the Web server logs for your site.

Log Analysis Software

Did you ever look at a Web log? Figure 12.1 shows part of a log from my server, which runs WebSTAR software. As you can see, it isn't exactly the easiest thing to decipher. And this particular log, which is for the month of May 2000, is over 26 MB in size !

Figure 12.1. The first bunch of lines from my server's Web log for the month of May 2000. Not exactly interesting reading material, is it?

Log analysis software can help. This software can take raw Web server log files and make sense of them by summarizing information in a format you can use.

Finding a Program

There are many log analysis software products out there. They range in price from free to thousands of dollars. In most cases, you can download a trial version of the software from its developer's Web site and see how you like it before you pay.

Finding Web analysis software isn't difficult. Begin by visiting the Web site for the developer of your Web server. Then follow links for "add-ons" or "utilities." Or just search the site for log analysis . You should be able to find some options. Figure 12.2 shows the list available on StarNine's WebSTAR pages (http://www.starnine.com/webstar/).

Figure 12.2. The StarNine Web site lists a number of Web log analysis programs that will work with the WebSTAR Web server software.

I tried several programs before settling on Summary Pro by Summary.Net (http://www.summary.net/). It costs $249 and has the features I need to generate reports for all the Web sites on my server. If you have only one site on your server, the basic version of Summary should be all you need; it costs only $59. Summary works with a variety of Web server software packages and is available for both Windows and Mac OS.

If Your ISP Runs the Server...

If your ISP runs the Web server on which your site resides, he's likely to offer one of three options:

• The ISP may give you access to Web log analysis software reports for your Web server. This will utilize the software he's decided to run and may not be customizable. But it's better than nothing. (This is what I do for the folks who maintain Web sites on my server.)

• The ISP may allow you to download your own Web usage logs. You can then run Web analysis software on them on your own computer. This gives you more flexibility as far as software is concerned , but downloading logs every time you want to run a report can be a real pain in the butt ” especially if your logs are 20 MB or larger in size!

• The ISP may not give you access to any usage statistics at all. How inconsiderate. Try reasoning with him. If that doesn't work, either live with it or get a new ISP.

What to Look At

If your Web analysis software is powerful, it will probably offer dozens of reports (see Figure 12.3). You can waste a lot of time trying to make use of them all. Here are the statistics that I find most useful.

Figure 12.3. Summary Pro offers dozens of reports ”far more than the average Webmaster needs.

Hits

The thing you're probably most interested in is hits, and your software should be able to summarize them in a variety of ways. Summary Pro reports hits right in its Summary Report (see Figure 12.4), which Summary Pro also e- mails to me each time it processes the logs (I have it set to do it once a day). But I also like to review hits in the Monthly Report (see Figure 12.5) to see how site activity has changed over time.

Figure 12.4. Here's Summary Pro's Summary Report. As you can see, it's chock full of information. In case you're wondering, the gap in the Time graph is a result of me accidentally trashing 11 days worth of logs.

Figure 12.5. The monthly report shows basic information on a month-by-month basis. It's satisfying to see that hits have been climbing steadily ”remember I lost 11 days worth of logs of April and only one day of June is included.

There are different kinds of hits:

• Page hits are hits to Web pages ”HTML documents. This is the number that should most interest you.

• Graphic hits are graphics and other image files that are downloaded with Web pages. For example, if a page includes references to five graphic buttons , a logo, and a photo of your storefront, when the page is visited, there are seven graphic hits. In most cases, graphic hits should be disregarded.

• Other hits include other types of files that were sent from your Web server to the visitor's computer. This could include PDF files, ZIP or SIT files, style sheets (CSS files), icon (ICO) files, and any other file that is neither a page nor a graphic. You may want to disregard these hits or you may want to investigate them further to see what files site visitors are downloading. It depends on what files your site includes.

The next thing you want to know about page hits is which pages are getting hit. Your Home page should be getting the most hits, but it might not be. Your software should be able to break down the number of hits by page. Use this information to learn what your site visitors find most ”and least ” valuable on your site. You can then add more of the useful material and, if desired, remove the material that doesn't get much attention.

In Figure 12.4, you can see this information in the Content section under Pages. It's not too meaningful here because my Home page uses a frameset that references the welcome.html and toc.html pages. But if I click the Pages link there, I can get a list of current period page hits in order of the number of hits (see Figure 12.6).

Figure 12.6. Here are the first bunch of links from the Page Requests report. It lists site pages in order of popularity. This is part of the first page of the report.

Referrers

Referrers are pages that site visitors were on when they clicked a link to your site. Most referrers will probably be pages on your own site, but if you have links to your site on other sites, they'll be counted, too. Links generated by search engines are also included as referrers.

In Figure 12.4, you can see some basic referrer information in the Referrers section. As you can see, 77% of the referring pages were on my own site. But 4% of the hits came from Yahoo!'s search engine. That's very comforting. The Referrer Report gives more details (see Figure 12.7) for the current period.

Figure 12.7. The Referrer Report lists pages that site visitors were viewing when they clicked links to your site. This is only part of the report.

Search Words & Phrases

My favorite reports are the Search Words (see Figure 12.8) and Search Phrases reports. These reports tell me what words or phrases visitors used in search engines to find my Web site. As you can see by looking closely at Figure 12.8, not everyone knows how to spell Wickenburg . Because I also included incorrect spellings in my keywords meta tag (as discussed in Chapter 11), bad spellers can still find my site.

Figure 12.8. The Search Words report tells me how people search for and find my site. This is only the first part of the report.

Visitor Statistics

Visitor statistics tell you things about how many visitors your site had, what software they used, and how they spent their time on your site. You can see this information in the Visitor and Visits sections of Figure 12.4. Detail reports can tell you more about specific statistics.

Errors

Although you don't want any errors in your logs, it's important that you investigate the errors that are reported . In general, errors fall into three categories:

• Bad links on other pages to pages on your site. This can occur if a link is created incorrectly or if you moved or deleted a page that was linked to. There's not much you can do about bad links on other sites, but you can fix bad links on your own site.

• Visitors entering incorrect URLs for your domain name . Again, this could be an error on the visitor's part or they could be trying to access a page that you've moved or deleted.

• Robots or Web browsers trying to access files that don't exist on your site. The two biggest culprits are robots.txt (a file with instructions for search engine robots) and favicon.ico (a file referenced by Internet Explorer 5 or later when someone bookmarks your site). Neither file is required on your server and you can ignore any error messages pertaining to them if they don't exist.

Other Information

This is just the basic information that I pay close attention to. You might find other information helpful, too. The point is, you should be monitoring your site's activity whenever possible, looking for specific, meaningful information. Only by doing this can you understand how effective your Web site is.