Web Logging

Another useful feature of IIS for troubleshooting problems with client connections to IIS is web logging, also called website logging or IIS logging. This feature lets you log the activity of users connecting to your sites in log files that can be viewed and analyzed to look for traffic patterns. Using these logs you can

• Determine which sites on your server or portions of your sites are most heavily viewed, and configure IIS quality of service (QoS) settings to optimize performance accordingly. You can also use logging information to more effectively target users with marketing information.

• Detect trends such as growth patterns in user visits and plan how to upgrade the hardware and software on your IIS machine to accommodate such growth in the future.

• Detect unusual traffic patterns that can indicate problems with the operation of your IIS machine or attempts by malicious users to hack into your sites. For example, repeated log entries showing failed authentication events can indicate an attacker probing your site.

Other good reasons for enabling logging on your IIS machines include the following:

• While IIS generates errors, warning, and informational events in the event logs, often additional information is logged in the web logs concerning these events. It’s a good idea to review your web logs whenever you find an IIS event in the System or Application logs using Event Viewer. For more information about IIS and event logs, see the later section “IIS and Event Logs.”

• Even when IIS is configured to generate friendly (custom) HTTP error messages using the Custom Errors tab on IIS Manager properties sheets, IIS will sometimes return only standard HTTP error codes with no substatus codes attached to help troubleshoot the problem. If web logging is enabled, however, and the W3C Extended Format is configured, substatus codes are always recorded in the web logs when IIS generates HTTP errors. This helps administrators who can review the web logs to gain more insight into the causes of HTTP errors.

  Note 

  In earlier versions of IIS, web logs were managed by the IIS Admin Service service. In IIS 6, web logging using W3C Extended, IIS, and NCSA formats is handled by the HTTP Listener (Http.sys) for better performance and to avoid concurrency issues. Web logging using ODBC, however, is managed by the worker process (w3wp.exe) associated with the site.

Enabling and Configuring Web Logging

Like other IIS properties, web logging can be enabled and configured at various levels, and settings at higher levels are inherited by lower levels in the usual way. You can enable web logging (turn it on) at two levels:

• Web Sites level Enabling web logging at this level turns it on for all websites on the IIS machine.

• Individual websites level You can also enable web logging on a per-website level.

Once web logging has been enabled for a site or for all sites, you can configure which specific home directories, virtual directories, subdirectories, and files should have their access logged.

Let’s say, for example, that you want to log access to only the default.asp page within the home directory of the Default Web Site, and not for files in any other sites on your server. There are three steps you need to take to make this happen:

1. Open the properties sheet for the Default Web Site and select the Enable Logging check box on the Web Site tab to enable web logging for this particular site (Figure 13-5). This turns on web logging, but no logs are created unless specific directories and files have been marked for logging within the site (see step 3). However, all directories and files are marked by default for logging in IIS, so it’s really a matter of specifying which directories and files you don’t want to log access to.

   
   Figure 13-5: Enabling logging for the Default Web Site

2. On the same Web Site tab, select the log file format you want your web logs to be recorded in using the Active Log Format list box. You can click Properties to further configure it by specifying where your log files will be stored and various other settings that depend upon the particular format you’ve selected. I’ll talk more about logging formats soon in the section “Log File Formats.”

3. Finally, you need to mark which directories and files you want to log. If you want to log access to everything (physical and virtual subdirectories and files) in your website, switch to the Home Directory tab and make sure the Log Visits check box is selected. Similarly, to log access to files in a virtual directory, use the Virtual Directory tab on the properties sheet for the virtual directory. To log one specific file, use the File tab on the properties sheet for the file. In the example, you only want to log access to the file default.asp; so open the properties sheet for the Default Web Site, select the Home Directory tab, clear the Log Visits check box, and click OK. This leaves all files and directories in the site unmarked for logging (no logging will occur). Now open the properties sheet for default.asp and select the check box for Log Visits on the File tab to mark this file for logging of client connection attempts (Figure 13-6). Click OK to apply the setting.

   
   Figure 13-6: Marking the default.asp page for logging

Log File Formats

There are five different logging formats that can be chosen for web logs:

• W3C Extended Lot File format This is a customizable format developed by the World Wide Web Consortium (W3C) that allows you to select which properties you want to write to your logs. By default, IIS selects this format for web logging, and all logging is done in ASCII unless UTF-8 logging is enabled on the IIS machine (see the upcoming “UTF-8 Logging” sidebar).

• IIS Log File format This is a fixed format that cannot be customized and was developed by Microsoft for early versions of IIS. Although this format records more information than the NCSA format, the IIS log file format is rarely used nowadays, having been superceded by the more powerful and flexible W3C Extended format.

• NCSA Log File Format This fixed format cannot be customized and was developed by the National Center for Supercomputing Applications (NCSA) for Mosaic. An advantage of this format is that it is used by the widest range of web server products, but W3C Extended format is more powerful and flexible and is preferred by most IIS shops.

• ODBC Logging format This lets IIS log connection attempts to an ODBC- compliant database like Microsoft SQL Server. A major disadvantage of ODBC logging is that enabling it disables kernel-mode caching in IIS, which can significantly degrade IIS server performance. As a result, ODBC logging is not recommended in most circumstances.

• Centralized Binary Logging format This is new to version 6 of IIS and allows multiple websites to write to a single log file using a binary format. Centralized binary logging is particularly useful in a web hosting environment where an ISP is hosting hundreds of websites on a single IIS machine and it would consume too much system resources and degrade performance if individual web logs were used for each website. See IIS Help for information on how to use Adsutil.vbs to enable and configure centralized binary logging.

  Note 

  While all five log file formats can be used for logging visits to websites, only three of them (W3C Extended, ODBC, and IIS formats) can be used for logging visits to FTP sites on IIS.

In addition to the preceding logging methods included with IIS, you can also develop your own custom logging modules using COM and use them for logging website visits. Unfortunately, using custom logging modules disables kernel-mode caching in IIS and thus degrades performance of your server; so unless there is some compelling reason for using this approach, it’s best avoided.

UTF-8 Logging

A new feature of version 6 of IIS is support for writing web log files using UTF-8 (a type of Unicode character encoding) instead of the traditional ASCII character encoding that uses the local character set on the machine. The advantage of UTF-8 logging is that you can write data in non-European character sets to the web logs. To enable UTF-8 logging, right-click the server_name node in IIS Manager, select Properties, and select the check box labeled Encode Web Logs In UTF-8. Two things to note about UTF-8 logging are that it is a global setting that is turned on or off for all web logging, and it cannot be used for logging visits to FTP sites.

Using W3C Extended Logging

Unless you are working in a web hosting environment where centralized binary logging could be advantageous, the best choice for web logging is W3C Extended format. Because of this, I’ll focus on the details of how to configure and interpret logs created using this format.

Try the following walkthrough. Enable logging for the Default Web Site on your IIS machine, and use Windows Explorer to navigate to the \Windows\System32\ LogFiles folder where IIS web logs are stored by default. There should be no files present in the folder.

Now open the URL http://localhost on your IIS machine and view the default document for the Default Web Site. In the \LogFiles folder in Windows Explorer, you should see a text file appear with a name something like ex030116.log, where

• ex stands for W3C Extended format

• 03 are the last two digits of the current year

• 01 is the current month

• 16 is today’s date

The name of your log file will be different, of course. Try opening the file—it will probably be blank because IIS first creates the web log file and then, a short time later, writes the first entry to it, so wait a few seconds and open the file again and you should see something like this:

#Software: Microsoft Internet Information Services 6.0 #Version: 1.0 #Date: 2003-01-16 23:16:46 #Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs- username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status  2003-01-16 23:16:46 127.0.0.1 GET /Default.asp—80—127.0.0.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.2;+.NET+CLR+1.1.4322) 200 0 0 

The first three lines identify the log, while the fourth line (beginning with #Fields) displays the names of the different properties that are logged. The fifth line (beginning with 2003-01-06) displays the log information for the visit that occurred when you opened http://localhost on the IIS machine. Because the lines are long and wrap in the log text in the text, and because fields are separated by spaces instead of commas or tabs, it’s difficult to read and interpret the text. To make it more understandable, I’ve reformatted the last two lines (fields header and record of first visit) as Table 13-3.

Table 13-3: Sample Record for W3C Extended Log File

Log Field	Sample Value	Interpretation
date	2003-01-16	Today
time	23:16:46	A few minutes ago
s-ip	127.0.0.1	The Default Web Site on localhost
cs-method	GET	HTTP GET request for default document
cs-uri-stem	/Default.asp	Default document
cs-uri-query	-	No data
s-port	80	Standard TCP port for HTTP
cs-username	-	No data
c-ip	127.0.0.1	Client browser is also on localhost
cs(User-Agent)	Mozilla/4.0+(etc)	Type of web browser used
sc-status	200	HTTP 200 status code indicates request was successful
sc-substatus	0	0 indicates no HTTP substatus code was logged
sc-win32-status	0	Used by Microsoft Windows

Tip 

A useful property you should always log for troubleshooting purposes is sc-substatus, which logs substatus codes (such as 404.3) for HTTP error messages.

Field Definitions for W3C Extended Logs

To better enable you to read and interpret W3C Extended logs such as the sample file just shown, here is a quick reference to the various fields and what they mean. Note that the different prefixes have specific meanings, for example:

• s- The action occurs on the server

• c- The action occurs on the client

• cs- A client-to-server action (a request)

• sc- A server-to-client action (a response)

Table 13-4 shows the various fields and their meaning for W3C Extended logging.

Table 13-4: Summary of Field Definitions for W3C Extended Log File Format

Field	Appears As	Description
Date	date	Date activity occurred
Time	time	Time activity occurred
Client IP Address	c-ip	IP address of client accessing server
User Name	cs-username	Name of authenticated user (anonymous users indicated by dash)
Service Name	s-sitename	Internet service and instance number accessed
Server Name	s-computername	Name of server
Server IP Address	s-ip	IP address of server
Server Port	s-port	Port number used
Method	cs-method	HTTP verb used in client request
URI Stem	cs-uri-stem	File accessed
URI Query	cs-uri-query	Query performed by client
Protocol Status	sc-status	Status of action in HTTP or FTP terms
Win32® Status	sc-win32-status	Status of the action in terms used by Microsoft Windows
Bytes Sent	sc-bytes	Number of bytes sent by server to client
Bytes Received	cs-bytes	Number of bytes received by server from client
Time Taken	time-taken	Time duration (milliseconds) consumed by action
Protocol Version	cs-version	Protocol version used by client
Host	cs-host	Contents of host header
User Agent	cs(User-Agent)	Web browser used by client
Cookie	cs(Cookie)	Content of cookie sent or received (if any)
Referrer	cs(Referer)	Last site visited by user if redirected

Scheduling W3C Extended Logging

To configure when your W3C web logs will be created and what information will be written to them, click Properties on the Web Sites tab of your website to open Logging Properties. The General tab on this properties sheet (Figure 13-7) is used for configuring the following aspects of logging:

Figure 13-7: Specifying a new log schedule and log file directory

• New Log Schedule Indicates the condition under which new log files are created. This can be hourly, daily, weekly, monthly, or when the current log reaches a specified size, or you can keep on logging to the same file and let it grow as big as you want. Daily is the default option, and it creates a new log file when the first visit occurs after midnight. On high-volume sites, you might choose the Hourly option, but don’t choose Unlimited File Size unless you’ve got terabytes of disk space available on your machine! Log files are named differently depending on the new log schedule you select, and these naming conventions are summarized in Table 13-5.

  Table 13-5: Naming Conventions for W3C Extended Log Files

  New Log Schedule	Pattern for Log Filename
  Hourly	exyymmddhh.log
  Daily	exyymmdd.log
  Weekly	exyymmww.log
  Monthly	exyymm.log
  Unlimited	extend#.log
  Specified size	extend#.log

• Use Local Time for File Naming and Rollover I just noted that the Daily option creates new log files at midnight, but for W3C Extended logging this means midnight Coordinated Universal Time (UTC), which is essentially midnight Greenwich Mean Time (GMT). You can cause IIS to create new log files at midnight local time by selecting this option, but even if you do so the visit times recorded in the log file will be UTC times. To see how to convert the times in your log files to your local time zone, see the section “Converting Log Formats” later in this chapter.

• Log File Directory The default location for saving web logs is within a subdirectory of the \Windows\System32\LogFiles folder. The name of the subdirectory depends on the log file format you’ve chosen, and for W3C Extended format the subfolder is \W3SVCsite_id, where site_id is the website identifier displayed for the site when the Web Sites node is selected in IIS Manager. The Default Web Site is assigned the identifier 1 (one) so the actual location where W3C Extended log files will be stored for the Default Web Site is the \Windows\System32\LogFiles\W3SVC1 folder.

Configuring W3C Extended Logging

In addition to scheduling when new log files will be created and where they will be saved, you can also configure what properties are saved in these files for each record generated by a client connection attempt. To configure which properties to log, use the Advanced tab on Logging Properties (Figure 13-8).

Figure 13-8: Specifying which properties to log

Refer to Table 13-4 to review what the various properties mean for W3C Extended logging.

Tip 

Don’t log too many properties in your web logs. Not only do you need adequate disk space for your logs, but too much logging also adds a performance hit to IIS by consuming memory and processor resources. Log only what you need, and only for the sites or portions of sites (virtual directories, physical directories, and files) that you are most concerned about for reasons of security or performance or for gathering statistics.

Securing Web Logs

You can better protect your web logs from intruders by moving them to a different directory than the default directory IIS uses to store them, which is the \Windows\ System32\LogFiles folder. To change the location of W3C Extended, IIS, or NCSA log files, click the Properties button on the Web Site tab of your site’s properties sheet and specify a new Log File Directory in the text box (refer to Figure 13-7).

Remote Logging

While earlier versions of IIS required the log file directory be located on the local machine, version 6 lets you log to a shared folder on a file server on your network. To do this, type the UNC path to the share in the Log File Directory textbox (shown in Figure 13-7). This new approach has the advantage of allowing you to centralize the backup of log files from multiple IIS machines to a single file server, but because of the performance penalty incurred it is generally not recommended. If you do decide to employ remote logging for your IIS machines, Microsoft suggests you consider using IPSec for the connection between your web and file server, as anyone sniffing your network can read your log files because they are being sent as plain text.

To secure your new log file folder, make sure you assign appropriate ACLs to it as follows:

• Administrators Full Control

• IIS_WPG group Full Control

• SYSTEM Full Control

Managing Web Logs

To delete web logs, first stop the website the logs are associated with, and then delete them using Windows Explorer. If you don’t stop the website, you may not be able to delete the files due to a sharing violation.

To review web logs like W3C Extended logs that are written in ASCII format, open them using a text editor like Notepad or import them into a spreadsheet or database program for analysis. If you’ve selected the Hourly or Daily new log option and you have too many log files to review, use the copy command to combine multiple text files into a single large file. Type copy /? at the command line for information on how to append multiple files.

Note 

If IIS runs out of disk space and can’t add records to the log file, IIS will automatically shut down and an event will be recorded in the Application log of Event Viewer. When more disk space is added (for example, by extending the volume), IIS starts and logging resumes.

Analysis and Reporting Tools

Generating logs is one thing, analyzing them another. Notepad is useful for a quick peek at log files that have small amounts of information in them, but web servers running in service provider or e-commerce environments need something more powerful for analyzing and generating reports from IIS logs. A good tool to look at is WebTrends Analysis Suite from NetIQ (www.netiq.com), which is powerful but a bit pricey. At the other end of the spectrum are free utilities like Analog (www.analog.cx) and inexpensive tools like Active LogView from SoftCab (www.softcab.com). You can also download the free Log Parser 2.0 command-line tool from Microsoft's website (www.microsoft.com) and perform SQL-like queries on IIS log files and display the results.

Converting Log Formats

The different web log formats supported by IIS are not compatible with one another. For example, while both NCSA and W3C Extended logging record dates using four- digit year format, IIS logging records dates earlier than 1999 as two-digit years, and four-digit years thereafter. Furthermore, IIS log file format separates fields using commas, but NCSA and W3C Extended formats use spaces as separators instead. Another difference is that while W3C Extended records event times using UTC (coordinated universal time, sometimes called Greenwich Mean Time or GMT), the IIS and NCSA formats employ local time instead.

To help you compare web logs generated using different log file formats, Windows Server 2003 includes the utility convlog.exe in \Windows\System32 that can be used to convert log files that use W3C Extended or IIS log file formats to standard NCSA format. The conversion process automatically performs a time zone offset (using an offset you specify) in order to change the times in the log from UTC to local time, making them easier to read and interpret. In addition, this utility can be used to replace IP addresses in log files with DNS domain names associated with them. For syntax on how to use this tool, type convlog /? at the command line or see IIS Help.