Log Format and Management

Log data can be structured in a variety of ways, including flat files, databases, and direct input from the system to a formal monitoring system, such as HP OpenView or the Windows event log.

Log Format

Here are some general observations on log format.

• Internationalization: Creating log data in the user 's target language can substantially improve usability. It can also score points with customers who prefer to use their native language for all system operations.

• Start with a time stamp: The first entry in a log file should be a time stamp.

• Always add a unique identifier, traceable to the actual source location: Log data are most useful when you know you can return to the source code to interpret what is happening.

• Provide a way to identify transactions: Many log entries are associated with various kinds of transactions. Once an ID is assigned to a transaction, put it in the log. This is the best way to tie together a single transaction's logging data.

Flat Files

The flat format is probably the most popular for log files. It's faster and more malleable than a database and usually trivially ported. Indeed, when I did a casual search on my laptop for .INI files, (mentioned in Chapter 13) I also did one for .LOG files. Once again I was surprised by the results. I had expected to find, at most, a dozen or so files. Instead I found 132, created by programs ranging from MS Outlook to Adobe Acrobat and my REX6000 PDA.

Here are some principles of good flat file design.

• Easy parsing: The contents of the flat file should be easily parsed. A little thought here can save a lot of work later. A good rule of thumb is that it should be trivial to import log data into a relational database or analysis tools like MS Excel.

• Easy reading: Log files that are easily parsed are oftenbut not alwayseasy to read. Again, with a little bit of planning you can create log files that are parsed easily by both computers and humans .

• Sensible location: Don't put log files where a user won't expect them, such as the root directory. Put them in a sensible location, preferably one associated with your application or that can be configured by the user/system administrator.

• Sensibly named: I prefer that log file names be easily understood , but sometimes this just isn't possiblefor example, when a separate log file is created for each invocation of an application. In these cases, you may want to generate a random name for the file, put it in a sensible location, part of which should be the date the log was created. The format YYYY-MM-DD-HH-MM-<log file name> has the advantage of automatically sorting log files by date (using a 24- hour clock).

• No garbage or special characters : Avoid putting anything into the log file that isn't a "normal" character. Use Unicode characters if you're concerned about multibyte languages.

• Documentation: Like configuration files, log files need precise documentation to ensure their proper use. Explain what logs are created and when and how they're created, what different settings will produce, and so forth.

• Reconciliation or audit ID: If you're using different log files, create some kind of identifier than can tie together related entries. If at all possible, use this identifier with third-party applications.

Log Management

Log management refers to such things as how and when logs are created, updated, and/or deleted and responses to error conditions. Consider the following.

Dynamic Logging

Complex systems benefit from the ability to configure logging parameters (such as level of detail or kind of operations to log) during operation. This allows customer support to work with customers in real time to resolve key issues. An even more sophisticated approach is to construct your system so that it can decide if you need to log data. If your fraud detector notices that something is amiss, have it log additional information for awhile.

Per-Thread Logging

Complex servers that distribute operations over multiple architectural layers often service each request in a separate thread. The net result is that log entries from several different components , each running multiple threads, may have to be correlated to produce useful results. If you're using multi-threading , make certain you support per-thread logging.

Logging Levels

In addition to starting/stopping logging, consider making the level of detail configurable. This is often expressed as logging levels, where you set a number to indicate the amount and kind of information that should be logged. For example, you might set developer logging to the highest number and default logging to something else. This helps prevent your logs from filling up with data that is only needed in specific or special circumstances.

A complementary approach to logging levels is to label the various logging data. These labels can be used in conjunction with levels to create a very flexible system. Here are some examples of labels.

• Debug: usually reserved for extended data; turned on when conducting error diagnosis

• Info : provides noteworthy information

• Warning: logs an entry when a potential error condition has been detected, such as when the system has detected a lower-than-desired amount of a critical resource

• Error: logs an entry when an error has occurred

Logging levels and labels should work in conjunction with the categories defined earlier. Thus, a warning entry for performance purposes might be generated when the system takes longer than expected to process an event, whereas a warning entry based on operational status may be generated when the system becomes low on memory or disk space. Not all combinations make sensethere is no good definition of a warning or an error for behavioral tracking and usage (unless you're concerned about a feature being over-used).

APIs

Controlling the behavior of the system with logging levels and labels should be through an appropriately designed set of APIs.

Configurable Syntax

There are times when a different format is easier to process. For example, Web server logging facilities, like those found in Apache, are completely configurable in syntax and contents.

Log Exceptions

Exceptions and logging are closely related. Log every exception.

Removable

Many logs outlive their usefulness , unnecessarily cluttering machines and potentially confusing log consumers. If the date the log was created is part of the file name, removing unnecessary logs becomes that much easier.

Security

Log data can be sensitive, so you may have to encode it or store it in a privileged location. Depending on the contents of the log, customers may not be willing to share it with your company.

Automatic Forwarding

Conversations between technical support and customers can become pretty tense when something goes wrong. Asking your customers to collect and forward log files to your support organization may result in additional frustration: Nontechnical customers may not know where log files are, and they may not know which ones are needed or how to send them. To make things easier for your customer, consider providing facilities to automatically capture and send log data to technical support. You'll not only get better results, you'll get happier customers.

Logging Standards and Libraries

Logging standards are either platform independent (such as the W3C Extended Log Format or the Common Log Format for Web servers) or platform dependent (such as Windows Event Logs). Basing your log files on these standards enables you and your customers to use a wide range of freely available analysis tools. An alternative is the several logging libraries available, such as log4j for Java. You should roll your own logging facilities, formats, or external viewers only after you've proven that the readily available, and often no-cost tools won't work for you.

Be Careful about Repurposing Log Data

Users often discover a variety of uses for log files. Unfortunately, these new uses can subtly conflict with a file's original functions, causing any number of problems. In one system I worked on we recorded various user operations for performance monitoring. It was suggested that we could also use them for billing. While it appeared that all of the data were available, this would have been a dangerous choice. Because the logs were simple flat files, they could be easily changed by the user, completely changing their bill. We had to come up with a different tracking mechanism for billing.