Log data can be structured in a variety of ways, including flat files, databases, and direct input from the system to a formal monitoring system, such as HP OpenView or the Windows event log.
Here are some general observations on log format.
The flat format is probably the most popular for log files. It's faster and more malleable than a database and usually trivially ported. Indeed, when I did a casual search on my laptop for .INI files, (mentioned in Chapter 13) I also did one for .LOG files. Once again I was surprised by the results. I had expected to find, at most, a dozen or so files. Instead I found 132, created by programs ranging from MS Outlook to Adobe Acrobat and my REX6000 PDA.
Here are some principles of good flat file design.
Log management refers to such things as how and when logs are created, updated, and/or deleted and responses to error conditions. Consider the following.
Complex systems benefit from the ability to configure logging parameters (such as level of detail or kind of operations to log) during operation. This allows customer support to work with customers in real time to resolve key issues. An even more sophisticated approach is to construct your system so that it can decide if you need to log data. If your fraud detector notices that something is amiss, have it log additional information for awhile.
Complex servers that distribute operations over multiple architectural layers often service each request in a separate thread. The net result is that log entries from several different components , each running multiple threads, may have to be correlated to produce useful results. If you're using multi-threading , make certain you support per-thread logging.
In addition to starting/stopping logging, consider making the level of detail configurable. This is often expressed as logging levels, where you set a number to indicate the amount and kind of information that should be logged. For example, you might set developer logging to the highest number and default logging to something else. This helps prevent your logs from filling up with data that is only needed in specific or special circumstances.
A complementary approach to logging levels is to label the various logging data. These labels can be used in conjunction with levels to create a very flexible system. Here are some examples of labels.
Logging levels and labels should work in conjunction with the categories defined earlier. Thus, a warning entry for performance purposes might be generated when the system takes longer than expected to process an event, whereas a warning entry based on operational status may be generated when the system becomes low on memory or disk space. Not all combinations make sensethere is no good definition of a warning or an error for behavioral tracking and usage (unless you're concerned about a feature being over-used).
Controlling the behavior of the system with logging levels and labels should be through an appropriately designed set of APIs.
There are times when a different format is easier to process. For example, Web server logging facilities, like those found in Apache, are completely configurable in syntax and contents.
Exceptions and logging are closely related. Log every exception.
Many logs outlive their usefulness , unnecessarily cluttering machines and potentially confusing log consumers. If the date the log was created is part of the file name, removing unnecessary logs becomes that much easier.
Log data can be sensitive, so you may have to encode it or store it in a privileged location. Depending on the contents of the log, customers may not be willing to share it with your company.
Conversations between technical support and customers can become pretty tense when something goes wrong. Asking your customers to collect and forward log files to your support organization may result in additional frustration: Nontechnical customers may not know where log files are, and they may not know which ones are needed or how to send them. To make things easier for your customer, consider providing facilities to automatically capture and send log data to technical support. You'll not only get better results, you'll get happier customers.
Logging Standards and Libraries
Logging standards are either platform independent (such as the W3C Extended Log Format or the Common Log Format for Web servers) or platform dependent (such as Windows Event Logs). Basing your log files on these standards enables you and your customers to use a wide range of freely available analysis tools. An alternative is the several logging libraries available, such as log4j for Java. You should roll your own logging facilities, formats, or external viewers only after you've proven that the readily available, and often no-cost tools won't work for you.