Chapter 10. Managing the Audit Trails

Raspberry. There's only one man who
would dare give me the raspberry: Lone Star!

Dark Helmet
Spaceballs

The word audit usually makes people a little nervous; even when they have nothing to hide. An audit, in the world of accountants, is to examine an individual or organization's financial records formally. The goal of an audit is either to validate that people or organizations have followed the letter of the law, or uncover their horrible misdeeds. The success of an audit must be based upon records of transactions. Without these records, performing an audit requires far more detective work or is rendered impossible.

In the computing world, audits can be formal or informal interrupt-driven processes performed by system administrators to answer questions. A question like "Why haven't we received the mail our client sent?" sends administrators scurrying through mail logs. A more difficult question to answer might be, "Why didn't that dynamic web page load right?" because web server access logs, error logs, and database query logs may need to be consulted to build a complete picture of what transpired. In a security context, an incident response team conducts an audit to try to uncover any transgressions and perform root cause analysis. All these questions and mysteries can be solved . . . as long as you have a record of the transactions, or logs.

A part of our job as system administrators is to keep an eye on the systems we have built and that we administer. We explored the topic of system health monitoring as one of the ways to do this in Chapter 4. An aspect of a system's overall health in some cases is useful as an indicator of security breaches. Likewise, a system's logs often contain events pertaining to the security of the system: unauthorized login attempts or connections, frequent application crashes, mail relay attempts, and attempts to write to read-only FTP folders are examples of events that might spur an investigation. Automated monitoring of logs helps us guarantee that these events will be noticed and an investigation, or audit, can then ensue.

This chapter is about monitoring our systems through log analysis and ensuring that we're able to answer questions that might arise. To accomplish this task we must first ensure that our systems are generating the logs in which we're interested. The logfiles themselves are of moderate use already, but consolidating them on a single host helps ensure their integrity and perform audits more effectively. Finally, with a single storage location for logfiles, we can deploy a log monitoring system to alert us when unusual events occur. We cover each of these topics in turn.