Logging and Monitoring

To maintain the Rule of the Three-Fold Process, logging and monitoring should be implemented in objects all across the environment. This includes objects like servers, routers, and applications, as well as physical areas. Having a firewall go unmonitored is exactly the same as parking a locked car in a dark alley with no one around. Every security device has a weakness, and if an attacker goes unnoticed, the attack will eventually be successful. The unwatched thief will eventually find a way into your car, and the unwatched hacker will eventually find a way into your networks and systems.

What to Log

Earlier we discussed the concept of creating stillness, which should be reviewed for logging purposes. It is extremely important that we log activities that are useful and filter out excessive noise. At the same time, it is important to avoid filtering out so much as to miss a potential attack. Suggestions for how to accomplish this were included in Chapter 5.

Centralizing Logging Efforts

It is nearly impossible to effectively monitor multiple objects if we must continually reference multiple sources for information. A small organization may be able to get away with every system performing its own logging, but for medium and large organizations, it is important to centralize the logs in one or two monitoring devices. By centralizing the logging effort, we will be better able to:

• Provide a single point of maintenance By centralizing our logging efforts we create a single location in which our log maintenance functions are performed. In medium and large environments, checking logs on each individual system is extremely time-intensive, and normally results in systems being skipped and important events being missed. Centralizing log review and administration will be a cost savings for most organizations.

• Correlate events We will rarely find unauthorized activities that affect only one system or device. When good logging practices are followed, hackers will leave a trail of events that will help us to better understand the nature of the attack and the extent of the damage. Centralizing logs from different objects gives us the power to correlate different events and derive a more complete picture. Logs can be grouped by time, similar events, neighboring systems, and in a variety of other ways to track a hacker's steps.

• Review in real-time If we are looking at logs in multiple locations, it will be impossible to monitor all objects at the same time. Thus, while looking at the logs of System X, we cannot see an attack appearing On System Y. Similarly, if an event occurs on both System X and System Y at the same time, we may not be aware of the correlation. Centralizing logs allows us to view events in real-time across the organization.

• Build archives It is important to archive logs for historical records. Archiving logs is very difficult to do if they exist in many different areas. There is a much greater chance that a log will be missed if we must rely on 10 systems backing up their own logs. It will also be hard to retrieve and organize such logs. Centralizing logs allows us to implement consistent and efficient solutions for archiving event information.

• Provide for disk management Log files are notorious for filling up hard drives over long periods of time. If not properly configured, such events can cause a critical system to crash or overwrite important events. By centralizing logging issues, we can remove the local log immediately, thus helping to protect the systems without having to manage individual log files.

There are many methods for centralizing log activity. Most devices are able to send events through SYSLOG, or through an SNMP trap. There are a variety of tools, ranging in price from free to expensive, that will capture logs from different devices, correlate events, and allow for real-time monitoring. Many centralized SYSLOG applications that can retrieve and parse logs from multiple devices are freely available.

There are also numerous commercial products that provide advanced reporting, Web-enabled browsing, and greater control over the presentation of logs. Some of these products specialize in one or two technologies; for example, one may specialize in firewall and IDS devices, while another specializes in operating system events. For large enterprises, purchasing two logging applications usually saves a great deal of time, effort, and cost in keeping track of security events.

Correlating the Logs

When centralizing logging efforts, correlating among different logging devices is crucial. If we receive a log from one device and a completely different log from another device for the same event, we must be able to correlate a relationship between the two. Lucky for us, software developers have been getting better and better about looking at logs from a standard view and correlating logs is not the horrible chore it once was. That being said, there are still crucial areas we must be aware of when centralizing logs.

Classify Similar Levels of Events

Different events are most often given different classifications from different reporting devices. For example, an IDS may report a TCP sweep as a medium-level event, warranting some attention, while it reports a strand of KLENZ as highly important, warranting immediate attention. Such classification systems are designed to provide some consistent method of prioritizing events and to help focus attention on those events that are the most important.

There are two problems with such classifications that we should be aware of as we go forward in the logging and monitoring process:

• The default classification for an event may not correspond to the environment. Something the product reports as "low" may be more critical within the environment, whereas something considered "high" may not be so important.

• Different products have different levels of classification. One product may report a TCP sweep as "high," where another product reports similar activities as "medium" or "low."

It is beneficial to make the centralized logging solution as clear and logical as possible. A high-level event from a firewall should mean the same to the organization as a high-level event from an IDS. A critical event from NT should correspond to a critical event from Solaris. This way, it is much easier to assess threats, filter logs, and search for potential security issues. Many devices permit the administrator to assign a level to each event, and many centralized logging products also provide a virtual representation of events, which allows us to correlate events from different devices.

Synchronizing Time

Running different logging devices off their own internal clock becomes a big issue when dealing with log correlation. Often, it is important to know where an attack started, which device picked it up first, and which events occurred at the same time. Devices that run off their own clock will always get out of sync with each other, causing chaos in the logs. Events occurring right now may appear at the bottom of a report if that device has fallen 30 minutes behind. Similarly, devices on the West Coast may never appear on a screen because they are logged as three hours before East Coast devices.

It is important in log correlation to make sure all logging devices are running off the same clock and in the same time zone (or in a program that compensates for time zone differences). One solution for this is to maintain a Network Time Protocol (NTP) server within the organization. Just about every modern device can synchronize its time with an NTP server, allowing for a consistent view of time across the organization. NTP is normally a low-cost solution that really helps with managing security. NTP services use minimal processing power and come free with many operating systems, networking products, and Open Source packages.

Archiving Logs

It is important to keep a history of logs and to not throw any away. We never think we will use something until an emergency comes along and we need it. When a hacker breaks into a network, he or she may not be discovered for several months, in which case, we will want to have the logs from the last several months on-hand to track activities that occurred since the penetration.

Luckily, logs do not normally take up a great deal of space, especially when compressed. Logs are usually easy to remove from a system and store on a CD or tape backup for future use. Some key points to consider, however, are:

• Store all logs, even mistakes Oftentimes, we are tempted to erase log files, especially when we find that we have been logging more than we need to. However, it is important to maintain all logs if possible. It should be made policy that logs from the centralized system cannot be removed without first reporting, approving, and documenting them.

• Make archives consistent Archives should be consistent with other backups. Log archives should be taken every week, month, or at whatever interval makes sense. When performing an investigation, logs should be easy to find despite the millions of logs in the archives.

• Make sure logs are retrievable Some log correlation applications require administrators to go through some specific steps before performing an archive. Some applications will not allow for an old log file to be viewed or restored unless it was archived using a specific process.

A Note on the Legality of Log Archives

Using logs as evidence can be tricky. For some organizations, maintaining logs as legal records is important. In such cases, it is important to maintain the integrity of logs by performing a hash function (like MD5) on each log and storing that value in an encrypted file. Then, all access to the log archives and the encrypted hash needs to be logged to help prove that no one could have tampered with the archives. For a great source on making logs and other electronic evidence legal, see Mandia and Prosise's book, Incident Response.