The Importance of Network Log Files

In many environments, system administrators largely ignore network log files; the administrators are constantly putting out fires and don't have time to devote to log file review. Actually, it's more accurate to say that the files are ignored until a major incident occurs. Then there's a mad scramble to find out what happened, which typically includes a check of log files, often done by an administrator who isn't very familiar with the log file format or the proper procedures and steps for performing good log file analysis. Sometimes the administrator discovers that detailed logging was not enabled, so no record of the event exists. After the crisis has ended, the log files are again ignored until the next catastrophe occurs.

Log files have many things to tell you, if you only stop to listen to them. Log files have several purposes of which you might not be aware, which will be reviewed in the "Purposes of Log Files" section later in this chapter. They are incredibly helpful in meeting a range of needs, such as intrusion detection, incident handling, event correlation, and troubleshooting. Each log file contains many pieces of information that can be invaluable if you know how to read them and, more importantly, if you know how to analyze the data from a perimeter defense viewpoint to identify scans, intrusion attempts, misconfigured equipment, and other noteworthy items. Before you can learn how to analyze the data, you first have to understand what sort of data you will likely have.

Characteristics of Log Files

Many different types of log files exist, and they are generated by various sources. Most operating systems, including all flavors of UNIX and most versions of Windows, are capable of performing extensive logging of events as they occur. Also, many applications log events of significance, such as authentication attempts, application errors, and configuration changes. For our purposes, we are most interested in analyzing network log files; to understand the events that occur on a network, you want to focus on log files that record network-related events, such as successful and failed connections. Many other logs (such as operating system log files) also contain valuable information that can be used to correlate the network activities, although the primary focus in this chapter is the examination of network logs. Devices such as firewalls, routers, and network IDS generate network logs.

Note

Although operating systems and many applications have great logging capabilities, they are often disabled by default. System administrators must take care to ensure that logging is enabled and that the logs are sufficiently detailed.

If you have ever looked at log files from several different devices or applications, you have surely noticed that the logs are typically in completely different formats. Later in this chapter, starting with the "Analyzing Router Logs" section, we will review some of the most commonly encountered log formats. Although it's important to realize that log formats differ, it's far more important to understand that different devices and applications might log very different information. Some logs contain little useful data, perhaps only a timestamp and the source and destination addresses and ports, whereas others contain just about everything you would possibly want to know about the event in question. Some logs record virtually every characteristic of traffic, plus an interpretation of the significance of that traffic. The two main factorsvarious log formats and different log informationare what make network log analysis a bit intimidating at first. Let's work through that by examining what data you might find in network log files.

Information That Log Files Usually Record

Most systems that utilize network log files record several core pieces of information about each connection or packet that they deem to be of interest:

• Timestamp, which typically includes the date, as well as the time (in seconds or fractions of a second) when the event occurred or when the event was recorded to the log

• Basic IP characteristics, such as source address, destination address, and IP protocol (TCP, UDP, ICMP, and so on)

• TCP or UDP source and destination ports, or ICMP type and code

Additionally, most log entries contain some sort of reason why the event was logged. However, this varies widely among log formats:

• Descriptive text explanation

• Rule number that matched traffic

• Action that was performed, such as accepting, dropping, or rejecting a connection

Although this might not seem like much data, you can do a lot with just this information. For example, you can analyze it to identify port scans, host scans, or other failed connection attempts. You can also do basic traffic analysis, but with just these elements, the amount of analysis you can do is quite limited. You need more data on events to perform a more in-depth analysis.

Information That Log Files Sometimes Record

Many network logs do record more information than just the core items. Other types of data that might appear include these:

• Other IP characteristics, particularly the IP identification number and the time to live (TTL)

• More TCP-specific characteristics, such as flags (SYN, ACK, and the like), TCP window values, and TCP sequence numbers

• The interface that saw the event

• The beginning of the payload's content, although this is much more the exception than the rule

These additional fields can be beneficial in terms of log analysis. For example, the time to live and TCP window values can be used to perform OS fingerprinting to determine the likely OS of the attacker (although attackers can modify telltale values such as the TTL). TCP flags might indicate the intent of an attacker; certain flag combinations tend to mean that the packets are trying to evade firewalls or IDSs. Some of these extra fields can also help determine what tool or malware is generating the attack. Many tools have a unique signature, and these data elements can help you to find that signature so that you know exactly what the attacker is trying to do. Unfortunately, even with this extra information, you still might not have enough data to perform as detailed of an analysis as you would like to do.

Information That Log Files Rarely Record

Most devices that perform network logging do not closely examine or even record the full payloads of the traffic they see. Doing so is generally outside the capabilities of the device, and it is incredibly resource-intensive. Most network logging is based on examining the core characteristics of the packets and connections and making decisions based on those characteristics alone. There's little need, from the perspective of that device, to examine or record all the packets in a connection.

However, it's often beneficial to have more information on traffic and to be able to do more careful evaluations of payloads. It's often useful to record full packet headers as well to capture all the pertinent information, not just the header values stored in the log. If full packets are recorded for connections or packets that are deemed suspicious, you could then perform a deeper examination as needed and have all the packets' information available. Unfortunately, many logs do not support packet recording; in these cases, you might be able to set up a dedicated packet sniffer or use a program such as Tcpdump to perform packet captures. However, in most environments, the volume of traffic and required storage space will be too high to record all packets, and organizational policies might forbid the recording of all traffic for privacy or liability reasons.

Most devices that perform network logging do not also perform protocol decoding at the application layer. A good example of this would be a DNS request. A network log might report that host 10.20.30.40 sent a packet to UDP port 53 on host 172.30.128.12. You see port 53 and think "DNS," but you really have no idea whether this was a DNS request. Even if it were a DNS request, you have no information about the request. With the exception of a proxying firewall, devices that perform network logging usually don't do protocol decoding or verification. You must rely on network intrusion detection and prevention systems, as well as proxies, to perform protocol verification.

Note

One great feature of proxying firewalls is that a web proxy can log URLs. If a new worm that exploits HTTP is spreading, other perimeter devices may log port 80 connection attempts, but the web proxy will actually log the URL that the worm uses, which can be invaluable in determining what is occurring.

Purposes of Log Files

Now that you have looked at the types of data that log files record, let's step back for a minute and think about the purposes that log files serve. Of course, the main purpose of a log file is to record events of significance or interest. But what you would really like to know is how these records can be useful to you and your organization. As you will see, log files have several important roles to play and can assist you in such areas as incident handling, intrusion detection, event correlation, and general troubleshooting.

Incident Handling

Probably the most obvious use of network log files is to provide data that can be used for incident handling. For example, if a network administrator receives a report that a device has been compromised, she might use network log files to determine which host or hosts might have been responsible for the attack and what methods of attack might have been used. When a web page is defaced, the system administrators are likely to consult the web server's logs, but also the logs of other devices through which the malicious traffic might have passed, which could provide additional information about the attack and the attacker. Network log files are an invaluable component of incident handling.

During incident handling, you will need to preserve evidence should you want to perform disciplinary actions against someone within an organization or pursue legal actions against an external party. Under those circumstances, log files are extremely important for forensic purposes. When properly preserved, network log files provide evidence of activityevidence that might be admissible in legal proceedings. The procedures you need to follow for evidence handling vary widely depending on where you are located, but a general guideline is that the original, unaltered log files should be preserved, preferably both electronically and on paper. Contact your local law enforcement agency for advice on how to preserve log file evidence. For additional information on incident handling, see Chapter 19, "Maintaining a Security Perimeter."

Intrusion Detection

Although log files are invaluable during the handling of many incidents, this is a purely reactive use of log files. A related proactive use is that of intrusion detection. By automatically and continuously monitoring log file entries, you can be notified when someone is scanning your network or performing reconnaissance, or when an actual incident is in progress. This will also help with incident handling; when a significant intrusion is detected and reported to you, you already have much of the data you will need to perform incident handling for that event.

Note

Even the simplest firewall and router logs can be used for some basic intrusion detection. For example, blocked packets or connections can be counted by source or destination IP address or by destination port, which is useful in identifying scans, probes, and other reconnaissance attempts.

Event Correlation

Event correlation is useful in performing both incident handling and intrusion detection. When we speak of event correlation, we mean that it's possible to use multiple logs from various devices or applications together. Event correlation can be done to confirm what has occurred. For example, you see a suspicious entry on an internal router's log that involves an external host, and you search the logs of the Internet firewall for entries that provide more information on the action. Another use of event correlation is to relate events to each other. If your email server is compromised, you would search through various network logs from routers, firewalls, and other devices to look for any evidence relating to the compromise, such as other connection attempts made to the email server, or other hosts on your network that have been targeted by the same attacker.

General Troubleshooting

A final use of network log files is to assist in general troubleshooting, particularly involving connectivity issues. For example, a user complains that application XYZ can't download data from an external server. By getting the IP address of the user's machine and finding out what time he tried to use the application, you could quickly search your firewall's logs to look for denied attempts to make the required connection. If your firewall also logs all permitted connections, you could look for a valid connection to the remote site, which would indicate that the problem most likely involves the remote server or the application, not your perimeter defense configuration. More information on troubleshooting is presented in Chapter 21, "Troubleshooting Defense Components."

Note

In most environments, configuring your firewall to log all permitted connections is not possible due to the negative impact on performance and resources. However, temporarily configuring your firewall to log all connections is often helpful when troubleshooting a problem.