Selecting and Using System and Network Logging Mechanisms | Protect Your Information with Intrusion Detection (Power)

Collecting and analyzing the data generated by the network, DBMS application or the operating system is a powerful and efficient method of intrusion detection. Further, this is sometimes the only possible approach. Log files contain information about all system, network or other events that have occurred during a specified time period. Neglecting the information from these logging mechanisms will significantly weaken your security system, to the point that it can render it virtually useless [CERT1-00]. Log files are necessary for:

Notifying administrators that there has been an attack
Helping to restore the system after an attack
Performing investigations
Obtaining proof that unauthorized actions are being performed

Initially, administrators and security professionals performed this analysis manually. Security personnel had to analyze large printouts in order to locate events indicating unauthorized activities. Naturally, by the time these events were located, it was already too late to stop the attack. Gradually, with the advances in information technologies, methods of automated analysis of log-file information appeared, and these methods were included in most commercially available system

Logged Information

Selecting the information that needs to be analyzed is the first step in using logging mechanisms. Next, it is necessary to define the logging mechanisms and the points where the logged information should be collected. Finally, the locations for storing the collected information must be defined.

Table 7.2 contains a general list of the most common types of log files and events registered there. Note that not all systems are capable of registering the information outlined in this table. This is especially true for custom application systems, which are generally designed without taking into account security considerations. On the other hand, it is pointless to register any useless information in a log file. If you are developing a custom application system with security in mind, you have to find a reasonable balance between the amount of data logged and the performance of the controlled system. This is necessary to avoid situations in which all of the controlled system's resources are being consumed by the task of logging an excessive number of events. This is where the network map becomes indispensable, since it contains a list of all information resources in the corporate network and specifies their level of importance. As a rule, the more important is a specific resource, the more information about this resource it is necessary to register. Further, the network map provides an understanding of the function performed by the controlled host, which simplifies the choice of information that needs to be logged. For example, if the host is a web server, then the actions of logging on to the server and accessing various HTML pages and CGI scripts are of primary importance.

Table 7.2. Types of Log Files and Information Contained There
Log-file type	Categories of logged information

User activities	System logon and logoff events (failed attempts, time and location of the logon or logoff), attempts to logon as privileged user
	Any attempts to change user information (password, privileges).
Process activities	Time and arguments of process startup
	User who started the process
	Time of process termination, termination status, duration of the process and the resources it accessed
	Files opened by the process and devices it accessed
System events	Actions requiring special privileges
	Errors and warnings issued by the hardware, software and file system
	Activities related to modem connections
	Any changes in the system status, including events such as shutdown and rebooting
	Information on system workload (average and peak values of CPU work-load, consumed and free memory and disk space)
Network events	Information on attempts to establish connections with the system (who, when, from where, using which protocol and during which time) and on any open connections
	Additional data (network packet headers, packet size, contents of the data field, network interface, rule)
	Information on the network load (including the number of network errors)
	Network traffic information (workload by percentage, bytes, connections etc), sorting by various criteria (protocol, source and destination addresses)
File access	Any changes in file attributes (access rights, CRC, size, contents, path, permissions)
	Access (type, access time, duration, user or process name)
Application activity	Application-specific information (for example, mail traffic, firewall events, financial transactions etc.)

The Sufficiency of Built-in Logging Mechanisms

Determine which logging mechanisms are available in your system, what the names of the log files are and where are they stored. Note that the names of these files might differ from version to version of the same operating system.

Focus on creating a situation in which your logging mechanisms (both built-in and auxiliary) cover the entire range of events described in Table 7.2 by determining more precisely which categories of information are processed by specific logging mechanisms. It is most effective desirable when these logs even overlap. Although this will result in some redundancy of information stored and increase the space required to hold it, at the same time, it will increase the probability of detecting unauthorized activities. If the available logging mechanisms are insufficient (this is most likely to be the case if you use only the OS built-in logging capabilities), consider the possibility of installing auxiliary tools to extend and enhance the built-in logging function. This is particularly important when dealing with Microsoft operating systems, which do not generally log all of the necessary information.

Event Logging

At first glance, enabling the event-logging mechanism appears to be an easy task. However, this process is not as simple as it might seem. For example, you must determine when the logging mechanism will begin functioning at once (at system startup, as the case for Windows NT-based operating systems, or at specified times, as is the case, for example, for the Tripwire consistency checker or for System Scanner security scanner?) On top of this, you must also bear in mind that different logging tools provide different levels of detail when logging information. For example, CheckPoint Firewall-1 provides two options: Short Log and Long Log. The RealSecure Network Sensor system also permits two logging modes. Beside the classical logging mode (log without raw option), when only the name, date and time of the detection of an attack are stored in the log file, there is an enhanced mode (log with raw option), in which, in addition to the standard information, the logging mechanism also registers the contents of the entire traffic or all commands issued within the range of the registered security event. This mode enables an administrator to analyze the methods used by the intruder and to take efficient countermeasures. Be careful not to overuse the enhanced logging mode, however, since it will result in the log file quickly becoming full of useless data, which will consume a considerable amount of the storage space.

Pay special attention to the locations of the log files and their access rights and privileges. You must have a sufficient amount of free disk space for storing these files, or some records will not be logged. As a result, some of the collected data will be lost, and you will be unable to determine with any certainty whether the security policy has been violated.

Protecting Log Files

Since log files contain very important information that must not be accessible to any-one except for security administrators, it is necessary to protect them from unauthorized access. It would be best if even the administrator could not access these files directly, but only via special tools intended for log-file processing. This will prevent the administrator from deleting or modifying specific records without authorization. In this section, I will describe the proper methods for protecting the collected information:

Store logged information on the dedicated host and make sure that appropriate measures of physical security are taken. Beside this, limit access to this host via the network.
Write log files to write-once storage media (for example, CD-ROM) to eliminate the possibility of modifying this data. As well as this, you can also print out this information or register the output of "in-process" activity logs with devices that do not allow data modification. The last option will complicate the analysis significantly. Simply using a printer to provide a backup mechanism in the event of difficulties is sufficient.
Prevent the possibility of changing any records already existing in the log files.
Use encryption mechanisms to protect log files from unauthorized modification.

Storing log files on the hard drive is the simplest method. On the other hand, this method is also the least reliable. Despite this, it is the most commonly used method. The use of add-on encryption tools to protect log files stored on the hard drives improves reliability. However, this method has its drawbacks. In particular, since log files are constantly appended, integrity control becomes exceedingly difficult. During peak workload hours, dozens of records might be appended to the log file simultaneously, which might make the usage of encryption methods impossible. The usage of write-once storage media for recording log files is more complicated from the technical point of view, but, at the same time, is more reliable. If it is impossible to work with these media in real-time mode, consider recording the collected information periodically, when log files grow to a predefined size or when specific criteria are satisfied (for example, after a specified time period has elapsed).

If the host generating log-file records is different from the host for which these records are generated, you will need to provide protection for the entire connection through which the registered data is transferred from host to host. If the distance between two hosts is not great, it is a good idea to establish a direct cable connection between these two hosts (without any intermediate hosts). However, if the distance makes this option unrealistic (which is most often the case), you will need to use other methods. First, it is best to minimize the number of intermediate hosts along the route of log-file information, which means that you will need to customize the routing table. You will probably even dedicate a special VLAN for this purpose. Second, use encryption mechanisms to protect log-file information transferred via the network.

Another important consideration is the appropriate customization of the logging-mechanism settings. This is necessary to prevent any attempts at disrupting the operation of the logging mechanisms by organizing, for example, DoS attacks. For example, in UNIX, or in Windows NT/2000, depending on the logging mechanism settings, logging either stops or the older records of the log files are overwritten. To organize an attack on the logging mechanism, the intruder can attempt to generate a large number of events to fill the log files, and then perform unauthorized actions, which will not be registered anywhere. Filling up the log files can also bring the host down. To provide protection against this type of attack, you can employ different mechanisms. One option is to create several log files storing different kinds of information (like in Solaris). You can also download the data from the log file to a protected location or to the central management console. The Microsoft Operations Manager system functions according to this principle. The RealSecure Network Sensor system also operates in a similar way. As log files at each sensor are filling, RealSecure dynamically synchronizes them to the database stored on the central console (Fig. 7.2).

click to expand
Fig. 7.2. RealSecure synchronization mechanism

Log-File Management Plan

All aspects related to the logging of all required information must be documented [CERT2-00]. The names of the resulting several sections correspond to the sections of the log-file management plan.

Managing the Volume of the Logged Information

It is usually recommended to log as much information as possible. However, as already mentioned, you should not follow this recommendation to the extreme.

First, this is the case because the amount of logged data will grow rapidly. Note also that it is difficult to predict which information will be the most important in case of attack. Initially, when enabling the logging mechanism, you will not be as aware of which events are the most important, and you will be forced to register them all. Some time later, when analyzing log files, you will understand which categories of events are critical and which are less important. Gradually, you will be able to stop logging events that have no relation to the security of the protected host.

Second, logging a large number of events has negative impact on the performance of the protected system. Therefore, it is highly recommended to find a balance between the amount of logged information and system performance. It is impossible to produce a universal set of practical recommendations in this case, and you can only achieve the necessary balance by assessing the operation of the system in actual conditions. To reduce the amount of information stored in log files, you can use physical or semantic compression methods. This way you will reduce the amount of free disk space reserved for storing log files and simplify the procedure of log-file analysis.

Assigning Priorities

If, while creating a network map, you have not determined which resources in your network are the most critical, you should do so now. Focusing your attention on the most important resources in the corporate network will enable you to use logging mechanisms more efficiently. Providing answers to the questions below will help you to solve this problem:

Which functional tasks are delegated to the host or network segment? If this is a firewall, you should concentrate your attention on logging events related to the firewall. If this is DBMS server, then you must pay the most attention to events related to data input, output and data modification.
How many users have access to this host? This will help you to evaluate the amount of logged information related to user logins and logouts.
Will log files be used to restore the compromised system? This will help you decide whether you need to enable the transaction rollback mechanisms when logging events.
Which programs and services can run at this host?
What is the minimum acceptable host performance? Knowing this, you will be able to determine the resource requirements necessary for timely logging and processing of the collected log information.

Log-File Rotation

Log-file rotation includes periodic creation of the backup copies for all log files and saving them at a physically protected location, as well as periodic cleanup of the log files.

These actions will allow you to reduce the amount of data that needs to be analyzed in real-time mode in order to detect unauthorized activity while, at the same time, reducing the potential danger in the event of compromising the currently active log file.

Backing up the log files will allow you to access them whenever you need to. However, do not forget that you need to perform this operation before they are cleaned to economize space, since some part of the information will be lost.

You should encrypt log files in order to protect them from unauthorized access. Encryption should be applied both to the current log files (if this is possible) and to backup copies. Note that, as opposed to the current copies, for which encryption is not mandatory (especially if the system writes vast amounts of information into log files), backup copies must be encrypted.

When performing such routine operations (such as clearing log files), you must ensure that the information stored in these files is reliably wiped away. If you store log files on the hard disk, it is recommended that you use specialized programs for this purpose (such as Wipe). Write-once media that stores backup copies of log files can be physically destroyed.

Joining Log Files

In some cases, you may want to unite several log files into one. This will enable you to analyze data from several hosts or from several applications more efficiently. Such an approach may prove to be useful when detecting distributed attacks. However, such a merger might cause two problems:

It may be necessary to convert several log files to a unified format. If you are joining several log files of a similar type (for example, logs from different hosts running the same OS), there are no difficulties created by this merging. However, if you are integrating log files from entirely different applications or systems, one or more of the log files may contain forms of data types that do not exist in the other log files.
Time synchronization. If you are joining log files from systems located in different time zones, this will most likely also create problems with the uniting of this data and the subsequent analysis and correlation of the integrated log.

The netForensics system (http://www.netforensics.con/) from netForensics.com is an example of a system that integrates log files from different devices. This system will be covered later in more detail. Another system of this type, which joins log files from different hardware and supports time synchronization, is RealSecure SiteProtector from ISS.

To summarize all event logging, I would like to mention once again that the following aspects must be included into the security policy: "what", "when", "why", "where", and "how" should you react, as well as "who" is responsible for taking these steps.