Analysis of Log Files | Protect Your Information with Intrusion Detection (Power)

Quite often, intruders leave traces of their activity in various log files. Viewing these logs on a regular basis is sometimes the only possible way to detect intrusions. Log files can contain evidence of various unauthorized or unexpected activities taking place in a system or a protected network. Log records can show that someone has compromised your system, or at least attempted to do so. By analyzing logs regularly, you can identify most attacks that have or are taking place and implement procedures to counteract them. It is necessary to distinguish between event log files and transaction log files. Transaction log files allow not only the registration of the event types, but also log actions performed by a specific user on a specific object. Because of this, in Windows NT/2000/XP, you can not always determine exactly what a specific user has done. For example, using only the Windows NT/2000/XP log file, you can state that your accountant has opened the file named Salary.xls (event code 650) for reading, writing or deleting. On the other hand, using only the Windows NT/2000/XP log file, you can not determine what modifications that particular user has made to the file.

Log-file management comprises three stages, in which you need to determine the following:

The information you need to know about your system
The log files that contain the required information
The attack indicators you are going to use

Consider, for example, a case in which you want to determine who has attempted to exploit the expn vulnerability in the sendmail implementation. This is the first stage. You have determined what you need to detect. At the second stage, you need to open the /etc/syslog.conf, where the paths to the log files are specified [Spitzner1-00]. In the case of sendmail, this is /var/log/maillog (in Linux 7.2). The line shown in Listing 5.1, serves as an indication of unauthorized activity.

Listing 5.1. Exploiting the expn Vulnerability in Sendmail Implementation

 Aug 18 04:03:44 luka sendmail[5453]: NOQUEUE: luka@infosec.ru[200.0.0.200]: expn root [rejected]

Methods of log-file analysis depend on the type and volume of data to be processed. When the volume of data is small, it is possible to analyze these files manually (as was done before) using any text editor or a built-in event-log manager (such as the Event Viewer tool in Windows NT/2000/XP). It is also possible to use spreadsheets (such as Excel) of database management systems (Access, Oracle and so on), enabling you to perform various operations on the data being analyzed (filtering, selection, sorting, building dependencies and so on). However, this method of analysis is rather time-consuming and tedious since, in order to detect a single record providing evidence of unauthorized activity, you normally need to view hundreds or thousands of records. For example, when you double-click a text-file shortcut in the Windows NT/2000/XP GUI, this file will be opened using the WordPad or Notepad programs. In the course of this procedure, more than 20 events related to file access are registered in the log file (for WinWord there will be even more events). During business hours, there might be several thousand events like these that will cause significant growth in the log file.

Besides normal database management systems, it is possible to use specialized systems for the analysis of log files (so-called "log checkers"), which are specially designed for the analysis of log files in a manner very close to the real-time mode. Examples of these systems include SWATCH and Cisco IDS Host Sensor. The next generation of these systems is in the form of log checkers that support mechanisms for semantic compression, which allow one to join several related events within a single one and output that event to the console. For example, five events of a "logon failure" type can be united as a single instance of "password cracking." The RealSecure Server Sensor, for example, in implementing the SecureLogic and Local Fusion mechanisms, works according to this principle.

I would like to provide an example that I encountered personally in November 2000. One morning, when viewing logs on one of my computers (I have two systems; on one of them I installed RealSecure Server Sensor and, on the other one, I have RealSecure Desktop Protector), I noticed Six Brute_Force_Login_Attack events, logged over the previous two days, meaning that the RealSecure Server Sensor system had detected brute force password attacks on my computer. Additional analysis showed that each of these events appeared as the result of a semantic compression of eight successive failed logon attempts over a short time interval. The RealSecure system had defined these events as Failed_login-bad_username_or_password (I hope that you remember the difference between an attack and a security event, which were described in Chapter 2). If I had not been using the intrusion detection system, I would have had to view several thousand events logged in the Windows NT security log (I now work in Windows 2000, but the event logging system has not improved significantly). The filtering of events (the result shown in Listing 5.2) is not always convenient. This is especially true when you need to control several hosts located in different network segments simultaneously.

Listing 5.2. Failed Attempts to Logon to Windows Nt 4.0 (Fragments of the Security Log File)

 Date        Time        Source      Category        Code   User      Computer 27.11.00    19:19:38    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:19:29    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:19:21    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:19:12    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:19:03    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:18:55    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:18:46    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    19:18:37    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:32:14    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:32:05    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:56    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:47    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:38    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:29    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:20    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    18:31:11    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:28:07    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:58    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:49    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:40    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:31    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:22    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:13    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    16:27:04    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:22:00    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:51    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:42    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:33    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:24    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:15    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:21:06    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    14:20:57    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:54    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:45    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:36    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:27    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:18    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:09    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:15:00    Security    Logon/Logoff    529    SYSTEM    NT-IIS 27.11.00    12:14:51    Security    Logon/Logoff    529    SYSTEM    NT-IIS

Code 529 corresponds to a "failed logon" event. A complete list of logged events can be downloaded from the Microsoft web server [Microsoft1-00, Microsoft2-00].

A potential danger also lies in the fact that, as the current log file grows, older records are overwritten by newer ones. Thus, if you do not view log files on a regular basis, you will miss a large number of events. I should mention that, recently, Microsoft has begun supplying a new product - Microsoft Operations Manager (http://www.microsoft.com/mom/), which is intended for managing log files located on remote hosts. However, this system is not oriented towards solving security problems either.

However, even semantic compression mechanisms do not always provide the level of analysis necessary for the efficient investigation of security incidents. When performing such an analysis, you can not just use the log checker or intrusion detection system alone. As shown in the previous example, the investigation of unauthorized activity is impossible without human attention, even though intrusion detection systems automate the analysis process. For instance, in the example we just considered, the intrusion detection system could not determine that password cracking was most likely attempted using some form of automated tool. Manual analysis, on the other hand, permitted the detection of this fact at first glance, since each logon attempt took place nine seconds after the previous one. Obviously, it is impossible to manually enter eight passwords in a nine-second interval, so there is clear evidence of the fact that an automated tool was used for this purpose.

Furthermore, to do a comprehensive analysis, it is necessary to use different log files from different systems, which most intrusion detection systems are currently unable to do. Correlating of data from dissimilar systems and different software still remains an unattainable dream, despite the fact that the first attempts at providing such a solution have already been made - RealSecure SiteProtector, netForensics, and so on. Consider, for example a situation in which the log file of a banking system contains a number of records indicating that one of the operators has made several payments on Wednesday, between noon and 1 p.m. The information would seem to be evidence of normal activity, typical for that employee. However, the operating-system log file (or that of the security system) contains other record(s), from which it is clear that the operator was not at a workstation during that time (there was a lunch break, for example). This discrepancy is hard to detect without additional manual analysis of the log files. Let us consider another example. Suppose that the system administrator performed some operations on server A at 14:00:12, and then performed other operations on computer B at 14:07:35. At first glance, this activity is quite normal. However, computers A and B are located in different buildings, and at least 10 minutes are required to get from the first location to the other. This is suspicious, and the results of an analysis reveal the fraud - several employees were working using the same user account, which is a security policy violation. Finally, let us return to the earlier example. Analysis of the Windows NT/2000/XP security-log file (SecEvent.evt) reveals attempts to access specific files, but it is impossible to state what changes, if any, were made to the file. However, if the file is analyzed in combination with the log files of the file integrity-control system, it is possible to determine the details of any writing to that file. Furthermore, by comparing the file's size before and after modification, you can determine whether the user was adding or deleting information.

Once again, I would like to mention that, unfortunately, log-file records themselves can not state that specific actions were unauthorized. Log files only register the information that is required to perform the analysis. Further results depend on the security policy of your organization.