18.2 Log Examples | Security Warrior

< Day Day Up >

This section briefly covers examples of audit logfiles. We discuss Unix logs, and then Windows.

18.2.1 Unix

The increasing popularity of commercial and free Unix systems makes Unix log analysis skills a growing priority. Unix and Linux installations produce a flood of messages (via a syslog or "system logger" daemon), mostly in plain text, in the following simple format:

 <date / time> <host> <message source> <message>

such as:

 Oct  10 23:13:02 ns1 named[767]: sysquery: findns error (NXDOMAIN) on ns2.example.edu? Oct 10 23:17:14 ns1 PAM_unix[8504]: (system-auth) session opened for user anton by (uid=0) Oct 10 22:17:33 ns1 named[780]: denied update from [10.11.12.13].62052 for "example.edu" Oct 10 23:24:40 ns1 sshd[8414]: Accepted password for anton from 10.11.12.13 port  2882 ssh2

This example will be familiar to anyone who has administered a Unix system for at least a day. The format contains the following fields:

Timestamp: The system time (date and time up to seconds) of the log-receiving machine (in the case of remote log transfer) or the log-producing machine (in the case of local logging).
Hostname or IP address of the log-producing machine: The hostname may be either the fully qualified domain name (FQDN), such as ns1.example.edu , or just a computer name, such as ns1 in the example above.
Message source: The source can be system software (sshd or named in the above examples) or a component (such as PAM_unix) that produced the log message.
Log message: The log message might have different formats, often containing application names , various status parameters, source IP addresses, protocols, and so on. Sometimes the process ID of the process that generated the log record is also recorded in square brackets.

The four log messages above indicate the following, in order :

There is a problem with a secondary DNS server.
A user, anton, has logged in to the machine.
A forbidden DNS access has occurred.
A user, anton, has provided a password to the Secure Shell daemon for remote login from IP address 10.11.12.13.

18.2.1.1 Analysis of Unix logging

Unix system logging is handled by a syslog daemon. This daemon first appeared in early BSD systems. Program and OS components can send events to syslogd via a system command, a socket ( /dev/log ), or a network connection using UDP port 514. Local logging is often implemented via the Unix logging API.

As described in the syslogd manual page, "system logging is provided by a version of syslogd derived from the stock BSD sources. Support for kernel logging is provided by the klogd (on Linux) utility, which allows kernel logging to be conducted in either a standalone fashion or as a client of syslogd." In standalone mode, klogd dumps kernel messages to a file; in combination mode, it passes messages to a running syslog daemon.

Remote logging requires the syslog daemon to be configured to listen on UDP port 514 (the standard syslog port) for message reception. To enable remote reception , you run syslogd -r in Linux. This functionality is enabled by default on Solaris and some other Unix flavors. Messages arrive from the network in plain text with no timestamp (it is assigned by the receiving machine). The arriving messages also contain severity and facility values, decoded by the receiving syslog daemon.

Arriving or locally generated logs are sent to various destinations (such as files, devices, programs, the system console, or other syslog servers) by the syslog daemon using priorities and facilities. Facilities include auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security (same as auth), syslog, user, uucp and local0 through local7. The syslog manual also provides this list of syslog priorities (in ascending order by importance): debug, info , notice, warning, warn (same as warning), err, error (same as err), crit, alert, emerg, and panic (same as emerg). The priorities error, warn, and panic are present for compatibility with older implementations of syslog.

The syslog configuration file is usually located in /etc/syslog.conf . As shown below, it allows you to configure message sorting into different files and other structures:

 *.*                                               @log host kern.*                                            /dev/console *.crit                                            anton,other,root local2.*                                         /dev/custom_fifo *.info;mail.none;authpriv.none;cron.none          /var/log/messages authpriv.*                                        /var/log/secure mail.*                                            /var/log/maillog cron.*                                            /var/log/cron uucp,news.crit                                    /var/log/spooler local7.*                                          /var/log/boot.log

Messages can be directed to local files (such as /var/log/messages ), sent to devices (such as a /dev/console ), or broadcast to all or selected logged-in users (anton, other, root) in a manner similar to the write or wall shell commands. In addition, messages can be sent to a remote host (see log host above) and directed to named pipes or FIFOs ( /dev/custom_fifo in the above example) created with the mknod or mkfifo commands. Even messages that just arrived from the network can be forwarded to further machines, provided the syslog daemon is configured to do so ( syslogd -h in Linux). Forwarding is disabled by default since it might cause network congestion and other problems (due to traffic duplication).

Remote logging is a great boon for those wishing to centralize all the audit records in one location. syslog implementations from different Unix flavors interoperate successfully. You can mix and match various Unix boxes in one syslog infrastructure.

Many syslog problems have become apparent over its lifetime. Here is a short list:

The log message format is inconsistent across applications and operating systems. Apart from date and host, the rest of the message is "free form," which makes analysis difficult if many different messages are present.
Message filtering by priority and facility is not very flexible, thus turning some logfiles into "wastebaskets" of motley message types. There is no way to filter messages by their content, and even adjusting the priority/facility of a log producing program often proves challenging.
UDP-based network transfer is unreliable; if the receiving end of the UDP link (not a connection, since UDP is connectionless) is down, the messages are lost with no chance of recovery.
UDP-based network transfer happens in plain text (unencrypted), with no authentication, little flood protection, and no message-spoofing protection. This can be a security disaster. Usually, it is not a serious problem, since syslog is used on an internal trusted network or even a dedicated management LAN.
When forwarding messages from host to host, only the last "hop" can be seen in the message. Thus, if one machine sends a message to another, which in turn forwards it elsewhere, the arriving message seems to originate at the second machine.
Storing logs in plain-text files makes it difficult to analyze large volumes of log data. Just try to run a complicated grep command on a 5-GB file and you will understand the scope of the problem. While log rotation, archival, and compression all help mitigate the problem, a relational database is highly desirable.
Stored logs are vulnerable to modification and deletion, especially when stored locally. It is difficult to check the logfiles for missing pieces and " cooked " data, especially if they are modified by an expert attacker with root access.

Several popular Unix syslog replacements address the above deficiencies. We will look at two well-known replacements : syslog-ng , by BalabIT (http://www.balabit.hu/en/downloads/syslog-ng); and msyslog , by CORE SDI (http://www.corest.com). (A third alternative, Darren Reed's nsyslog , does not appear to be actively updated anymore.) These programs feature reliable TCP communication with message buffering, more filtering options (in addition to Severity and Facility of standard syslog), non-root secure operation in chroot jail, log database support, better access control and encryption, and even logfile integrity support.

Let's look at setting up msyslog for a small network. Unlike the above syslog.conf example that sent all the messages to a log host machine over UDP, in this case we will use TCP with buffering and store the logs in the database and a plain=text file. Additionally, we will enable cryptographic protection for the plain-text version of a logfile that allows us to detect changes to stored logs.

On client machines that generate and forward logs, we deploy and configure msyslog. msyslog uses the regular /etc/syslog.conf file with minor changes, as follows :

 *.*         %tcp -a -h log host -p 514  -m 30 -s 8192

This sends all messages from localhost to the log host via a TCP port 514 connection, buffering 8,192 messages in case of connection failure and waiting up to 30 seconds to retry the connection to the log host. Other lines in /etc/syslog.conf can be in the usual syslog form, as described above. Run the daemon via the msyslogd -i linux -i unix command or use the default startup script provided with the msyslog package.

On the server, we configure msyslog to run as follows:

 msyslogd -i linux -i unix -i 'tcp -a -p 514'

This makes the daemon listen for connections on TCP port 514 and allows logging from all machines. Access control rules can be applied to restrict by IP address the hosts that can send logs. We also add crypto protection to more important messages (such as those of priority crit). To enable this, add a line to /etc/syslog.conf as follows:

 *.crit %peo -l -k /etc/.var.log.authlog.key %classic  /var/log/critical

Next , stop the msyslog daemon, clean or rotate logs, and generate the initial cryptographic key using the enclosed utility:

 peochk -g -k /etc/.var.log.authlog.key

Restart the daemon, and log protection is enabled. Upon receiving a new message, msyslog updates the signature. To check the log integrity, run the following:

 peochk -f /var/log/messages -k /etc/.var.log.authlog.key

If everything is fine, you'll see this:

 (0) /var/log/critical file is ok

If the logfile was edited, you'll see:

 (1) /var/log/critical corrupted

Additionally, to send all the messages to a database, another line should be added to /etc/syslog.conf as follows:

 *.*     %mysql -s localhost -u logger -d msyslog -t syslogTB

This line saves a copy of each message in the MySQL database. However, before database collection starts, you should create a schema and grant insert privileges to a "logger" user. This is accomplished via the following command:

 echo "CREATE DATABASE msyslog;"  mysql -u root -p

which creates a database instance. Obviously, MySQL must be installed and running on your system for this to work. The next command,

 cat syslog-sql.sql  mysql msyslog

defines tables for log storage. syslog-sql.sql is shown below:

 CREATE TABLE syslogTB (   facility char(10),       priority char(10),       date date,               time time,               host varchar(128),       message  text,           seq  int unsigned auto_increment primary key );

The last step is to grant access privileges for message insertion:

 echo "grant INSERT,SELECT on msyslog.* to logger@localhost;"   mysql -u root -p

The above database setup can safely store millions of records. The data can be browsed via a command-line interface (mysql) or one of many GUI database frontends and web frontends (such as PHPMyAdmin, written in PHP).

To conclude, msyslog and syslog-ng interoperate with classic syslog implementations if log transfer is done via UDP. In this case, a mix of new syslog and classic syslog is deployed across the network and a new syslog is deployed on the log-collection server. Many of the advanced features (filtering, integrity checking, database collection) are then available, and only the log network transfer is handled the old-fashioned way.

18.2.2 Windows

Windows (at least in its more advanced NT/2000/XP versions) also provides extensive system logging. However, it uses a proprietary binary format ( *.evt ) to record three types of logfiles: system, application, and security.

Figure 18-1 is an example of a Windows security log. The system log contains various records related to the normal (and not-so-normal) operation of the computer. This example shows normal activity on a Windows XP workstation. Double-clicking on an entry drills down to show details (Figure 18-2). To read the Windows event logs, you need to use a viewer or another program that can read the * .evt files. The viewer can also be used to export the files into the comma-separated values for analysis or viewing using a text editor.

Figure 18-1. Windows security log showing normal operation

Figure 18-2. Double-clicking to drill down for detail on the Windows security log

18.2.3 Remote Covert Logging

A chapter on logging would be incomplete without a section on covert logging. In some situations (such as for honeypots and other scenarios), it is highly desirable to hide the presence of centralized remote logging from visitors to your network. Normally, the syslog configuration files reveal the presence of remote logging and pinpoint the location of logging servers. This enables hackers to attack, possibly take over the log servers, and erase the evidence. In contrast, stealthy logging is difficult for an attacker to detect.

The most basic stealthy logging option is actually not very stealthy. It just provides a backup site for log storage. In addition to the designated log servers (visible to attackers ), a sniffer (such as the Snort IDS in sniffer mode, tcpdump, or ngrep) is deployed on a separate machine. For example, if the server with IP address 10.1.1.2 sends logs to a log server at 10.1.1.3, a special machine with no IP address is deployed on the same subnet with a sniffer running. Most sniffers can be configured via the Berkeley Packet Filter (BPF) language to receive only certain traffic. In this case, we will run a command similar to:

 ngrep "" src host 10.1.1.2 and dst host 10.1.1.3 and proto UDP and port 514 >  /var/log/stealth-log

This command allows the sniffer (in the case, ngrep, available from http://ngrep.courceforge.net) to record only the remote syslog traffic between the two specified hosts and to stream the data into the file /var/log/stealth-log .

Obviously, the tcpdump tool may be used to record all the syslog traffic in binary or ASCII format, but ngrep seems better for this job, since it only shows the relevant part of the syslog packet.

A second stealthy log option sends logfiles to a log host that does not run syslog (or any other networked services). In this case, the host firewall running on the log server simply rejects all incoming UDP port 514 packets. How would that constitute logging, you ask? A sniffer that picks up each UDP packet before it is rejected by the firewall is now deployed on that log server. While none of the host applications see the packet due to its rejection by the firewall, the sniffer (using the above command line) records it into a file.

This might be implemented to avoid a "let's hack the log server" scenario. In practice, we have used just such a setup on a honeynet decoy network; the messages are sent to a router (which obviously does not care for receiving syslog messages). One can point such a message stream just about anywhere , but using a host with no syslog has the additional benefit of confusing the attacker (and might be considered a configuration error on the part of system administrators).

A third, ultimate stealthy logging option involves sending log data to a nonexistent host and then picking up the data with a sniffer, as outlined above. In this case, one extra setting should be changed on the machine that sends the logfiles: The TCP/IP stack should be tricked into sending packets to a machine that can never respond (since it doesn't exist). This is accomplished by the following command:

 arp -s 10.1.1.4 0A:0B:OC:OD:78:90

This command tricks the IP stack on the log-sending machine into thinking that there is something running on the 10.1.1.4 address. In this case, both the MAC and IP addresses can be bogus , but the IP address should be on the local network. Please note that the MAC address does not have to belong to an actual log server.

The nonexistent server option is preferable if a higher degree of stealth is needed. It might not be applicable for a typical corporate LAN, but it comes handy in various special circumstances.

18.2.4 Other Logging Variations

To conclude, let's briefly look at other Unix logfiles. In addition to the standard Unix syslogd and klogd logging daemons, there is also the BSD process accounting facility, commonly seen on Linux, Solaris, and BSD variants. Process accounting records the processes running on a Unix system and stores the data in a binary file. Several utilities are provided to examine this data, which looks similar to the following:

 lastcomm          S   X root     stdin      3.19 secs Sat Nov  2 22:16 head              S     root     stdin      0.00 secs Sat Nov  2 22:16 egrep                   root     stdin      0.01 secs Sat Nov  2 22:16 grep              S     root     stdin      0.01 secs Sat Nov  2 22:16 bash               F    root     stdin      0.00 secs Sat Nov  2 22:16 bash              SF    root     stdin      0.00 secs Sat Nov  2 22:16 dircolors               root     stdin      0.00 secs Sat Nov  2 22:16 stty                    root     stdin      0.00 secs Sat Nov  2 22:16 bash              SF    root     stdin      0.00 secs Sat Nov  2 22:16 tput                    root     stdin      0.01 secs Sat Nov  2 22:16 bash              SF    root     stdin      0.00 secs Sat Nov  2 22:16 tput                    root     stdin      0.01 secs Sat Nov  2 22:16 su                      anton    stdin      0.04 secs Sat Nov  2 22:16 head                    anton    stdin      0.01 secs Sat Nov  2 22:16

The above record (produced by the lastcomm head -20 command) shows that commands including grep , egrep , bash , and even the lastcom command itself were run on the machine by "root", and that user "anton" switched to "root" by using an su command at 10:16 p.m. on November 2. This binary part of the Unix audit trail completes the picture provided by the syslog by adding more details on running processes. Unfortunately, there is no mechanism for the remote transfer of these audit records.

The Unix logging framework can even be integrated into Windows machines by using solutions such as Kiwi Syslog, available for free at http://www.kiwisyslog.com.

Overall, interpreting Unix messages becomes easy after you have administered a system for a while. The challenge of the analysis is to recreate a complete picture of an intrusion from logs collected by different devices spread across the network, while taking into account the events that occurred over the period of time in question.

< Day Day Up >