11.3 System Logging

Unix acquired a system-logging function early in its development. System logging is implemented as a syslog daemon ^[4] that receives messages sent by various programs running on the system. In addition, other computer and network devices, such as routers, can send log messages to the logging server. System logging is extremely valuable for many purposes, from troubleshooting hardware to tracking malicious attacks ”provided somebody is actually reading the system logfiles. Here's an excerpt showing several messages received by a syslog daemon on the machine "examhost". The logfile records the date and time of the message, the name of the computer that sent it, the program that produced the message, and the text itself:

^[4] A daemon is a program that listens on the network port. Sometimes a daemon is also called a server or even a service .

 Dec 13 10:19:10 examhost sshd[470]: Generating new 768 bit RSA key. Dec 13 10:19:11 examhost sshd[470]: RSA key generation complete. Dec 13 10:20:19 examhost named[773]: sysquery: findns error (NXDOMAIN) on dns. example.edu? Dec 13 10:21:01 examhost last message repeated 4 times Dec 13 10:26:17 examhost sshd[20505]: Accepted password for user from 24.147.219.231  port 1048 ssh2 Dec 13 10:26:17 examhost PAM_unix[20505]: (system-auth) session opened for user anton  by (uid=0) Dec 13 10:30:28 examhost PAM_unix[20562]: (system-auth) session opened for user root  by anton(uid=501) Dec 13 10:35:10 examhost2 sshd[456]: Generating new 768 bit RSA key. 

In this example, you can see there was a login via SSH. In addition, you can see some problems with the DNS server, and you can see that the syslog is configured to receive messages from other hosts (note the message from "examhost2").

The syslog daemon is configured by the /etc/syslog.conf file, as follows :

 # Log all kernel messages to the console. kern.*                                                 /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none                       /var/log/messages # The authpriv file has restricted access. authpriv.*                                              /var/log/secure # Log all the mail messages in one place. mail.*                                               /var/log/maillog # Log cron stuff cron.*                                                  /var/log/cron # Everybody gets emergency messages, plus log them on another # machine. *.emerg                                                 * # Save mail and news errors of level err and higher in a # special file. uucp,news.crit                                         /var/log/spooler #send everything to loghost *.*                            @loghost.example.edu 

In this case, the Linux syslog.conf daemon sorts messages by priority and facility. Possible priority values, in order of increasing importance, include: debug, info, notice, warning (warn), error (err), crit, alert, emerg (panic). The facility parameter differs according to the flavor of Unix. Linux supports the following values for facility: auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, syslog, user, uucp, and local0 through local7. Based on comments in the file (lines denoted by the leading "#" character), you can see how the messages are sorted. All log messages are sent to a different machine ( loghost.example.edu ) via the last line in the file. This excerpt also demonstrates the typical location for logfiles on the system: /var/log , or sometimes /var/adm .

Remote logging is implemented via UDP. As we discussed in Chapter 6, UDP over IP is an unreliable and connectionless protocol, which means that log messages can be lost or faked. In spite of these drawbacks, setting up a dedicated logging server with no other network services increases security; it is difficult for attackers to avoid being logged, because they are forced to attack a discrete machine with few entry points. Attackers can flood the logging server so it starts dropping messages, but you can configure certain Unix systems to shut down in such a situation. Moreover, you can configure syslog to log to an IP-less machine via a serial link, which makes it very difficult to attack the logging server. To avoid faked messages, you can configure some versions of syslog to accept log messages only from designated machines, via command-line options to syslog. While attackers can also bypass this defense using spoofed packets, it is still an important security measure. There are even some experimental syslog implementations (such as CORE-SDI) with cryptographic support and TCP/IP reliable network logging. Unfortunately, they have not yet been integrated into mainstream Unix.

Some Unix logs are binary logs, such as that generated by a login program. This file is typically called /var/log/wtmp . To produce human-readable output, you can use commands such as w (shows currently logged-in users based on /var/log/utmp ) or last (shows recently logged-in users), as follows:

 user    pts/0        ne.isp.net Fri Dec 14 19:11   still logged in    user    pts/0        ne2.isp.net Fri Dec 14 18:19 - 18:23  (00:03)     user    pts/0        ne3.isp.net Fri Dec 14 16:03 - 16:10  (00:06)     friend   pts/0        ool.provider.net Fri Dec 14 09:32 - 12:58 (03:26) 

This excerpt shows that users "user" and "friend" have logged in remotely from certain machines at certain times. "user" is still logged in to the server on the terminal "pts/0". These logfiles are difficult to manage due to their binary nature. It is also difficult for attackers to modify them; however, multiple tools exist to do just that. Nevertheless, these files are very useful for high-level user monitoring.

Overall, logs comprise a vital part of Unix security ”provided, of course, that somebody actually reads them. If hundreds of machines log to the same server, the amount of syslog information quickly becomes unmanageable. Fortunately, most of the Unix logfiles are plain-text files that can be parsed by programs or scripts to condense the information and increase its usefulness . Log monitoring programs such as logwatch and host-based intrusion detection systems such as Symantec ITA and Dragon Squire automate and simplify log monitoring.

Some other utilities also leave an audit trail on Unix systems. Process accounting is one of these. It is very useful for security purposes and general system accounting. Some readers may be old enough to remember that process accounting has its roots in the age when people were charged based on the CPU time that they used. Process accounting is implemented as a kernel feature that is controlled by a user-space program. It records all processes started by the system in a binary file (called /var/log/pacct on Linux and /var/account/pacct on BSD versions of Unix). To bring the data to userland, the lastcomm command may be used as follows:

 sendmail          SF    root     ??         0.06 secs Thu Dec 13 10:30 egrep                   root     stdin      0.01 secs Thu Dec 13 10:30 grep              S     root     stdin      0.01 secs Thu Dec 13 10:30 dircolors               root     stdin      0.00 secs Thu Dec 13 10:30 stty                    root     stdin      0.00 secs Thu Dec 13 10:30 bash              SF    root     stdin      0.00 secs Thu Dec 13 10:30 

This example shows the process name, username under which the process runs, controlling terminal (if any), amount of CPU time used by the process, and the date and time the process exited. It is possible for malicious attackers to fake the process name in accounting records, but not the username. Unfortunately, the full command line is not recorded. Other tools come to the rescue here.

Many modern Unix shells (tcsh, bash, and others) record a history that can be viewed as a sort of log and can be used to track an intruder. For example, just typing history at the shell prompt displays something similar to the following:

 999  less sent-mail  1000  clear  1001  ls -l  1002  cat /etc/hosts.*  1003  h tail -10 

This snippet shows several commands that the user has run. They aren't timed or dated, but simple correlation with process accounting records will reveal the missing details.

Unix logging provides a wealth of information about system behavior. If reviewed by a competent administrator, the logfiles, accounting records, and shell histories reveal meticulous details about attackers. Clever hackers will try to erase the evidence, so you should make an effort to safeguard the logs.