The Role of Logs

   

---

Almost every server process on your computer generates a logfile. What is logged, and when, is up to the software that is writing to the log. A centralized process, called syslog , manages the incoming log data from other applications and directs it to the system console, files, or remote syslogd servers. Many applications send data to syslog , some to syslog and local logfiles, others only to local files. Our first priority will be getting you familiarized with the primary Mac OS X logfiles as they are initially configured on your computer.

Tracking Activity

Apple designed Mac OS X so that an average user need never look at logs nor even know that they exist. At the same time, they provide the same ( potentially unknowledgeable) users the ability to enable services such as SSH and FTP ”both excellent entry points for attackers .

Assuming that "because it's a Mac, it's safe" is no longer wise. Logs are your window to check up on how your machine is working, who is logged in, what they've been doing, and when. They also provide a means of tracking the path of an intruder through a compromised system, and are often the first target of an attack. Disable a machine's capability to report your presence, and you're virtually invisible.

Assume an attacker gains SSH access to your computer through an exploit. Unless you have something of value on your system that he's after, his next step will be to use your computer to monitor and potentially exploit other machines, or even set up file trading services for his 3L1t3 friends . An intelligent attacker will lie low and draw as little attention to himself as possible.

Your computer, however, has logfiles that make it easy to identify an unknown presence. The file /var/run/utmp stores information about the currently logged-in user, whereas /var/log/wtmp tracks all user logins/logouts, shutdowns, reboots, and crashes.

For example, last uses wtmp to display a user login history:

 %  last  jray      ttyp1    dhcp024-210-090- Tue Nov 19 21:17   still logged in jray      ttyp0    dhcp024-210-090- Sat Nov 16 23:44 - 01:26  (01:42) jray      ttyp0    despair.ag.ohio- Fri Nov 15 15:11 - 17:19  (02:08) ohioline  ttyp0    dhcp35-112.ag.oh Fri Nov 15 15:02 - 15:10  (00:08) jray      ttyp1    postoffice.ag.oh Fri Nov 15 11:08 - 11:08  (00:00) hlaufman  ttyp0    theotormon.ag.oh Fri Nov 15 08:09 - 11:17  (03:08) jray      ttyp1    despair.ag.ohio- Thu Nov 14 15:23 - 17:03  (01:39) jray      ttyp2    despair.ag.ohio- Fri Nov  8 16:01 - 17:00  (00:59) ... hlaufman  ttyp1    theotormon.ag.oh Fri Nov  8 15:14 - 16:52  (01:37) reboot    ~                         Thu Nov  7 09:50 shutdown  ~                         Thu Nov  7 09:49 hlaufman  ttyp1    theotormon.ag.oh Wed Nov  6 14:42 - 15:52  (01:10) jray      ttyp1    despair.ag.ohio- Mon Nov  4 12:16 - 12:17  (00:00) robyn     console  www              Mon Nov  4 11:57 - 11:59  (00:01) jray      ttyp1    despair.ag.ohio- Mon Nov  4 11:20 - 12:00  (00:39) hlaufman  ttyp0    theotormon.ag.oh Mon Nov  4 08:35 - 16:48  (08:12) jray      console  www              Mon Nov  4 00:04 - 00:04  (00:00) hlaufman  ttyp1    dhcp065-024-072- Sun Nov  3 18:54 - 20:38  (01:43) jray      ttyp0    despair.ag.ohio- Fri Nov  1 09:29 - 14:24  (04:54) wtmp begins Fri Nov  1 09:29 

Someone who has gained login access to your computer would quite clearly show up in both utmp and wtmp as either an unknown account and/or an unknown remote hostname. Unfortunately, these files will be some of the first targets for someone who intends to commandeer your system.

Attackers have a number of options to cover their tracks. They can simply remove these accounting files and replace them with an empty copy:

 %  last  wtmp begins Tue Nov 19 21:31 

Alternatively, they can copy an archived wtmp / utmp file over the originals and set the logs' immutable flags. The result is a user login history that looks normal to an overworked and unobservant administrator ”but never updates.

A third and more devious approach is to not modify the logfiles, but to modify the output of the utilities that display the files. My username is jray on a number of systems. If I've gained access to a machine and created an account for myself (or gained access via someone else's account), I'd want to hide my logins when the administrator used reporting tools such as last to show account activity. Assuming I've gained access to the machine via a root exploit, I can simply replace the existing reporting commands with a modified "filtered" copy:

 %  last  ohioline  ttyp0    dhcp35-112.ag.oh Fri Nov 15 15:02 - 15:10  (00:08) hlaufman  ttyp0    theotormon.ag.oh Fri Nov 15 08:09 - 11:17  (03:08) hlaufman  ttyp1    theotormon.ag.oh Fri Nov  8 15:14 - 16:52  (01:37) reboot    ~                         Thu Nov  7 09:50 shutdown  ~                         Thu Nov  7 09:49 hlaufman  ttyp1    theotormon.ag.oh Wed Nov  6 14:42 - 15:52  (01:10) robyn     console  www              Mon Nov  4 11:57 - 11:59  (00:01) hlaufman  ttyp0    theotormon.ag.oh Mon Nov  4 08:35 - 16:48  (08:12) hlaufman  ttyp1    dhcp065-024-072- Sun Nov  3 18:54 - 20:38  (01:43) wtmp begins Fri Nov  1 09:29 

To generate this conveniently jray -free output, an attacker would first rename the last command: /usr/bin/oldlast . Next, a simple script could be installed in /usr/bin/last that performs the filter:

 #!/usr/bin/perl $last=`oldlast`; $last=~s/jray.*?\n//g; print $last; 

Most modern rootkits include modified versions of commands such as last , who , netstat , and ps that will filter output to remove logins, processes, and connections, effectively allowing the attacker to remain invisible until other attacks are traced back to your computer.

With a number of methods for " beating " the system, how can logs be of any use in identifying attacks?

First, if an attacker takes the approach of removing, modifying, or locking a logfile, the output of your reporting functions will be noticeably different from that of a healthy system. If, however, you don't know what the logs should contain, you're in trouble. This is exactly what we aim to correct in this chapter.

Second, I have yet to see a rootkit or attacker who is thorough enough to completely hide its presence. Discrepancies between logs and reporting functions are often obvious, no matter how meticulous an attacker has been. The ac command, for example, outputs the total amount of time each user has been logged into your system and a systemwide total. A filter would obviously result in inaccurate system totals. Given enough insight into the operating system, it's certainly possible to modify or remove every single log of activity, but it's extremely unlikely that a vigilant admin would remain completely ignorant of all evidence of system compromise. Additionally, software such as Tripwire (http:// sourceforge .net/projects/tripwire/) can be installed to report on any changes to your system files, identifying the altered reporting utilities immediately.

NOTE

Interestingly enough, I've seen a large number of Linux rootkits that, while modifying the output of ls and find to hide directories, do not remove the locate database or modify locate to hide files or directories. Searching for files that consist of periods and spaces quickly reveals the attacker's working directory.

Policy Violations

Not all nefarious actions on your computer system are the result of attack. Early in this book we discussed the necessity of creating a comprehensive security policy for your network and appropriate-use guidelines for computers on the network. Unfortunately, although it is possible to put constraints on individual accounts, it isn't feasible to micromanage a server with hundreds of users.

Web servers, for example, often provide FTP access to user accounts for uploading content. Unfortunately, many users are far from protective of their login information. Entire offices may share a single account, and passwords may be used at home or at work, and are rarely maintained in the possession of the original account holder.

A highly trafficked university Web server recently became a pirate software trading zone after a student who was entrusted to update a department's Web site passed the information out to his friends. Thankfully, a weekly audit of the FTP transfer logs (which aren't enabled by default in Mac OS X) revealed hundreds of megabytes in zip files moving to and from the server. The logs provided the time, date, account, and hostname ”more than enough information to locate the machine and subsequently the student who had caused the trouble.

In this case, there wasn't an "attack," nor was there a software exploit, or any Apple Security update that could have solved the problem. Yet the incident is definitely a breach of security.

NOTE

For the record, if you're running a Web server with a large number of accounts, you may want to consider using WebDAV for file transfers. WebDAV does not require "real" user accounts for connections, and thus eliminates the potential for those accounts to be compromised and gain access to other parts of the server. Depending on your user needs, this isn't always a possibility.

Now consider that Mac OS X enables an inexperienced user to turn FTP access on, but doesn't provide any easy way for the same user to see the logs generated by the FTP server. In fact, the FTP server isn't even configured by default to log what files are transferred via FTP ”it records only logins. If FTP was being misused on an out-of-the-box Mac OS X system by an inexperienced administrator, it would be virtually impossible to detect that anything was afoot.

Knowing what logs are on your computer and understanding what they contain and how to configure them is an important part of maintaining a secure system.

Mac OS X Logfile Locations

Mac OS X, like other Unix systems, uses /var/log/ as its primary repository for logfiles. For some reason, however, Apple has also chosen to create /Library/Logs for a handful of other "Apple-specific" logfiles and /Library/Receipts for logs of installed software. This isn't really a problem ”just somewhat unusual in that there isn't an obvious reason for the separation.

NOTE

Logfiles, by their nature, log things. In doing so, they use up drive space. It is common practice to use a separate partition for /var/logs (or reconfigure system processes to log to a different drive) so that an attacker can't fill up a volume with logfile entries, thus disrupting processes that are attempting to store critical data on the same drive.

So, what is stored on your system? Table 19.1 lists the common default Mac OS X logfiles, their location, and what they contain.

Table 19.1. Mac OS X Default Log Locations

Logfile	Description
/var/log/cups	The cups directory contains access_log , error _ log , and page_log ”providing information on access to the CUPS Web interface, errors encountered by the CUPS process, and a print job log, respectively. CUPS, incidentally, refers to the Common Unix Printing System ”the printing "backbone" of Mac OS X.
/var/log/ftp.log	Logs login/logout information from the Mac OS X FTP server lukemftpd .
/var/log/httpd	The httpd directory contains a Web server access_log with each Apache request, and an error_log containing errors reported by the server process.
/var/log/lastlog	Similar to /var/log/wtmp , identifies the last time a user has logged in (binary logfile).
/var/log/ lookupd .log	Typically, the location for errors from the Mac OS X lookupd process. lookupd , however, does not use this file in Mac OS X 10.2, and it should remain empty (unless configured otherwise in NetInfo) on your system. Instead, lookupd logs to syslog , which places lookupd messages in /var/log/netinfo.log .
/var/log/lpr.log	Logs print requests made via lpr .
/var/log/mail.log	Contains sendmail errors and message delivery logs.
/var/log/netinfo.log	Includes messages logged from the Mac OS X netinfo ”the central repository for user/ group /host information on your computer. Also contains error messages logged by lookupd .
/var/log/samba	The samba log directory contains the log.smbd and log.nmbd , which contain access and error messages from the Samba file server, and the Samba NetBIOS name server, respectively.
/var/log/secure.log	Logs authorization failures, such as failed FTP logins.
/var/log/system.log	Contains operating system notices, authorization failures, errors, and other logging goodness get stored in the main Mac OS X system.log .
/Library/Logs/DirectoryService	The Apple Directory service logs ( DirectoryService.server.log and DirectoryService.error.log ) log messages and errors related to Apple's directory service architecture ”BSD files, NetInfo, LDAP, and so on.
/Library/Logs/CrashReporter	Consists of detailed logs containing application-specific crash information for use in troubleshooting software instability.
/Library/Logs/AppleFileService	AppleFileServiceAccess.log and AppleFileServiceError.log record logins and errors in the built-in Mac OS X Appleshare server. The Access log is not enabled by default.
/Library/Logs/SoftwareUpdate.log	Logs all automatic updates performed via the Software Update preference panel.
/var/log/wtmp	Contains a history of user logins/logouts and system restarts. (Binary logfile)
/var/run/utmp	Stores the currently logged-in users.

Looking at this list you might be wondering, "Why are logs for multiple services showing up in a single logfile?" or "Why are multiple logfiles used for a single service?"

Earlier it was mentioned that many processes log their output via a centralized process called syslog . Messages that are sent to syslog can be stored in multiple locations and directed to different logfiles depending on their level ”a 32-bit value that contains an encoded facility and level. As you add new Unix software to your computer (such as Snort, in the preceding chapter), you'll find that many applications can and do log via syslog ”and the only real way to know what is going where is to understand how syslog works, and how you can configure it.

WHY ARE THERE LOGFILES WITH THE EXTENSION .#.GZ AND NO RECORDS OLDER THAN A WEEK?

Many of the Mac OS X logfiles are rotated on a daily, weekly, or monthly basis by the files /etc/daily , /etc/weekly , and /etc/monthly , respectively. To view the contents of a file that is rotated out of use, you need to unzip it ( gunzip <filename> ) first.

In some cases, you may want to disable the rotation altogether, or change it so that it operates at a different interval. You can do this by either commenting out or simply moving the appropriate portions of the daily, weekly, and monthly files. This code, for example, rotates the /var/account/acct file (in /etc/daily ):

[View full width]

 
 [View full width] 
 if [ -f /var/account/acct ] ; then     if [ -x /usr/bin/gzip ]; then gzext=".gz"; else gzext=""; fi     echo ""     echo "Gathering accounting statistics:"     cd /var/account     if [ -f "acct.2${gzext}" ] ; then mv -f "acct.2${gzext}"  "acct.3${gzext}" ; fi     if [ -f "acct.1${gzext}" ] ; then mv -f "acct.1${gzext}"  "acct.2${gzext}" ; fi     if [ -f "acct.0${gzext}" ] ; then mv -f "acct.0${gzext}"  "acct.1${gzext}" ; fi     cp -pf acct acct.0     if [ -x /usr/bin/gzip ]; then gzip -9 acct.0; fi     sa -s > /dev/null fi 

To change the rotation, simply move these lines to a script that is executed at a different interval, or comment them out.

Additionally, these same script files remove some of the logfiles after a certain number of days. This is not a good idea if you intend to maintain any historical data. You can disable the removal by editing the script files, looking for these lines, and commenting them out:

[View full width]

 
 [View full width] 
 if [ -d /var/log/httpd ]; then     echo ""     echo -n "Cleaning web server log files:"     cd /var/log/httpd && \         find . -type f -name '*_log.*' -mtime +7 -exec rm -f --  {} \; >/dev/null  2>&1; fi 

In this code fragment the httpd (Apache) logfiles that are older then seven days are found and removed.

NOTE

Some system messages aren't logged to any file, but to the system console device. The console can be viewed (along with other system files) through the Mac OS X application Console or the Logs tab of the Apple System Profiler .

Additionally, the dmesg command displays the contents of the system message buffer, which may contain useful messages held in memory.

Using Syslog

The Syslog daemon can be a bit confusing for users coming from other platforms, or migrating from earlier versions of the Macintosh. The first point to understand is that syslogd is not generating the messages that are showing up in your logfiles. It provides a logging system that other software can take advantage of, rather than that software having to write its own files ”similar to Window's Event Viewer. Syslog can redirect incoming log information to remote syslog servers, the console, logged-in users, and, of course, files ”providing far more flexibility to developers than if each had to develop logging functions independently. The second point is that syslog -generated files are often configured to be "message-centric" rather than " application-centric ," meaning that logs contain a similar type of message (failed logins, perhaps) from multiple different daemons, rather than multiple messages from a single daemon.

NOTE

In centralizing the system's logging, syslog also presents a single point for disrupting critical log messages for attackers. As you'll see shortly, syslog could easily be configured to dump high-level messages to /dev/null .

Facilities and Levels

To configure syslogd , one first needs to understand the "vocabulary" of the logging system. There are four terms used by syslog , its documentation, and our discussion that you must understand before proceeding:

• Facility . An identifier for the portion of the system that is sending the log entry to syslog , such as the kernel, mail, or FTP processes.

• Level . A ranking of the importance of the incoming log message, from simply informational notifications to emergency warnings.

• Selector . A combination of one or more facilities and levels that are matched against log messages coming into syslog . If the selector matches the level and facility of the message, syslog executes an action .

• Action . A syslog action determines what happens to an incoming log message if it matches a selector in the configuration file. Actions can write to files and devices, forward the messages to remote log servers, and notify logged-in users.

Levels and facilities are not arbitrary values that the user determines. The syslog daemon defines them. Table 19.2 provides the documentation for the available Mac OS X facilities as defined in /usr/include/sys/syslog.h :

Table 19.2. Syslog Logging Facilities

User-Level Name	Direct Name	Logging Purpose
kern	LOG_KERN	Kernel messages
user	LOG_USER	User-level messages
mail	LOG_MAIL	Mail system
daemon	LOG_DAEMON	System daemons
auth	LOG_AUTH	Security/authorization messages
syslog	LOG_SYSLOG	Internal syslog messages
lpr	LOG_LPR	Line printer subsystem
news	LOG_NEWS	Network news subsystem
uucp	LOG_UUCP	UUCP subsystem
cron	LOG_CRON	Clock daemon
authpriv	LOG_AUTHPRIV	Security/authorization messages (private)
ftp	LOG_FTP	FTP daemon
netinfo	LOG_NETINFO	NetInfo daemon
remoteauth	LOG_REMOTEAUTH	Remote authentication/authorization
mark	LOG_MARK	Logs an info message every 20 minutes

What facility a daemon uses to log is determined not by syslogd but by the daemon itself. You should check the documentation of your applications to determine what facilities are used, and how, if desired, to change them.

Note that there are "user-level" facility names and "direct" facility names. You'll use the short "user-level" names when configuring syslogd actions for each facility, but will likely need to use the direct name (which maps to an internal constant) when configuring to what facility individual daemons are logging. (See Chapter 18's Snort syslogd configuration as an example.)

As there are different facilities, there are a number of predefined "levels" that can be sent to syslog with a log message. Levels indicate the nature of the incoming message, and are also defined within /usr/include/sys/syslog.h . Table 19.3 documents the syslogd priorities ranked from most to least serious.

Table 19.3. Syslogd Priorities

User-Level Name	Direct Name	Level Meaning
emerg or panic	LOG_EMERG	System is unusable.
alert	LOG_ALERT	Action must be taken immediately.
crit	LOG_CRIT	Critical conditions.
err or error	LOG_ERR	Error conditions.
warn or warning	LOG_WARNING	Warning conditions.
notice	LOG_NOTICE	Normal but significant condition.
info	LOG_INFO	Informational.
debug	LOG_DEBUG	Debug-level messages.
none	INTERN_NOPRI	No priority.

/etc/syslog.conf

The syslogd process is configured via /etc/syslog.conf ”a simple text file that matches selectors (facilities and levels) with an action to perform.

A selector is written as <facility>.<level> , where the values are taken from Table 19.2 and 19.3. An asterisk ( * ) can be substituted for the facility and/or level to match any facility (except mark ) or level.

NOTE

The level in a selector is the base level for a match. All levels equal to (or greater than) the base level are matched.

Actions can consist of any of the following:

• A path to a logfile (for example, /var/log/ftp.log ).

• A remote syslog server specified with the syntax @ <hostname or ip> (for example, @192.168.0.100).

• A device (for example, /dev/console ).

• A comma-separated list of usernames (for notifying users logged in via terminal sessions) or an asterisk ( * ) to notify everyone.

For example, to log all messages logged to the ftp facility at the level of error or higher to the file /var/log/ftperror.log , one would use a line like:

 ftp.error                     /var/log/ftperror.log 

CAUTION

The selector and action must be separated by a tab . If spaces are used, the entry will not work.

If you want multiple selectors to log to the same file, you can use multiple /etc/syslogd.conf lines, or add multiple selectors to a single line by separating them with semicolons. For example, to log all authentication information to the console, one could write:

 authpriv.*;remoteauth.*            /dev/console 

The asterisk can also be applied to the facility in a selector to select all facilities at a particular level. This can be combined with the none level to exclude certain facilities from the selector. For example, to log all errors, except those coming from the mail facility:

 *.err;mail.none                /var/log/allerrorsexceptmail.log 

A final "shortcut" is to write multiple facilities that share the same selector as a comma-separated list with only the last facility also containing a level, such as ftp,mail,netinfo.error , which is the same as writing ftp.error;mail.error;netinfo.error .

The Mac OS X default syslogd.conf file should look like this:

 *.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/console *.notice;*.info;authpriv,remoteauth,ftp.none;kern.debug;mail.crit /var/log/system.log # Send messages normally sent to the console also to the serial port. # To stop messages from being sent out the serial port, comment out this line. #*.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/tty.serial # The authpriv log file should be restricted access; these # messages shouldn't go to terminals or publically-readable # files. authpriv.*;remoteauth.crit                              /var/log/secure.log lpr.info                                                /var/log/lpr.log mail.*                                                  /var/log/mail.log ftp.*                                                   /var/log/ftp.log netinfo.err                                             /var/log/netinfo.log *.emerg                                                 * 

As you can see, many of the Mac OS X logfiles covered earlier are created via syslog , and the system.log file and console contain selectors that will capture messages from all available facilities. This will result in some of the redundancy you'll find when viewing the logfiles.

NOTE

Any changes to /etc/syslogd.conf can be put into effect immediately by sending the -HUP signal to the syslogd process ( killall -HUP syslogd ).

TIP

Apple does not include the output of TCP Wrappers in any of the default Mac OS X logs. If you use TCP Wrappers (which log to the syslog facility), you may consider adding this line to /etc/syslog.conf:

 syslog.err                         /var/log/secure.log 

A Network syslog Server

Creating a centralized network log server can help get you keep an eye on all your systems without having to directly watch dozens of different machines and files. Any centralized server should not be directly Internet accessible, or it could easily be subjected to a DoS attack based on an overwhelming number of logging requests.

To start syslogd in " insecure " mode, you must modify /System/Library/StartupItems/SystemLog so that syslogd starts with the -u switch. In this mode, syslogd listens on UDP port 514. Edit the file /System/Library/StartupItems/SystemLog/SystemLog and change the line

 syslogd 

to read

 syslogd -u 

Client machines can then be configured to send log messages to the centralized server. The default Mac OS X syslog.conf , for example, could be rewritten to send its logs to a logserver at 192.168.0.100 like this:

 *.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/console *.notice;*.info;authpriv,remoteauth,ftp.none;kern.debug;mail.crit @192.168.0.100 # Send messages normally sent to the console also to the serial port. # To stop messages from being sent out the serial port, comment out this line. #*.err;kern.*;auth.notice;authpriv,remoteauth.none;mail.crit /dev/tty.serial # The authpriv log file should be restricted access; these # messages shouldn't go to terminals or publicly-readable # files. authpriv.*;remoteauth.crit      @192.168.0.100 lpr.info                        @192.168.0.100 mail.*                          @192.168.0.100 ftp.*                           @192.168.0.100 netinfo.err                     @192.168.0.100 *.emerg                                                 * 

Thankfully, syslog -written logs contain the hostname of the machine generating the message. Logs follow the format:

  <time stamp> <host generating message> <process name> <log message>  

This enables an administrator to quickly grep through the contents of a combined logfile for a specific machine's messages.

Whether you choose to create a centralized server for a network or simply monitor the activity on your personal computer, a knowledge of syslogd will help make it easier. The remainder of this chapter will be dedicated to helping you decipher the contents of your Mac OS X logfiles and automate the process of tracking them.

NOTE

Unfortunately, knowing how to use syslog isn't going to help you read and analyze your logs. The messages that are written to the logging daemon are arbitrary ”determined by the program that sends them. The facility and level are also arbitrary, which can lead to log messages that you don't necessarily feel are appropriately classified , but the author did.

---

Top