Recipe 9.27 Directing System Messages to Log Files (syslog)

9.27.1 Problem

You want to configure the system logger to use an organized collection of log files.

9.27.2 Solution

Set up /etc/syslog.conf for local logging:

/etc/syslog.conf: # Messages of priority info or higher, that are not logged elsewhere *.info;\ mail,authpriv,cron.none;\ local0,local1,local2,local3,local4,local5,local6,local7.none \                                 /var/log/messages # Messages of priority debug, that are not logged elsewhere *.=debug;\ mail,authpriv,cron.none;\ local0,local1,local2,local3,local4,local5,local6,local7.none \                                -/var/log/debug # Facilities with log files that require restricted access permissions mail.*                          /var/log/maillog authpriv.*                      /var/log/secure cron.*                          /var/log/cron # Separate log files for local use local0.*                        /var/log/local0 local1.*                        /var/log/local1 local2.*                        /var/log/local2 local3.*                        /var/log/local3 local4.*                        /var/log/local4 local5.*                        /var/log/local5 local6.*                        /var/log/local6 # Red Hat usurps the local7 facility for boot messages from init scripts local7.*                        /var/log/boot.log

After you modify /etc/syslog.conf, you must send a signal to force syslogd to reread it and apply your changes. Any of these will do:

# kill -HUP `pidof syslogd`

or:

# kill -HUP `cat /var/run/syslogd.pid`

or:

# /etc/init.d/syslog reload

or:

# service syslog reload                            Red Hat

9.27.3 Discussion

When your kernel needs to tell you something important, will you notice? If you are investigating a potential break-in last night, will you have all of the information you need? Staying informed requires careful configuration and use of the system logger.

The system logger collects messages from programs and even from the kernel. These messages are tagged with a facility that identifies the broad category of the source, e.g., mail, kern (for kernel messages), or authpriv (for security and authorization messages). In addition, a priority specifies the importance (or severity) of each message. The lowest priorities are (in ascending order) debug, info, and notice; the highest priority is emerg, which is used when your disk drive is on fire. The complete set of facilities and priorities are described in syslog.conf(5) and syslog(3).

Messages can be directed to different log files, based on their facility and priority; this is controlled by the configuration file /etc/syslog.conf. The system logger conveniently records a timestamp and the machine name for each message.

It is tempting, but ill-advised, to try selecting the most important or interesting messages into separate files, and then to ignore the rest. The problem with this approach is that you can't possibly know in advance which information will be crucial in unforeseen circumstances.

Furthermore, the facilities and priorities are insufficient as message selection criteria, because they are general, subjective, and unevenly applied by various programs. Consider the authpriv facility: it is intended for security issues, but many security-related messages are tagged with other facilities. For example, the message that your network interface is in "promiscuous mode" is tagged as a kernel message, even though it means someone could be using your machine as a packet sniffer. Likewise, if a system daemon emits a complaint about a ridiculously long name, perhaps filled with control characters, someone might be trying to exploit a buffer overflow vulnerability.

Vigilance requires the examination of a wide range of messages. Even messages that are not directly associated with security can provide a valuable context for security events. It can be reassuring to see that the kernel's "promiscuous mode" message was preceded by a note from a system administrator about using Ethereal to debug a network problem. [Recipe 9.17] Similarly, it is nice to know that the nightly tape backups finished before a break-in occurred in the wee hours of the morning.

There is only one way to guarantee you have all of the information available when you need it: log everything. It is relatively easy to ignore messages after they have been saved in log files, but it is impossible to recover messages once they have been discarded by the system logger: the fate of messages that do not match any entries in /etc/syslog.conf.

Auxiliary programs, like logwatch [Recipe 9.36], can scan log files and effectively select messages of interest using criteria beyond the facility and priority: the name of the program that produced the message, the timestamp, the machine name, and so forth. This is a good strategy in order to avoid being overwhelmed by large amounts of logging data: you can use reports from logwatch to launch investigations of suspicious activities, and be confident that more detailed information will always be available in your log files for further sleuthing.

Even very busy systems using the most verbose logging typically produce only a few megabytes of logging data per day. The modest amount of disk space required to store the log files can be reduced further by logrotate. [Recipe 9.30] There are, nevertheless, some good reasons to direct messages to different log files:

• Some of the messages might contain sensitive information, and hence deserve more restrictive file permissions.

• Messages collected at a higher rate can be stored in log files that are rotated more frequently.

Our recipe shows one possible configuration for local logging. Higher priority messages from a range of sources are collected in the traditional location /var/log/messages. Lower priority (debug) messages are directed to a separate file, which we rotate more frequently because they may arrive at a higher rate. By default, the system logger synchronizes log files to the disk after every message, to avoid data loss if a system crash occurs. The dash ("-") character before the /var/log/debug filename disables this behavior to achieve a performance boost: use this with other files that accumulate a lot of data. Exclusions are used to prevent messages from being sent to multiple files. This is not strictly necessary, but is a nice property if you later combine log files [Recipe 9.35], as there will be no duplicate messages.

Priority names in the configuration file normally mean the specified priority and all higher priorities. Therefore, info means all priorities except debug. To specify only a single priority (but not all higher priorities), add "=" before the priority name. The special priority none excludes facilities, as we show for /var/log/messages and /var/log/debug. The "*" character is used as a wildcard to select all facilities or priorities. See the syslog.conf(5) manpage for more details about this syntax.

Messages tagged with the authpriv , mail, and cron facilities are sent to separate files that are usually not readable by everyone, because they could contain sensitive information.

Finally, the local[0-7] facilities, reserved for arbitrary local uses, are sent to separate files. This provides a convenient mechanism for categorizing your own logging messages. Note that some system daemons use these facilities, even though they really are not supposed to do so. For example, the local7 facility is used by Red Hat for boot messages.

The facility local7 is used by Red Hat Linux for boot messages. Use care when redirecting or ignoring messages with this facility.

The system logger notices changes in /etc/syslog.conf only when it receives a signal, so send one as shown. The same commands also cause the system logger to close and reopen all its log files; this feature is leveraged by logrotate. [Recipe 9.30]

When adding new log files, it is best to create new (empty) files manually so that the correct permissions can be set. Otherwise, the log files created by the system logger will be publicly readable, which isn't always appropriate.

9.27.4 See Also

syslogd(8), syslog.conf(5).