Hack86.Fine-Tune the syslog Daemon

Hack 86. Fine-Tune the syslog Daemon

You can't see problems that aren't being reported. Correctly setting up the system log daemon and logging levels ensures that you always know what's going on.

Linux systems log boot information, process status information, and a significant amount of access and error information in the system logfile, /var/log/messages, using a system daemon known as syslog. But when was the last time you looked at this file? If you've never spent any time fine-tuning the syslog daemon, your system logfile probably contains a tragically jumbled mess of cron completion notices, boot notices, MARK entries, and any number of other service or daemon log messages. Imagine if you could configure syslog to dump all that information where you wanted it, and sort it all too…. Well, this is Linux we're talking about here, so of course you can configure syslog any way you want!

9.10.1. Making Sense of syslog.conf

A configuration file called /etc/syslog.conf controls the syslog daemon. As unimaginative as the config file's name might be, learn it well because this is a file you'll need to become very familiar with if you want to master the intricacies of Linux system logging. The file may not make a whole lot of sense upon first glance, but here's a simple syslog.conf file that I'll use to explain the syntax further:

 # Log all kernel messages to the console. # Logging much else clutters up the screen. # kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg   * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log 

As you can see in the noncommented lines in this example, there are three main parts to each active line of the configuration file. The first entry on a line is called the facility, which is the underlying subsystem that creates the log messages for the logfiles. There are 13 predefined system facility values: auth, authpriv, cron, daemon, ftp, kern, lpr, mail, mark, news, syslog, user, and uucp. In addition to these, there are also eight others, named local0 through local7, which are for programs to use when implementing their own syslog messages. Each of the predefined facilities refers to a specific aspect of the system. For instance, auth refers to the Linux authorization system, including programs such as login and su. The mark facility is used internally for syslog, and should be left alone for the time being. The daemon facility is for other system daemons that are not listed specifically. You can represent all available facilities by using the asterisk (*) symbol.

The second part to a configuration line is the priority, which is separated from its associated facility by a period. Every time a part of the system sends a message to syslog, that message is coded with a priority. Basically, the program is letting syslog know how important this message is. From lowest to highest, the priority levels are debug, info, notice, warning, err, crit, alert, and emerg. The higher the priority, the more important the message is. Once you hit the emerg priority, the system is rapidly approaching a kernel panic and is probably unusable. You can represent messages of any priority by using the asterisk symbol. For example, local7.* means "messages of any priority from the local7 facility."

The third and final aspect of the configuration line is the action. This is basically just a short section that tells syslog what to do with the information it has received. To better explain this, let's look at an example line from the sample configuration file provided above:

 # Log cron stuff cron.* /var/log/cron 

Few things are more annoying than scrolling through /var/log/messages and having to wade through all the cron messages, so this kind of configuration option comes in handy. This example means that messages of all priorities issued by the cron facility should be sent to the /var/log/cron logfile. As mentioned previously, the asterisk is a wildcard feature that tells syslog to apply the same rule to every message from cron, regardless of its priority. You can do similar things with the asterisk wildcard for the facility, such as instructing syslog to send every message of priority warning or higher to a specific logfile:

 *.warning /var/log/problems 

9.10.2. Real-Time Alerts from the System Log

Other wildcard features that can be used include the at sign (@), for sending messages to remote syslog hosts; a dash (-), for telling syslog not to sync the disks after every message; and an asterisk in the actions section of the configuration to alert everyone on the system to an issue. For instance, look at the following example from the sample configuration file:

 # Everybody gets emergency messages *.emerg   * 

The final asterisk on this line tells syslog to send a message out to every user online via the wall (Write to ALL users) command to let them know of any emergency conditions. These messages will appear in every active terminal window on the system. You can think of configurations like this as Linux's emergency broadcast system.

Another interesting line in the example syslog.conf file shown earlier in this hack is the line that addresses kernel syslog messages. Rather than being sent to a logfile, all these messages are sent to the console instead. One popular trick using this feature is to direct many of the syslog messages to a virtual console instead of the main console. I often do this on machines that aren't used much for local work but still have monitors. For example, specifying this line:

 auth,kern.* /dev/tty5 

allows me to see the syslog messages of everyone who logs onand any issues with the kernelsimply by switching the machine to virtual console 5 (Alt-F5) and leaving it there with the monitor on. Now, whenever I walk by that machine, I can keep track of users logging on and off, or anything else I've set it up to do. When I need to work on the server and that would be in the way, I just switch back to my primary console (Alt-F1), and the messages continue to be sent to console 5.

9.10.3. Centralizing Logs for Convenient Access

Another interesting syslog option is remote logging. While syslog itself allows for remote logging, there is a more robust solution to be found in syslog-ng [Hack #87], a new version of syslog. syslog allows you to send messages to remote hosts, but it does so in plain text across the network, so you should use this feature with caution. Here's how it works: by adding an at sign and a hostname or IP address in the action section of the configuration file, you can specify that syslog send its messages to another waiting remote syslog server. The remote syslog server will need to have the syslog daemon started with the r option to allow it to listen on port 514 for incoming syslog messages. The following line shows an example of sending all critical kernel messages to the remote machine aardvark for safekeeping.

 kern.crit @aardvark 

Remote logging can be extremely helpful in the event of a system crash, as it allows you to see log messages that you might otherwise be unable to access (since the system that issued them is down). As previously mentioned, these messages are sent in plain text across the network, so be sure to use syslog's remote logging with cautionand never do it across the Internet. Also, note that if you send certain types of messages to a remote log server, they are not recorded locally unless you create another entry that also sends those same messages to the local log, as in the following example:

 kern.crit @aardvark kern.crit /var/log/messages 

Another interesting potential security issue with syslog's remote logging is that starting the syslog daemon with the r option to receive remote log entries means that any host can send a log message to that host. The syslog facility doesn't have a way of identifying specific hosts that it should receive messages from, so it just holds up a big electronic catcher's mitt and accepts anything that comes its way.

The syslog daemon can be customized in many different ways, but it's somewhat dated in terms of both capabilities and security. "Centralize System Logs Securely" [Hack #87] provides newer and even more configurable approach to system logging.

9.10.4. See Also

• man syslog

• "Centralize System Logs Securely" [Hack #87]

Brian Warshawsky