Linux Logging Facility

One of the key requirements of enterprise systems is to log pertinent events that happen on the system to aid in system management and post failure system debugging. Fortunately, Linux provides an excellent, fully configurable, and simple logging facility.

All Linux logs are in plain text, so any text tool can be used to view them, such as vi, tail, more, or less. A browser, such as Mozilla, can be used to display a log file and provide search capability. Scripts can also be written to scan through logs and perform automatic functions based on the contents.

The main location for Linux logs is in the /var/log directory. This directory contains several log files that are maintained by the system, but other services and programs can put their log files here as well. Most log files require root privilege, but this can be overcome by simply changing the access rights to these files.

/var/log/messages

The /var/log/messages log is the core system log file. It contains the boot messages when the system comes up as well as other status messages as the system runs. Errors with I/O subsystem, networking, and other general system errors are logged in this file. Messages from system services, such as DHCP servers, are also logged in this file. Messages indicating simple actions on the system, such as when someone becomes root, are also listed here.

/var/log/XFree86.0.log

The /var/log/XFree86.0.log shows the results of the last execution of the XFree86 X Window server. If there are problems getting the graphical mode to come up, this file usually provides an answer as to what is failing.

In addition to these two log files, there might be other log files in the /var/log directory that are maintained by other services and applications running on the system. For example, there might be log files associated with running a mail server, resource sharing, or automatic tasks.

Log Rotation

Log files can become large and cumbersome, especially on systems that have been running for long periods of time. To solve this problem, Linux provides a tool, logrotate, to rotate the logs so that the current log information does not get mixed up with older messages. The logrotate command can be run manually as needed, or it can be run automatically on a periodic basis. When executed, logrotate takes the current version of the log files and adds a sequence number to the end of the log filename. The larger the sequence number after the log filename, the older that file is. For example, messages.2 is older than messages.1, which is older than the current messages file. The automatic behavior for logrotate can be configured using the /etc/logrotate.conf file. More details are available on the logrotate man page.

In addition to /var/log/messages, dmesg provides a quick view of the kernel messages, which can be helpful when you want to know what happened during the last system boot.

Logger

The logger facility generates system log messages out of your own scripts and programs that are recognized and processed by the syslogd daemon. This lets you send messages to the log files without worrying about the format of the log files or whether the logging facility has been customized.

Customized Logging

The Linux logging facility consists of two daemons: klogd for kernel messages and syslogd for user-space messages. These daemons can be configured through the /etc/ syslog.conf and /etc/sysconfig/syslog files. You can edit the /etc/syslog.conf file to specify what you want to do with a particular type of message. For example, you can specify that critical kernel messages should be put on a remote host for security reasons.

Here is an example of customized logging taken from the /etc/syslog.conf man page:

 kern.*                         /var/adm/kernel kern.crit                      @finlandia kern.crit                      /dev/console kern.info;kern.!err            /var/adm/kernel-info 

The first statement directs any message from the kernel to the file /var/adm/kernel.

The second statement directs all kernel messages of the priority crit and higher to the remote host finlandia. Sending critical log messages to the remote host can help prevent malicious users from modifying the message log files on the local system to cover their tracks. It can also be useful in the event the local system crashes and the disks get irreparable errors.

The third statement directs these messages to the actual console, so the person who works on the console will see them.

The fourth line tells the syslogd to save all kernel messages that come with priorities from info up to warning in the /var/adm/kernel-info file. Everything from err and higher priority is excluded.

The ability to customize logging like this provides a great deal of flexibility and control over the Linux environment.