13.1 Logging

Logging is the process by which almost every aspect of machine and cluster operation can be recorded for future reference. In a cluster, just like in a stand-alone Linux machine, the operating system and system services will normally be configured to log significant event information. At a higher level, workload management software, application libraries, and even user applications can and often do generate logging information.

Many techniques are available for logging, but in the Linux environment a service called syslog, and its associated library functions (see the syslog man page) is the standard way used by system logging services. While logging performed using syslog may be the most common, any application or service can use any technique it wishes for logging. The most simple logging technique is opening a file and writing textual event or error messages to it.

Just as logging techniques can be very different, log file locations can also vary. The most common location where Linux distributions place log files is in the '/ var/log' directory.

Managing clusters, and especially reviewing activity and failures, requires in-depth knowledge of all the available log files and the information commonly stored in them. The following sections describe the major types of logging that cluster administrators should be familiar with.

13.1.1 Kernel logging

The Linux kernel records log messages to a special memory location called the ring buffer. Two major categories of information logged to the ring buffer are kernel and driver initialization information and significant and unrecoverable hardware failures or other unexpected kernel state information..

The kernel provides its own logging capability using this in-memory ring buffer because it needs to have logging capability independent of any other system services. To view the ring buffer use the dmesg command.

 Linux version 2.4.18-3smp (bhcompile@daffy.perf.redhat.com) (gcc version 2.96 20000731 (Red Hat Linux 7.3 2.96-110)) #1 SMP Thu Apr 18 07:27:31 EDT 2002 BIOS-provided physical RAM map:  BIOS-e820: 0000000000000000 - 000000000009f400 (usable)  BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)  BIOS-e820: 00000000000d8000 - 00000000000e0000 (reserved)  BIOS-e820: 00000000000e4000 - 0000000000100000 (reserved)  BIOS-e820: 0000000000100000 - 000000003fef0000 (usable)  BIOS-e820: 000000003fef0000 - 000000003fefc000 (ACPI data)  BIOS-e820: 000000003fefc000 - 000000003ff00000 (ACPI NVS)  BIOS-e820: 000000003ff00000 - 000000003ff80000 (usable)  BIOS-e820: 000000003ff80000 - 0000000040000000 (reserved)  BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved)  BIOS-e820: 00000000ff800000 - 00000000ffc00000 (reserved)  BIOS-e820: 00000000fff00000 - 0000000100000000 (reserved) 127MB HIGHMEM available. found SMP MP-table at 000f6760 hm, page 000f6000 reserved twice. hm, page 000f7000 reserved twice. hm, page 0009f000 reserved twice. hm, page 000a0000 reserved twice. On node 0 totalpages: 262016 zone(0): 4096 pages. zone(1): 225280 pages. zone(2): 32640 pages. ...

With some Linux distributions ring buffer contents are saved to the file '/var/ log/dmesg' at boot time, in effect preserving all the kernel initialization log information. This is valuable because, as the name implies, the ring buffer is circular. When this fixed-size buffer fills up, messages wrap around to the beginning and start overwriting the oldest messages. If you are concerned about preserving all ring buffer messages you should configure a regular cron job to write ring buffer contents to a '/var/log' file.

If you ever suspect a kernel-related issue the first place to look is in dmesg output or in files containing dmesg output in the '/var/log/' directory.

The following is an example of a kernel crash (also called an "oops") that would be logged to the ring buffer. The line starting with the label "Process" identifies the active process when the kernel crash occured.

 Oops: 0000 CPU:    0 EIP:    0010:[journal_dirty_metadata+98/368]    Not tainted EIP:    0010:[<c015fcc2>]    Not tainted EFLAGS: 00010206 eax: 0a60c4ed   ebx: 00000000   ecx: 00000bb8   edx: dfc32a40 esi: d53ccd40   edi: dfee4800   ebp: c8c2b8e0   esp: dfc35c4c ds: 0018   es: 0018   ss: 0018 Process nfsd (pid: 727, stackpage=dfc35000) Stack: dfb31480 dfc32a40 00000001 c8c2b8e0 c0158ff ...

13.1.2 System service logging

Most of the useful facilities on a machine are provided by system services or daemons. Examples include, sshd, the ssh server daemon, xinetd/inetd, the extended Internet service daemons, cron for executing scheduled commands, and others. These and most other system services log using the syslog facility to files in the '/var/log' directory.

The most common and useful system service log files that a cluster administrator should be familiar with are:

Table 13.1: Most useful system log files.
File	Contents

'/var/log/messages'	Many common system service messages like sshd, automount, ntp, and some kernel messages
'/var/log/auth.log'	security and authorization messages
'/var/log/kern.log'	kernel boot time messages (dmesg)
'/var/log/daemon.log'	system service messages

13.1.3 Workload Logging

Whether you are running PBS, Maui, Condor, or any other scheduling or resource management software, you will probably have log files produced by these software packages. OpenPBS or PBSPro, for example, default to logging in the '/var/ spool/pbs' (or '/var/spool/PBS'). You should check the documentation of your workload management software for information on logfile locations and become familiar with the contents of those files.

13.1.4 Syslog capabilities and limitations

The syslog utility has the ability to both record events to the log files described earlier and to forward the events to other machines. In a cluster environment a common and recommended practice is to designate a central machine where the most commonly referenced log messages get collected and to configure this machine to receive and log syslog messages from other machines.

If you have configured a central syslog server you will encounter a significant architectural limitation from the standard syslog software. Because it uses unreliable data network packets (UDP), under high syslog traffic syslog messages may be lost. Fortunately where there is a need there is an open-source tool to address the need. One useful tool that overcomes this scalability challenge is syslog-ng. For additional information and to download syslog-ng visit:

http://www.balabit.com/products/syslog_ng/.

When a machine stays up for a long time the contents of the '/var/log' directory can grow large. This is especially true on the central machine where you collect logging for the entire cluster. Many Linux distribution automatically run log rotation cron jobs that periodically rename and compress the contents of '/var/log' files. Log rotation often also includes automatic deletion of the oldest rotated log files. Whether your distribution has pre-configured logrotate capability or not you should review all the logging files generated by system services and scheduling and resource management services to ensure that log files are rotated, compressed, and retained based on your particular requirements.

13.1.5 Tools to Monitor Log Files

When failures happen you will often need to investigate them using log files. As you become more familiar with the most frequent failures and the log entries than accompany them you may find yourself wanting to take automated corrective action, or at a minimum wanting automated e-mail notification that the failure occured. Both of these are possible thanks to a class of tools called log watchers that can constantly watch log files for configurable log entry strings and take configurable action.

The following are several examples of log-watching tools:

LogCheck, http://www.psionic.com/abacus/logcheck/
swatch, http://swatch.sourceforge.net/
LogSentry, http://www.linux-sxs.org/files/psionic/
LogDog, http://caspian.dotconf.net/menu/Software/LogDog/