Logging | Beginning Fedora 2

One of the best features of UNIX-based operating systems is their extensive support for logging, and in Fedora Core the same is true. Most of the system s core components , including the kernel as well as other services and applications, such as telnet and ftp , write the status of their significant actions to specific text files. You can use this information to get a snapshot of the system to see what is going on at any given time. Viewing the contents of these log files enables you to find the following:

What is happening as the system boots
If a device failed to initialize at boot time
When a user logged in and from what address
Any suspicious security alerts, such as failed logins
If a user tried to send e-mail to a particular address
All e-mail received, from whom, and at what time
Status of all cron jobs

Your role as the system administrator is to use this information to properly maintain your system and take any corrective actions, if necessary. There is no point in having all of this valuable information at your fingertips if you do not make good use of it. For example, if you notice from the log files that a particular user is repeatedly trying to telnet in from a specific unknown location and is unsuccessful , you can most likely assume that a hacker is trying to gain entrance to your system. At this point, you can take some decisive actions. You can use the hosts .deny file that was discussed in the previous section to deny complete access to that particular service for the specific host or address, and also try to inform the hacker s Internet service provider of the malicious activities.

Viewing Logs

Log files are plain text files, much like the configuration files you saw earlier in this chapter. You can find most of the system log files in one directory, /var/log , so keeping track of them is rather easy. The following table lists some of the more important files that you will find in this directory.

File	Description
boot.log	Boot messages
Cron	crond messages
Maillog	All e-mail- related messages
Messages	All informational messages except for cron, mail, and secure
Secure	Secure authentication messages

You can use either the cat , more , or less utilities from the command line or the System Logs GUI application (by selecting Main Menu>System Tools>System Logs) to view the contents of these plain text log files. Here is an example:

   # more /var/log/messages   May 22 21:22:02 localhost syslogd 1.4.1: restart. May 22 21:22:02 localhost syslog: syslogd startup succeeded May 22 21:22:02 localhost kernel: klogd 1.4.1, log source = /proc/kmsg started. May 22 21:22:02 localhost syslog: klogd startup succeeded May 22 21:22:02 localhost kernel: Linux version 2.6.5-1.358 (bhcompile@bugs.build.redhat.com) (gcc version 3.3.3 20040412 (Red Hat Linux 3.3.3-7)) #1 Sat May 8 09:04:50 EDT 2004 May 22 21:22:02 localhost kernel: BIOS-provided physical RAM map: May 22 21:22:02 localhost kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable) May 22 21:22:02 localhost kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved) May 22 21:22:02 localhost kernel:  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved) May 22 21:22:02 localhost kernel:  BIOS-e820: 0000000000100000 - 000000000c000000 (usable) May 22 21:22:02 localhost kernel:  BIOS-e820: 00000000fec00000 - 00000000fec01000 (reserved) May 22 21:22:02 localhost kernel:  BIOS-e820: 00000000fee00000 - 00000000fee01000 (reserved) May 22 21:22:02 localhost kernel:  BIOS-e820: 00000000fffc0000 - 0000000100000000 (reserved) ...

You can also view the messages log file using the System Logs GUI application (under System Tools); it shows a list of log files on the left side, and the contents of the selected log file on the right side (see Figure 11-1).

Figure 11-1

Go ahead and experiment with the other log files to see what type of information is logged. Simply select a log file from the left side, and its contents will be shown in the right frame. Unfortunately, this application does not allow you to add other log files to the menu; you are limited to the ones listed.

Application-Specific Log Files

We have found a number of system log files in /var/log . However, a number of applications, such as Web servers and database servers, store their log files in other locations. How do you go about finding them? First, you should read the individual application s documentation, or its manual pages, for more details. Usually, the manual page will have a FILES section that will list all files used by the application, including configuration and log files. If you cannot locate the files in this manner, you should look for any application-specific subdirectories in /var/log or even in the /etc directory. If that too proves unsuccessful, the best bet is to proceed to the application installation directory and try to find any files or subdirectories that contain the word log in their filename, like so:

   # find /usr/local/thttpd name '*log*' -print

Here, we are looking for any possible log files or directories in the /usr/local/thttpd directory; this is the installation directory for the thttpd Web server.

Searching Log Files

As you look through the various log files, you may soon realize that it is very difficult to keep track of failures and other alerts because of the large amount of information that is logged. For example, you may want to keep track of all login failures or certain hardware problems. To do this, you could schedule some time every day to wade through the entire content manually. However, this is far from efficient, and you could instead use your time for other system administration “related tasks . Or, you could use simple search tools to find specific information in the log files quickly and easily.

Searching Log Files Using the System Logs Application

The System Logs application also provides a filtering option (you can see it at the bottom of the window in Figure 11-2) ”this allows you to search for specific words or phrases within a log file. For example, try selecting the System Log, typing the word fail into the Filter for textbox; click the Filter button to execute the search. We found a few error messages.

The application found seven entries from the System Log file that contained the word fail. In addition, it also highlighted entries where it found the word failed; you can configure what words trigger this effect by using the application s Edit>Preferences>Alerts option.

Searching Log Files Directly

Of course, you don t have to use the System Logs application to analyze or search a log file. You can also examine a log file directly by viewing it in a text editor (such as gedit ), and using its search capability to find particular keywords, such as fail , error , denied , or unable . Or, you can use the grep command from the command line to perform the search. Here is an example that you can use to search the messages log file for all occurrences of the string fail :

   # grep -i fail /var/log/messages   May 22 21:22:05 localhost kernel: There is already a security framework initialized, register_security failed. May 22 21:22:05 localhost kernel: Failure registering capabilities with the kernel May 22 21:29:17 localhost mdmpd: mdmpd failed May 22 21:31:49 localhost gconfd (root-2686): file gerror.c: line 225 (g_propagate_error): assertion `src != NULL' failed May 22 22:42:24 localhost kernel: There is already a security framework initialized, register_security failed. May 22 22:42:24 localhost kernel: Failure registering capabilities with the kernel May 22 22:43:09 localhost mdmpd: mdmpd failed ...

As you can see, these results are nearly identical to the output produced by the System Logs application; in both cases, we found seven entries that contained the word fail. However, having the capability to search for particular log entries from the command line affords you great flexibility and convenience. For example, you could do the following:

Write a simple shell script to periodically check for certain keywords in the log files, and send a report of this search to the administrator. In fact, there are already a number of applications that do this, including Todd Atkins swatch, which is available from http://swatch. sourceforge .net .
Build a Web interface to these log files, which allows you to view warnings and suspicious activity from other computers on a local network. In fact, we will build our own such application, which you can find on the bonus online chapter (posted at www.wrox.com ).

Figure 11-2

Understanding the Logging Process

This is all great, but how do all of these messages get in the various log files, especially when these messages are all being generated by different applications? In fact, we discussed this briefly in the first section of this chapter, when we looked at the syslog.conf configuration file. Let s look more closely at this subject now.

The magic behind the entire logging process is the syslogd daemon . The syslogd daemon acts as a switchboard for messages of all types, produced by various services, daemons, and system software processes. More specifically , the daemon process listens for messages on a UNIX domain socket, /dev/log , and routes them according to the rules specified in the syslog.conf configuration file.

Note

There is one interesting detail to note. The kernel does not communicate with the syslogd daemon directly. Instead, the kernel injects messages into another daemon, klogd, which is designed specifically to handle messages from the kernel. Eventually, however, the klogd daemon communicates with syslogd, and the messages get routed to the intended destination.

Here are a few sample entries that you might find in a typical syslog.conf configuration file:

 kern.*                                   /dev/console           ## tty kern.crit                                @dev2                  ## dev2 host *.info;mail.none;authpriv.none;cron.none /var/log/messages      ## file authpriv.*                                /usr/local/bin/audit ## app. mail.*                                   /var/log/maillog       ## file cron.*                                   /var/log/cron          ## file *.emerg                                  *                      ## all users *.alert                                  root,johnson           ## users

Each entry in syslog.conf can be broken into three parts :

The first part of each entry is the type of message to be considered . For each type of message to log, there are two components: the facility and the level . In the preceding examples, kern , mail , authpriv , and cron are facilities, and crit , info , emerg , and alert represent the different logging levels.
The second part is the destination to which the message should be sent. Valid destinations include files, terminals, logged-in users, applications (via pipes), and even remote syslogd servers.
The third column simply represents user-defined comments, which are optional.

As you can see, the syslogd application is a very powerful and flexible logging mechanism. Most of the types of messages that you would ever want to see are already configured by default when you install the operating system. However, you should monitor the logs on a regular basis for any possible problems, either using the search techniques discussed in the previous section or by using other specialized applications, such as swatch . That way, you can keep your system running safely and efficiently , and intervene only when necessary.

Of course, it is very important to back up and archive log files because they represent a snapshot in time of the system s activity and status. Therefore, the next section looks at several tools and applications that allow you to archive your data safely.