Troubleshooting with Logs | Building Tablet PC Applications (Pro-Developer)

To paraphrase an old song associated with a restaurant in Berkeley, California: You can have almost any log you want in a Linux restaurant. The menu of available log files is impressive. You can configure logs by service or the severity of the problem.

You ve already seen the workings of installation log files in Chapter 03 . In this section, you ll review log files to see what happened with many Linux services.

Log files are governed by the syslog and kernel log daemons, syslogd and klogd , as configured in /etc/syslog.conf . Both daemons are active by default.

Log File Categories

You can use a system log file to diagnose a problem with installation, booting, specific services, and more. Logs can be further subdivided into eight categories, listed here in descending order of importance:

emerg (emergency)
alert
crit (critical)
err (error)
warning
notice
info
debug

Log files are organized as described in the Linux /etc/syslog.conf configuration file. Take a look at the default Red Hat Linux version of this file in Figure 13.5.

Figure 13.5: /etc/syslog.conf

As you can see, most logs are located in the /var/log directory. If you activate kernel messages, they are normally sent to the console (your screen). Some daemons, such as Internet Network News ( innd ), include additional specifications in this file.

Logs are maintained through a standard cron job, logrotate . As discussed earlier, it rotates log files on a weekly basis. Thus, the /var/log/boot.log.1 file is from the previous week.

Take the first active line in this file, which specifies messages associated with several daemons. For example, the first statement, *.info , sends all messages of info level and higher ( notice , warning , err , crit , alert , and emerg ) to the appropriate log file.

System Logs

Now let s look at some of the system logs in the /var/log directory. The dmesg file consists of basic boot messages associated with starting Linux. The message file includes additional process messages after Linux boots on your computer. The boot.log file lists messages related to the starting and stopping of daemons. And wtmp helps you monitor logons .

Getting the Rest of the dmesg

In Chapter 11 , you learned about how /var/log/dmesg helps you determine whether Linux detected your hardware. There s one other critical item at the end of this file: whether Linux has properly mounted your filesystems and swap space. If the mount was successful, you should see messages similar to the following:

 EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,6), internal journal Adding Swap: 257032K swap-space (priority -1) kjournald starting. Commit interval 5 seconds

This tells you that Linux has successfully mounted an ext3 filesystem with an internal journal on a partition. The kjournal daemon ( kjournald ) does the actual work of keeping the filesystem journal up to date. You ll see additional lines like this for each Linux partition.

Other /var/log/messages

Other messages associated with hardware and services are documented in /var/log/messages . A couple of examples are shown in the following excerpt:

 Sep 23 11:03:00 RH9 kernel: sb: No ISAPnP cards found, trying standard ones... Sep 23 11:03:00 RH9 kernel: SB 4.13 detected OK (220) Sep 23 11:46:46 RH9 sshd(pam_unix)[1129]: session opened for user root by LOGIN(uid=0) Sep 23 11:46:51 RH9 sshd(pam_unix)[1129]: session closed for user root

Each line in this file includes some basic characteristics ”such as the date, time, hostname, and service associated with each message. If available, the username and process identifier are also listed.

You can see two important developments in the code. First, Red Hat Linux has detected a Sound Blaster sound card during the boot process. Next, someone has successfully accessed the RH9 computer through sshd , the Secure Shell daemon. As you ll see in Chapter 22 , sshd is a critical tool that can help you administer a computer remotely.

But you may have a security breach. If the noted login is not authorized, a cracker may have broken into your system. See Chapter 22 for techniques that you can use to secure your Linux system.

Note	In the Linux world, hackers are good people who just want to create better software. Crackers, on the other hand, are people who try to break into your system.

Analyzing the /var/log/boot.log

When services or daemons start and stop, they are listed in /var/log/boot.log . Take the example shown in Figure 13.6. The first line shown is, in fact, the last message of a shutdown on January 15. The second message is the first daemon started when you boot Linux.

Figure 13.6: boot.log

Some services are associated with other parameters. For example, the keytable parameter shown in Figure 13.6 loads the keymap associated with your keyboard. In addition, the network service initializes the loopback interface.

Detecting Remote Logins

Login records are kept in a database file, /var/log/wtmp . You can use the utmpdump command to make this file readable. Take a look at Figure 13.7; this is part of the output when I ran utmpdump /var/log/wtmp . Note the login from IP address 128.99.1.64.

Figure 13.7: Checking login activity

Warning

You should know the IP addresses for your LAN. If your network does not include some of the addresses shown in Figure 13.7, and you don t have any remote users, be afraid. Someone may have tried to break into your system. Chapter 22 includes techniques designed to block logins from suspicious networks.

Daemon Logs

Most Linux daemons, such as crond , httpd , and smbd , are configured with log files in the /var/log directory. Each log file can provide clues as to the success or failure of any particular service. A clean example is shown in Figure 13.8, a view of the /var/log/cron file.

Figure 13.8: /var/log/cron

Figure 13.8 illustrates the date and time cron jobs were executed. You may recognize these as the standard cron jobs listed earlier in this chapter. While the times are different from those specified in /etc/crontab , it simply means that this particular Linux computer was not running at the specified time.

The following excerpt from /var/log/httpd/access_log tells you about one of the clients for your web server; for example, that particular client used the Lynx web browser from the U.S. eastern time zone during daylight saving time (-0400).

 localhost.localdomain - -[23/Sep/2003:14:05:26 -0400] "GET / HTTP/1.0" 403 2898  ? "-" "Lynx/2.8.5dev.7 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/0.9.6b"

The following excerpt (from /var/log/samba/ smbmount .log ) shows a connection to a Microsoft Windows share through Samba:

 [2003/09/23 13:53:32, 0] client/smbmount.c:send_fs_socket(383) mount.smbfs: entering  ? daemon mode for service \laptop2\downloads, pid 5711

As you add more daemons to your Red Hat Linux system, more log files will appear in the /var/log directory. However, log files don t have to be stored in /var/log; it s determined by the configuration files associated with each daemon.

Other Logs

The /var/log directory contains a number of other log files. As you add more services, more log files will appear. Therefore, Table 13.4 is not a comprehensive list.

Table 13.4: */var/log* log files
File	Function
ksyms	Exported kernel symbols, such as drivers and modules.
lastlog	Specifies last login time and location, based on the lastlog -u username command.
maillog	Anything related to mail servers, such as startup, shutdown, aliases, and errors related to sendmail .
news	A directory of log files related to the InterNetNews (INN) server.
rpmpkgs	Currently installed RPMs.
scrollkeeper	For documents, especially in GUIs.
secure	Anything related to secure connections, including ssh and xinetd .