11.6. So Many Server Logs
I believe it's helpful for any Linux user to review her own logs on a regular basis. Familiarity can help any geek learn the value of logs. For one thing, log entries will be associated with failed logins, which suggest that a cracker is trying to break into your system. But logs can do much more. For example, web logs can give you a feel for where your customers are coming from, in terms of geography; clicked links associated with web ads; how long they stay on your web site; and more.
As a Linux administrator, chances are good that you're administering a substantial number of Linux computers. It may be useful to consolidate the logs on a single system. If a server goes down, you'll have the logs from that server available on the central log server. When there are problems, such as "critical" error messages, you may want an email sent to your address. You may need tools to help you go through all of these logs.
11.6.1. Central Log Servers
Logs on our selected distributions are governed by the system and kernel log daemons. While Red Hat/Fedora and Debian combine these daemons into a single package (sysklogd), SUSE includes them in separate packages (syslogd and klogd). While there are minor variations in how they're configured, they're all governed by a logfile in the same location, /etc/syslog.conf.
If you want to dedicate a specific system as your central log server, first make sure you have enough space on that system. It may help to configure logs, such as those in the /var/log directory, on a separate partition so that they can't fill up critical system partitions if they get too big. For more information, see the next annoyance.
On the system that you're configuring as a central log server, you'll have to configure the system log daemon (syslogd) to accept remote connections. The simplest way to do so is to stop the daemon, and then start it again with the -rm 0 switch. The way you implement this varies slightly by distribution:
Once you've made the configuration changes, you can implement them by restarting the system log daemon on each computer with the following command:
On Debian Linux, the script's location is slightly different:
Naturally, if there's a firewall between the log server and log clients, you'll need to make provisions in that firewall to allow traffic through port 514. As you can see in /etc/services, that's the standard port for system log communications. To make sure your system log service now receives from remote computers, check your /var/log/syslog (or, if that file doesn't exist, /var/log/messages) for the following entry (the version number may vary):
syslogd 1.4.1: restart (remote reception).
11.6.2. Forwarding Server Logs to a Central Server
Now that the central server is ready, you can configure your other Linux systems to send copies of their logs to that computer. The log configuration file on each of our preferred distributions is /etc/syslog.conf, and the key directive is straightforward. If you want a copy of all logs sent to the logmaster.example.com computer, all you need in that file is:
Unfortunately, the system log service can't handle fully qualified domain names. Logs on the central server from your remote systems will have only the regular hostnames.
If you're just concerned with kernel-based issues, to help diagnose shutdowns, you can send just the kernel messages to the remote log server:
11.6.3. Logwatch Monitoring
There are many excellent tools for monitoring logfiles. Many geeks even create their own scripts for this purpose. One excellent source for different monitoring tools and scripts is Automating Unix and Linux Administration by Kirk Bauer (Apress).
One of the major standards for log monitoring is known as Logwatch. It's available from both the Debian and Red Hat/Fedora Linux repositories. A logwatch RPM that works on SUSE is available from the Logwatch home page at http://www.logwatch.org.
Logwatch is organized into three groups of files. The overall configuration file is logwatch.conf. Other logfiles for many individual services are organized in a services/ subdirectory. The logfiles are placed in groups based on configuration files in a logfiles/ subdirectory. The actual directory varies by distribution, or by the release that you may have installed from http://www.logwatch.org.
22.214.171.124. Basic Logwatch configuration
Before I show you how to configure the basic Logwatch configuration file, I need to review its location on your system. If you've downloaded the latest version from http://www.logwatch.org, you'll need to make sure key settings are compatible with the scripts and configuration directories for your distribution.
The standard Logwatch configuration file is logwatch.conf. You can find it in the /etc/log.d/conf or /etc/logwatch/conf directories. As described earlier, there is no standard SUSE Logwatch package.
If you've downloaded the latest version from the Logwatch home page, you'll find key configuration files in different locations. The logwatch.conf configuration file (as well as default services and configuration logfiles) is stored in /usr/share/logwatch/default.conf; detailed configuration changes can be added to files in the /etc/logwatch/conf directory.
Administrators are now encouraged to add changes to Logwatch settings to override.conf and patterns that Logwatch should drop to ignore.conf. But those are advanced settings beyond what I can cover in this annoyance. Refer to the Logwatch web site for the latest information.
Logwatch's standard directives include:
126.96.36.199. Logwatch service configuration files
The service configuration files associated with Logwatch are stored in a service/ subdirectory. While the list of files may seem extensive, don't worry about configuring each file. The defaults are generally fine, unless you want to specify a special file group. One example where you may want to specify a special group is with the Clam AntiVirus software (www.clamav.net). The following is based on the package downloaded from the Clam AV web site.
By default, the following directive in the clamav.conf file (along with the LogDir=/var/log directive in logwatch.conf) sends logs from this service to the standard /var/log/messages:
LogFile = messages
As it may be inconvenient to have so much traffic in /var/log/messages, you could send logs to a different file, such as /var/log/clam-update, with the following directive:
LogFile = clam-update
188.8.131.52. Logwatch log groups
The Logwatch service can help you organize a wide variety of logfiles, as configured in the logfiles/ subdirectory. It includes a number of configuration files (with .conf extensions).
184.108.40.206. Logwatch scripts
Logwatch includes a wide variety of scripts in the shared/ subdirectory. They're generally configured to help you search through logs collected by this service.