Section 30.1. Objective 1: System Logging | LPI Linux Certification in a Nutshell (In a Nutshell (OReilly))

30.1. Objective 1: System Logging

System logging on Linux (and Unix) is done via the syslog service. One of the strong points of this service is that it can log over a network, so you can set up a central logging server. Events from many systems are then logged in chronological order and important messages can be checked in one place instead of on each of your hosts.

Warning: The weak point about using syslog over a network is that there is no security. Your central syslog server has no way to authenticate who sent a given message. In addition, messages are sent by UDP, which means that a message can be lost. UDP further weakens security, because it is easily the victim of IP spoofing. On top of that, the message is transported in plain text.

Warning: In other words, your remote logging is about as secure as your network. You can trust your central syslog as far as you can trust all the users that have access to your network. If you need more security than this, you should look at alternative syslog implementations, such as syslog-ng (see http://www.balabit.hu/products/syslog-ng/ for more information).

30.1.1. Setting Up a syslog Server

The default configuration of the Linux system log daemon, syslogd, is secure because it does not listen on the network. To enable network listening, you have to customize your syslog init script, and syslogd must be started with the -r option.

In Debian, write directly into /etc/init.d/sysklogd to insert the -r into the SYSLOGD variable. On Red Hat and many other distributions, edit /etc/sysconfig/syslog, and change the SYSLOGD_OPTIONS variable. After changing the initialization script, restart the service. On Red Hat, the rc filename is /etc/init.d/syslog.

You can verify that syslogd listens on the network by running the following command. It should produce at least one line exactly like the one shown. The 0.0.0.0:514 means that your syslogd listens to the "any" interface at port 514.

 $ netstat -an | grep 514 udp        0      0 0.0.0.0:514             0.0.0.0:*

Tip: You can restrict which hosts your syslogd will log from by specifying the -l option and a colon-separated list of simple hostnames (not FQDNs). An informed enemy can defeat this check trivially due to the inherent insecurity of UDP. You should have this in the back of your head before trusting logged data.

You should increase the security using some firewall rules such as these:

 iptables -P INPUT DROP iptables -A INPUT -p udp --dport 514 -j ACCEPT

On a host dedicated to being a log server, these commands restrict it to accepting only packets addressed to the 514 UDP port and dropping all other incoming traffic.

30.1.2. Setting Up syslog Clients

Setting up a client for your central syslog server is a simple matter of adding one line, similar to the following, to your /etc/syslogd.conf file. This line logs messages from all syslog facilities with priority debug and higher to the host roke.langfeldt.net.

 *.debug                         @roke.langfeldt.net

Both facility and priority are syslog concepts. Recognized facilities are authpriv, cron, daemon, kern, lpr, mail, mark, news, security or auth, syslog, user, uucp, and local0 through local7. An asterix (*) is allowed as a wildcard to cover all facilities.

Priorities are a hierarchy, at the bottom of which is debug. When you specify a priority, all priorities from that level up are sent to the log destination. Recognized priorities in ascending order are info, notice, warning or warn, err or error, crit, alert, and emerg or panic.

The syntax of the syslogd.conf file is quite rich and flexible. You can also have it both ways: logging remotely does not exclude logging locally.

With your configuration file prepared, restart the syslog service on the client by using the initialization script or by just typing:

 kill -HUP 'cat /var/run/syslogd.pid'

which forces the daemon to reread the configuration file. If you now look at /var/log/messages (on most distributions), you should see the same message as is logged on the client at restart:

 ... Jan 12 12:30:00 roke CROND[3580]: (root) CMD (   /sbin/rmmod -as) Jan 12 12:37:45 roke -- MARK -- Jan 12 12:37:47 10.0.0.5 syslogd 1.4.1: restart.

The restart message shows that communication is established. If no message is received, check your setup and firewalls. You can send messages from the command line with the logger program. It lets you specify the priority and the prefix or label on the message. The following command:

 $ logger -p notice -t logspam 'hello world'

results in a log message such as the following on the server:

 Jan 12 12:59:51 10.0.0.5 logspam: hello world

30.1.3. Using the Central Logs

The great thing about central logs is that they are much easier to manage than logs scattered about individual hosts, and it becomes very simple to search for interesting events in them.

Exactly what constitutes an interesting event is for you to decide. Some syslog analysis packages on the Net offer good ideas of many different kinds of interesting events; these packages might make the job a whole lot easier for you.

For the sake of example, we'll assume that all log lines containing the strings error and warning are interesting. The grep command can be used to extract lines matching a pattern (regular expression) from a file or stream. The straightforward approach to a log extraction command is something like this:

 #  grep -i warning /var/log/messages#  grep -i error /var/log/messages

As you add new strings to search, for this will tend to become tedious, because each command has to read through the whole file. A good alternative is to use egrep with a more complex regular expression:

 #  egrep -i '(warning|error)' /var/log/messages

This at least will make only one pass of the logs. But the command line will become unruly as you add strings. As luck would have it, someone has already thought about this and all the commands of the grep family accept a -ffile option. This causes them to read the regular expressions in from a file, which is much neater and easier to maintain. Just put the things you're interested in into /etc/interesting-in-logs and the command line to invoke the search stays simple.

Another problem in searching for short and simple patterns is that you usually get back a lot of log messages you just know will never turn out to be interesting. On an IDE system that runs smartd, for example, you will get a lot of messages saying Raw_Read_Error_Rate changed, which is probably not very interesting and which certainly can be worrisome for someone who does not understand what's going on. They can be removed by a negative grep command such as grep -v Raw_Read_Error_Rate. This negative search expression should be as specific as possible, so as to not eliminate error messages that really are important. Thinking you know about all your problems can be quite a bit worse than knowing that you don't know about them. Here, too, you will quickly run into the problem of too many strings to remove. The solution again is the -f option.

A complete example command therefore is:

 # grep -f /etc/interesting-in-logs /var/log/messages | \ >    grep -v -f /etc/interesting-in-logs-not

Remember that Linux also creates several other log files, such as the security or auth log files that contains events about logins, failed logins , and other security-related issues. A failed login, for example, results in "Failed password for" messages. Occasional failed logins are to be expected because users mistype their names and passwords, but repeated messages of that sort can indicate an attack. So let's consider how to tell when you're being inundated with failed login attempts.

The full message looks like this:

 Jan 12 08:07:07 roke sshd[1935]: Failed password for root from 10.0.0.5 port 37370 ssh2

If you remove the timestamp and the process ID at the beginning of the message, as well as the port number on the end, you get lines that can be counted and presented as a failed logins summary. The cut command was introduced in Chapter 5, in the section "Process Text Streams Using Text-Processing Filters," and is suited for this task.

 # grep 'Failed password' secure | cut -d' ' -f 4,6-11 roke Failed password for root from 10.0.0.5 roke Failed password for root from 10.0.0.5 roke Failed password for janl from 10.0.0.5 roke Failed password for janl from 10.0.0.5

If you add some more plumbing to this, you get a summary that's easily readable:

 # grep 'Failed password' secure | cut -d' ' -f 4,6-11 | sort | uniq -c | \   sort -n       2 roke Failed password for janl from 10.0.0.5       2 roke Failed password for root from 10.0.0.5

And this may well be the sort of activity summary that can alert you to security problems. This illustrates another great thing about a central logging server: as long as an intruder does not compromise the server itself, your logs are out of reach for an intruder on one of your other hosts, and she cannot modify the logs to hide her activities. For this reason, a central log server is even better if it is well secured.