Monitoring Log Files with LogSentry

Fedora has the ability to monitor and log nearly every activity that can occur on your computer. On a busy system, massive amounts of informational and error messages are produced and placed in log files. For the administrator, the hard part of monitoring log files isn’t detecting or logging security problems; the hard part is remembering to check the log files and sift out those messages that pose a threat from all the other stuff that gets logged. This section describes how to monitor log files with the LogSentry package.

The LogSentry package is a handy tool that you can use to easily manage your system log files. Because you’re more likely to glance through an e-mail than you are to remember to check log files, LogSentry puts information in front of you that may otherwise go unnoticed.

LogSentry checks log files produced by the standard Linux syslog facility, attempts to filter out messages that don’t represent any security threat, and then categorizes messages that could represent a threat and e-mails those messages to the system administrator. By default, LogSentry will check messages in the messages, secure, and mail log files in the /var/log directory. By changing the LogSentry configuration files, you can change which log files are checked, how messages are filtered, and how often log-summary e-mail messages are sent. You may also want to change which features the syslog facility monitors and the level of syslog monitoring if you are interested in debugging a particular system feature.

Note 

Fedora includes the logwatch package in its distribution. Logwatch is configured to run once each day to monitor your log files. While logwatch is not as comprehensive a tool as LogSentry, you can expect to see a logwatch e-mail message produced daily that highlights any dangerous system activity that it detects. That e-mail is sent to your computer’s system administrator.

Downloading and installing LogSentry

You can find the logsentry package at rpmfind.net and several FTP sites on the Internet. After logsentry is downloaded, to install the package, simply run the following command from the directory you downloaded it to:

 # rpm -Uhv logsentry* 

The logsentry package consists of four configuration files (in the /etc/logsentry directory), the cron file that runs logcheck.sh (in /etc/cron.d/logsentry.cron), and the logcheck.sh and logtail commands (in /usr/sbin). There are also README files in the /usr/share/doc/logcheck* directory.

Note 

Because there are different versions of LogSentry floating around on the Web, including older versions named logcheck, the locations of commands and instructions for using LogSentry may be different from what I describe here. For example, the original logcheck package may include a logcheck.sh script in /usr/bin for running logcheck. The logsentry package used here is logsentry-1.1.1-1.i686.rpm.

Setting up LogSentry

You don’t have to do anything to get LogSentry working. After LogSentry is installed, it runs once a day at midnight. The results are then e-mailed to the root user on the local host computer. There are several things you can do, however, to tailor LogSentry to suit your particular needs. See the “Configuring LogSentry to suit your needs” section later in this chapter for more information.

Running LogSentry

When you install the logsentry package, a script named logsentry.cron is placed in the /etc/cron.d directory. In this case, the /etc/cron.d/logsentry.cron script simply runs the /usr/sbin/logcheck.sh script.

Because the logsentry.cron script runs hourly, this means that you (or the administrator) will receive 24 e-mail messages each day, each containing the filtered log messages. You can create your own cron script to have the logcheck.sh launched on any schedule you like.

Note 

The /usr/sbin/logcheck.sh script uses the /usr/sbin/logtail command to gather only those log messages that you haven’t already seen. The logtail command does this by creating a .offset file for each log file that Logcheck monitors. The next time Logcheck is run, only log messages that have arrived since the previous run are checked.

Using LogSentry

After LogSentry has been set up and run, to begin using LogSentry you start by simply reading the e-mail that LogSentry sends you. By default, the root user on your Fedora system will receive an e-mail message from LogSentry each hour. Log messages that are matched, and not excluded, are sorted under one of the following three headings in each e-mail message:

• Active System Attack Alerts — Represents messages that may represent an attack on your system.

• Security Violations — Includes failures and violations that may indicate a problem, but not necessarily an attack on your system.

• Unusual System Events — Includes all log messages that are neither matched nor excluded.

The following is an example of a LogSentry e-mail message.

Return-Path: <[e-mail]>  Received: (from root@localhost)  Subject: maple 04/06/04:19.01 ACTIVE SYSTEM ATTACK!  Status: R      Active System Attack Alerts  =-=-=-=-=-=-=-=-=-=-=-=-=-=  Apr 6 18:02:28 maple portsentry[1102]: attackalert: Possible stealth  scan from unknown host to TCP port: 111 (accept failed)  Apr 6 18:33:26 maple sendmail[1863]: f371XJw01863: "wiz" command from  duck.handsonhistory.com [10.0.0.28] (127.0.0.1)  Apr 6 18:33:29 maple sendmail[1863]: f371XJw01863: "debug" command from  duck.handsonhistory.com [10.0.0.28] (127.0.0.1)      Security Violations  =-=-=-=-=-=-=-=-=-=  Apr 6 18:02:28 maple portsentry[1102]: attackalert: Possible stealth  scan from unknown host to TCP port: 111 (accept failed)  Apr 6 18:39:14 maple -- root[1121]: ROOT LOGIN ON tty1      Unusual System Events  =-=-=-=-=-=-=-=-=-=-=  Apr 6 18:01:35 maple last message repeated 291877 times  Apr 6 18:14:10 maple last message repeated 297510 times  Apr 6 18:20:58 maple kernel: SB 4.16 detected OK (220)  Apr 6 18:20:58 maple kernel: SB16: Bad or missing 16 bit DMA channel  Apr 6 18:20:58 maple kernel: sb: 1 Soundblaster PnP card(s) found.  Apr 6 18:38:37 maple syslog: syslogd startup succeeded  Apr 6 18:38:37 maple kernel: klogd 1.4-0, log source = /proc/kmsg  started.  Apr 6 18:38:37 maple kernel: Inspecting /boot/System.map-2.4.2-0.1.49  

In the preceding e-mail, under the Active System Attack Alerts heading, you can see that a scan of port 111 (portmapper service) was detected by the PortSentry service. The next two messages indicate that a user from the host duck.handsonhistory.com tried to scan sendmail to see if debug and wiz services could be accessed. Under the Security Violations heading, the possible stealth scan appeared again. A normal login by the root user was also detected (although it represented no particular threat in this case).

Under the Unusual System Events heading, as noted earlier, are included all messages not matched (specifically added to another category) or excluded (specifically ignored) by any of the filter files. A lot of normal system-activity messages appear here. Over time, you may want to explicitly include or exclude messages that you see all the time or that catch your eye as a potential problem you need to watch. For example, the preceding lines that say last message repeated 291877 times indicate a potential denial-of-service attack. You may want to add the keywords "message repeated" to your logcheck.hacking list. Likewise, the words "Soundblaster PnP card(s) found" could be added to the logcheck.ignore file, because that message reflects normal processing.

In general, you want to react to attacks that you detect by preventing an attacker from gaining access to your system. If an attacker does get in, you want to get that attacker out of your system and clean up the damage as best you can. An excellent tool for detecting, logging, and denying access to your system by attackers is the PortSentry package discussed later in this chapter.

Configuring LogSentry to suit your needs

After LogSentry is installed, it will run without requiring any configuration. However, to better suit your needs, there are several configuration files you can modify. The following section describes those files.

Editing the logcheck script

The /usr/sbin/logcheck.sh script scans your log files and sorts the log messages that are e-mailed. You can change much of the behavior of the logcheck.sh script by changing the values of the variables within the script. To change the behavior of the logcheck.sh script, follow these steps:

1. Make a copy of the /usr/sbin/logcheck.sh file. For example:

    # cp /usr/sbin/logcheck.sh /usr/sbin/logcheck.sh.old  

2. Open the script in any text editor while logged in as the root user and make any changes to the script. The following bullet list describes values that you may want to change.

   • SYSADMIN — This variable defines the root user as the one to receive the e-mail messages resulting from running Logcheck. By modifying the following line, you can change root to anyone you want to receive the LogSentry messages. This can be either local users or users on other computers (that is, user@hostname).

     SYSADMIN=root 

   • TMPDIR — Sets where LogSentry writes its temporary files during processing. The directory is created when LogSentry starts and is removed before Logcheck finishes. You can change the location by modifying the following entry:

     TMPDIR=/tmp/logcheck$$-$RANDOM 

   • GREP — The logcheck.sh script relies on a grep command that supports the -i, -v, and -f options to search the log files. By default, the egrep command is used for this purpose. You could change the value to grep or another command by modifying the following variable:

     GREP=egrep 

   • MAIL — E-mail is sent by LogSentry using the mail command. To have LogSentry use a command other than the mail command to send e-mail messages to the administrator, change the following mail variable:

     MAIL=mail 

   • Filter files — There are four filter files defined by LogSentry. The files each contain keywords that are either used to find messages or exclude messages that contain those keywords. The following variables are set to indicate the location of the four LogSentry filter files:

     HACKING_FILE=/etc/logsentry/logcheck.hacking  VIOLATIONS_FILE=/etc/logsentry/logcheck.violations  VIOLATIONS_IGNORE_FILE=/etc/logsentry/logcheck.violations.ignore  IGNORE_FILE=/etc/logsentry/logcheck.ignore 

For more information on filter files, see the “Changing LogSentry filter files” section, later in this chapter.

• Log files — The last entries in the /usr/sbin/logcheck.sh script that you may want to change designate which log files are monitored by the logcheck.sh script. By default, the script runs the logtail command to check the messages, secure, and maillog files (in /var/log). The following lines define which log files are checked and determine where the output of logtail is temporarily written.

  $LOGTAIL /var/log/messages > $TMPDIR/check.$$  $LOGTAIL /var/log/secure >> $TMPDIR/check.$$  $LOGTAIL /var/log/maillog >> $TMPDIR/check.$$ 

  Tip 

  If you like, you can have more log files checked by adding more lines like the preceding ones. Just make sure that the first line includes a single arrow (to overwrite a previous check file) and that all subsequent lines contain double arrows (to append to the current check file).

Changing LogSentry filter files

The /etc/logsentry directory contains four filter files that define which messages are matched and e-mailed to the administrator. The contents of these files are simply keywords. Log files are searched (or grepped) for these keywords and sorted or discarded based on the results of the search.

Some keyword filter files are intended to uncover words or phrases that would appear in a log message in the event of a system break-in or misuse. Other keyword files are intended to find messages that pose no security threat (so that the messages can be excluded from the e-mailed log messages). Messages that match neither the included or excluded keywords are appended to the Unusual System Events heading of the e-mail summary output.

You can use filter files as they are. However, over time, you may want to modify these files for several reasons. If you are receiving repetitive, non-threatening messages in the LogSentry e-mails, you can add keywords that can filter out those messages. Also, you can add keywords later as you learn about new types of security breaches that you may want to look for.

Besides including alphanumeric characters, keywords can also include wildcard characters. For example, you could use an asterisk (*) to match any string of characters, a question mark (?) to match any single character, or a dollar sign ($) to match a keyword that appears at the end of a line.

Caution 

Use wildcards carefully. A mistaken wildcard character can result in too many or too few messages being included or excluded.

The four LogSentry filter files in the /etc/logsentry directory are:

• logcheck.hacking — Contains keywords that appear in log messages that represent known hacking attacks.

• logcheck.ignore — Contains keywords that represent messages that should always be ignored.

• logcheck.violations — Contains keywords that represent negative activities that may or may not represent real intrusions on your system.

• logcheck.violations.ignore — Contains keywords that represent messages that should be ignored from those found as part of the violations check.

Each of these files is described in the following sections.

Note 

It is important to note that messages that are neither explicitly matched (from logcheck.hacking and logcheck.violations) nor explicitly excluded (from logcheck.ignore and logcheck.violations.ignore) are included in the e-mail sent by Logcheck to the administrator. Those messages are displayed under the catchall heading Unusual System Events.

logcheck.hacking

Keywords from the /etc/logsentry/logcheck.hacking file are meant to uncover log messages representing attacks on your system. Log messages that are matched by keywords in this file are output in e-mail messages to your system administrator under the logcheck.hacking heading.

Messages that appear under this heading are the first messages to appear in the e-mail message. You can add other keywords to this file as you learn about messages that represent different types of attacks on your system.

Following are some examples of keywords that appear in the logcheck.hacking file:

"wiz"  "WIZ"  "debug"  "DEBUG"  ATTACK  nested  VRFY bbs  VRFY decode  VRFY uudecode  rlogind.*: Connection from .* on illegal port  rshd.*: Connection from .* on illegal port  sendmail.*: user .* attempted to run daemon  uucico.*: refused connect from .*  tftpd.*: refused connect from .*  login.*: .*LOGIN FAILURE.* FROM .*root  login.*: .*LOGIN FAILURE.* FROM .*guest  

Most of the messages in this file are used to match log messages that result from someone probing your system with the Internet Security Scanner (ISS). ISS is a tool that can scan a set of IP addresses for potential security weaknesses. Though most of the security holes ISS checks for have been plugged over time, these log messages alert you to the fact that someone is checking the security of your system.

The debug and wiz keywords shown in the previous file will catch attempts by ISS to access wiz and debug services from the sendmail service. Likewise, VRFY is a command that ISS sends to requests of the sendmail service to ask for different user names. Other keyword phrases shown in the previous example match failed attempts to connect using different Linux network services (such as rlogind, rshd, sendmail, uucico, and so on).

logcheck.ignore

Keywords in the /etc/logsentry/logcheck.ignore file are used to find log messages that should be excluded (that is, ignored) by LogSentry and will therefore not appear in e-mail summaries. The keywords in this file reduce the number of log messages that are e-mailed to the administrator, making it easier to find the real problems. The following are some examples of keywords from the logcheck.ignore file.

cron.*CMD  cron.*RELOAD  cron.*STARTUP  ftp-gw.*: exit host  ftp-gw.*: permit host  ftpd.*ANONYMOUS FTP LOGIN  http-gw.*: exit host  http-gw.*: permit host  identd.*Successful lookup  identd.*from:  named.*Response from 

Most of the entries in this file are used to match messages that represent the normal operation of various system services. The keywords shown in the previous example represent normal processing of cron, ftp, http, identd, and named features.

Notice that all of the keyword phrases shown in the preceding example include an asterisk (*) wildcard. If you add your own keywords to this file, using asterisks and other wildcards can help you be specific about the log messages you exclude.

logcheck.violations

There are certain words that imply negative behavior occurring on your computer. Though these words may not be associated with any particular attack, LogSentry notes messages containing these words and displays them under the following heading and e-mails them to the system administrator:

Security Violations  =-=-=-=-=-=-=-=-=-= 

The following is an example of some of the keywords that appear in the logcheck.violations file:

ATTACK  BAD  DEBUG  FAILURE  ILLEGAL  REFUSED  denied  failed  unapproved  attackalert 

Adding your own keywords to the file can help flag log messages that may be of particular concern for your computer. For example, you can add keywords that represent improper use of services that are not standard Linux features but can be accessed from the network.

logcheck.violations.ignore

Use the logcheck.violations.ignore file to exclude messages that were matched from the logcheck.violations file, but are known to not represent security problems. By default, only the following entry is contained in this file:

stat=Deferred 

The previous keyword causes LogSentry to ignore log messages from sendmail that represent e-mail messages that haven’t been sent because the receiving server was temporarily unavailable. As you use LogSentry, you are likely to repeatedly encounter certain log messages that represent no security threat. Add keywords here (specifically as possible) to exclude those messages from appearing continuously in e-mail messages from LogSentry.

Modifying syslog

The syslog service gathers the log messages of system activity that are used by LogSentry. Fedora, as well as most other Linux and UNIX systems, comes with syslog installed and operational by default. You can modify the /etc/syslog.conf file to tailor the behavior of syslog to best suit the way you use your system.

The syslog service is part of the sysklogd software package and is started automatically from the /etc/init.d/syslog start-up script. After that script has been run on your system, two daemon processes should be active on your system: syslogd and klogd. To see if they are running, type the following:

 # ps ax | grep log  

The /etc/syslog.conf file contains information that defines which activities are logged, and from which system. LogSentry monitors three of the log files created by syslog (in the /var/log directory): messages, secure, and maillog. The following lines in the /etc/syslog.conf file instruct syslog to create those log files:

*.info;mail.none;news.none;authpriv.none;cron.none /var/log/messages  authpriv.*                                         /var/log/secure  mail.*                                             /var/log/maillog 

Services on your Fedora system produce messages of different levels. Message levels, from most critical to least critical, are listed in Table 14-4.

Table 14-4: Message Levels

Level	What It Means
debug	Detailed processing information
info	Purely informational
notice	Important, but not an error
warning	Potential error
err	Error condition
crit	Critical
alert	Immediate action needed
emerg	System unusable

The line shown in the example indicates that all messages from the info level (*.info) and above are logged to the /var/log/messages file. However, messages of types mail, news, authpriv, and cron are excluded because they are sent to other log files. All authpriv (authpriv.*) messages are logged to the /var/log/secure file. mail messages (mail.*) are all logged to the /var/log/maillog file.

With this default configuration of syslog, LogSentry should catch all major security-related activities. There are a few situations, however, where you may want to modify the /etc/syslog.conf file. For example, if you are receiving a lot of log messages for a particular type of service (such as ppp if you are having trouble with a dial-up connection), you may consider directing messages for that service to its own log file. Then, if LogSentry uncovers a problem, it’s easier to go through only that log file for those messages relating to the problem service.

Another temporary change you may consider is if you need to debug a problem with your system. Changing *.info to *.debug temporarily can give you more details on a problem. (Make sure to change it back later, or syslog will chew up too much system resources.) After making changes to your syslogd service, run killall -HUP syslogd to have the changes take effect.