10.5. Automated Log Monitoring

 < Day Day Up > 

Monitoring the logs you collect is vital. A variety of programs exist that will monitor log messages as they come in and notify administrators when suspicious or unusual activity is detected. We will examine two such programs available for both FreeBSD and OpenBSD: logcheck and swatch.

Both these programs look for certain patterns in logfiles and then perform one or more given actions. To know what to look for, you will have to spend considerable time looking at system logs and determining what log entries you need to be notified about. It might be helpful to conduct some attacks against your systems to generate key events. Using tools like nessus and nmap and picking up a few exploits from the Security Focus web site (http://www.securityfocus.com/) will help you generate some of the logs you will want to capture. As new vulnerabilities are discovered, run them against your systems. Did they cause your log monitoring scripts to alert you in the way you desired? If not, adjust your configuration.

Regardless of the log monitoring solution you choose, configure the program to not need root access. In general, this requires little more than ensuring your logfiles are readable by some group to which the system user running your log monitoring application is a member.


10.5.1. Automated Auditing Using logcheck

logcheck is based upon a component of the Gauntlet firewall system by Trusted Information Systems, Inc. but later completely written by Craig Rowland of Psionic at the time it was called logsentry. This is all ancient history. logcheck has not seen much development in more than seven years. Nevertheless, it requires very little effort to set up and its functionality continues to be useful. The way logcheck works is best described by its README:

1) By reporting everything you tell it to specifically look for via keywords.

2) By reporting everything you didn't tell it to ignore via keywords.

With this design, logcheck notifies administrators by email when key warning events occur, and when new, unexpected, log entries turn up. logcheck runs periodically through cron and reviews all messages since last execution. If there is anything to report, logcheck sends an email.

10.5.1.1 Installation

logcheck can be installed from ports/security; the name of the port is logcheck on FreeBSD and logsentry on OpenBSD. Both of these ports download the same compressed source tarball. As with other ports, logcheck is installed by running make install from the logcheck or logsentry ports directory. This will install the logtail binary and several sample configuration files and documents. Run pkg_info -L on the logcheck or logsentry package to determine exactly where the files have been installed on your system. Although the logcheck program has not been modified in years, the installation defaults do change from time to time.

10.5.1.2 Configuration

The heart of logcheck is logcheck.sh. This file contains key configuration elements like the address to which notifications should be sent, the location of additional configuration files, and the location of the logfiles to monitor. Each of these topics deserves some discussion as follows:


Email address

The logcheck.sh script extracts "interesting" log entries and sends them to an email address. Given the sensitivity of log information, be sure you have protected the path this mail message will take in some way. How you accomplish this might involve encrypted tunnels, unencrypted traffic over an administrative network, or local delivery.


Location of keyword specification files

There are four files that control how logcheck parses logs and determines what qualifies as interesting. These files are named logcheck.hacking, logcheck.violations, logcheck.violations.ignore, and logcheck.ignore. The former two enumerate keywords that, when found, identify log messages as interesting. The latter two contain longer phrases that more fully describe specific log messages which should be ignored.


Location of logfile(s)

The easiest way to run logcheck is to point it at one single logfile. It's trivial on a loghost system to take all incoming logs and put them in a file for logcheck alone. This file will grow quickly and should therefore be truncated frequently. There is no need to keep compressed archives of this file.

For the sake of convenience and for a little added security, you may want to move the logcheck.sh script and configuration files into a directory under /usr/local/etc accessible only as root. Whether you choose to take this approach or not, the next step is to tailor the additional configuration files for your environment or at the very least review them.


logcheck.hacking

When entries from the logcheck.hacking file are found in log messages, the subject line of the email logcheck sends will say "HOSTNAME DATE ACTIVE SYSTEM ATTACK!" Be careful about what keywords are found in this file.


logcheck.violations

The logcheck.violations file is the second file containing keywords for interesting messages. Again, if you are too general, you will get too many false positives.


logcheck*.ignore

Finally the logcheck.[violations.]ignore and logcheck.ignore files tell logcheck which log entries to ignore. Try to avoid merely using keywords in this file as you might be ignoring more than you bargained for.

Notice that being too general in either the hacking or violations configuration files will result in too many false positives whereas being too general in the ignore files will result in interesting log messages being missed. While logcheck is easy to configure, it's even easier to configure poorly. This phase of logcheck configuration is usually iterative, involving the following five steps.

  1. Set up a single logfile that logcheck will parse. This file should be well protected from all users. Allow this file to accumulate several hours (or even a day's) worth of data.

  2. Edit the hacking, violations, and two ignore files and remove both keywords and ignore lines you feel do not show up in your logfiles.

  3. After you are comfortable with your changes, run logcheck and examine the output.

  4. If you miss log entries you wanted to see or if you have too many log entries mailed to you, revise the hacking, violations, and two ignore files to compensate.

  5. Repeat steps 3 and 4 until the four configuration files are producing output to your satisfaction.

Substrings in all the hacking and violations files are matched using case-insensitive egrep(1). The ignore files specify patterns used by case-sensitive egrep. Bear this in mind when working with these configuration files. The manpage for grep(1) is a worthwhile read if you are not familiar with regular expressions.


Your goal in configuring logcheck this way is to be able to run logcheck very frequently but receive very little email. The more false positives you receive, the less likely you are to pay attention to the results produced by logcheck. Configure carefully, and configure well. As systems and applications are deployed in your network, revisit and revise your logging and logcheck configuration.

10.5.1.3 Drawbacks

logcheck is not perfect. It suffers from lack of active development, which means that you will need to move on to another piece of log monitoring software if you need additional functionality. While logcheck does a very good job monitoring logfiles, it can be inconvenient to deal with an additional logfile containing all messages. Moreover, if you choose to monitor logfiles from applications that cannot log via Syslog, you will need to make duplicates of these logs or inject them into Syslog using the -f flag of the logger command. This can quickly become an administrative headache. Fortunately, the log monitoring world is full of options. Next, we explore another popular log monitoring utility, swatch.

10.5.2. Automated Auditing Using swatch

swatch , the simple watcher, was originally designed to monitor logfiles written by syslogd but can now monitor arbitrary logfiles and is quite flexible. swatch expects a single configuration file that contains actions to perform when specified patterns have been found. swatch is almost as easy to install and configure as logcheck, but is more flexible and can perform a variety of actions besides sending email.

10.5.2.1 Installation

In OpenBSD and FreeBSD, the installation procedure for swatch on a loghost system is identical:

% cd /usr/ports/security/swatch % make && sudo make install

On both platforms, this will install two binaries (swatch and swatch_oldrc2newrc) and the manpages for both. While example configuration files are included in the compressed swatch tarball, they are not installed during the install process. In OpenBSD the two example files are located in the w-swatch-version/swatch-version/examples/ subdirectory of the swatch ports directory; in FreeBSD, check in the work/swatch-version/examples/ subdirectory. Additional sample configuration files for swatch may be found at http://www.loganalysis.org.

10.5.2.2 Configuration

You can start using swatch with very little configuration, and getting things operational is trivial. As with logcheck however, fine-tuning swatch to work well in your environment will take some time. Before we get carried away, let's explore swatch's basic functionality.

swatch reads configuration from ${HOME}/.swatchrc by default. You can, and usually should, change this behavior, by providing the -c argument and pointing to a configuration file you create, perhaps in /usr/local/etc/. A contrived configuration file is provided in Example 10-7. As mentioned previously, swatch expects a series of patterns and actions in its configuration file. Patterns are provided as arguments to the watchfor and ignore statements, which do exactly what you think they do.

Example 10-7. Simple swatch configuration
ignore /refused connection from 10.1/ ignore /BAD SU john to root/ watchfor /inetd.* refused connection/   echo normal   throttle 01:00:00   mail addresses=root,subject=Refused\ Connection,when=2-6,9-5 watchfor /file system full/   echo bold   throttle 00:05:00   mail addresses=root,subject=FILE\ SYSTEM\ FULL,when=1-7,1-24 watchfor /BAD SU.*to root/   echo bold   throttle 00:01:00

ignore statements are best placed early in the configuration file to weed out logs that don't need to be parsed. This statement is followed by a regular expression, enclosed in slashes, which will cause log messages to be ignored if the pattern expressed matches the log message.

watchfor statements are also followed by a regular expression. When the expression matches a message, various actions are performed.

If you are unfamiliar with regular expressions, do not fear. A simple series of words is often enough for ignore and watchfor statements. If you would like more flexilibility, see the manpage for grep(1).


As the actions are fully documented in the manpage for swatch(1), we will not go into great detail here. However, the actions shown in Example 10-7 are the most popular and deserve some treatment as follows:


echo modes

The echo action will result in the matched line being sent to standard output. A mode may be specified to change the color of the output, make it bold, and so on.


bell N

When bell is specified as an action, a bell character will be sent to standard output.


throttle

throttle tells swatch to limit notifications. When a throttle period of one hour is specified (as in the first watchfor statement in Example 10-7), a specific log message matched by this pattern will not be displayed until the hour is up. By default, throttle is done on a per-message basis so that other messages that also match the pattern, have their own throttle period. To make the throttle period apply to all messages that match the pattern, append ,use=regex at the end of your throttle period specification. At the end of the throttle period, a summary is displayed listing the number of times the specific message was seen.


mail

The mail action will, of course, send an email to the specified recipient(s) when messages are matched.


when

The when option can be applied to any of swatch's regular actions. This option restricts the action to a specific time period governed by days and hours. The first day of the week is Sunday and is expressed as the number 1. Hours must be specified in a 24-hour format.

At this point, you should have enough information to configure swatch. All that remains is that you know what to look for. For that, you'll need to study your logs, not this chapter.

10.5.2.3 Running swatch

After you have created a configuration file for swatch, you need to start it. swatch may be run on the command line or as a daemon by using the --daemon command-line argument. It might also be helpful to write a startup script that starts swatch as a daemon and redirects standard output to the console or a separate logfile. Be aware that swatch will need to be restarted whenever the logfile it reads is rotated. To do this, specify a restart time using the --restart-time command-line argument, which ensures the daemon restarts as soon after the log is rotated as possible.

10.5.2.4 Catching new messages

swatch is not designed to alert you when new messages show up in your logs. Whereas logcheck sends you alerts unless you tell it not to, swatch will only send you alerts when you tell it to. Fortunately, due to swatch's flexibility, it's possible to configure it very much like logcheck using a series of ignore lines and watchfor /.*/. This expression will match all entries that have not been already ignored or caught by other watchfor statements.

10.5.3. Ongoing Monitoring

Once you have configured logcheck or swatch for your environment, you might be tempted to never evaluate your log monitoring capability again. If you used logcheck or have configured swatch to alert you of new messages, you will still have to return to log monitoring to cut down on unimportant messages that did not exist when you initially configured the log monitoring system.

Despite the fact that your logs are automatically monitored and you have little motivation to return to manually scanning them, you should nevertheless make a point to do so. A periodic manual scan or tail of logs in the background can help you catch activity that log scanners will not. You may notice certain entries appear that you would actually like to know about, yet somehow you have managed to configure your log scanner to ignore them. Watching logs scroll by periodically may also alert you to changes in the frequency with which certain log entries appear a minor warning may not be a problem now and again, but several thousand warnings an hour might be worth examining. The point is, don't underestimate the value of periodic human scans of your logfiles.

Finally, make yourself revisit your log monitoring configuration. Continue running vulnerability test kits or sample exploits to help validate the configuration of your log monitoring solution.

     < Day Day Up > 


    Mastering FreeBSD and OpenBSD Security
    Practical Guide to Software Quality Management (Artech House Computing Library)
    ISBN: 596006268
    EAN: 2147483647
    Year: 2003
    Pages: 142
    Authors: John W. Horch

    flylib.com © 2008-2017.
    If you may any questions please contact us: flylib@qtcs.net