Section 16.3 Using Logcheck to Check Log Files You Never Check

   


16.3 Using Logcheck to Check Log Files You Never Check

graphics/fourdangerlevel.gif

The Logcheck program analyzes a system's often massive log files, extracts anything that might indicate (1) an attack, (2) a security violation, or (3) other abnormality (listed in order of severity), generates a report summarizing these events, and informs the SysAdmin via e-mail. For those that do not check their log files on a daily or hourly basis (which is almost everyone), it does this task well. It is invoked from each system's root crontab, typically either hourly or daily. It is important to recognize that this is an Intrusion Detection System (IDS) that does not operate in real time.[1] Even if it is invoked hourly, it still gives a cracker an average of 30 minutes to do damage under the best circumstances and an entire weekend under the worst circumstance.

[1] Snort is recommended as a full feature real-time IDS. It is discussed briefly in "Firewall Vulnerabilities" on page 361and in "The Snort Attack Detector" on page 598. Snort should be used in conjunction with Logcheck, not instead of it.

In its standard form from Psionic, it provides a "general feel" for what type of activity your static firewall has been triggered by, and it also shows what Portsentry[2] has triggered on and what sites it has blocked as a result. It also will show what failed SSH attempts have occurred, what mail relaying attempts occurred, etc. If a breach has occurred, it may enable you to discover this through the logging of failed attempts by a cracker to elevate her privileges. It makes clever use of pattern matching to recognize ominous errors that have attack, violation, and similar words that indicatespecific errors for major subsystems. Logcheck has a list of phrases to ignore so you will not be worried about the clones attacking because you already know about that.

[2] PortSentry is Psionic's excellent Adaptive Firewall capability. It is discussed in "Using PortSentry to Lock Out Hackers" on page 613. I find its principal advantages over my Cracker Trap is that it will detect stealth scans and that it can take less effort to configure. (Stealth TCP scans are when the Cracker sends only the first of the two packets a client needs to establish a TCP connection or where defective packets are sent.) The Cracker Trap has better notification. Frequently, I will install both on a firewall to get the best coverage and for an extra ring of security.

While Psionic renamed Logcheck to LogSentry on their Web site, including the gzipped tarball, the name of the script that you run from the root crontab remains logcheck and otherwise is unchanged from earlier versions. In fact, the LogSentry tarball now offered by Psionic is identical to the previous one that was offered on the companion CD-ROM of the first edition of this book down to the last byte.

While using Logcheck daily for monitoring client networks, each with dozens to hundreds of systems, and even for the systems on my own network, I have fixed a number of flaws:

  1. Even on fairly inactive systems, anyone can expect e-mail almost hourly. Most e-mails will contain only information in the least important of Logcheck's three levels of severity. However, ignore the e-mail and you risk ignoring a major attack. It would be nice if this was indicated in the initial portion of the subject line so it could be seen easily from a mail browser's summary screen without having to open the message.

  2. It would be nice to have the option of directing Logcheck to notify you by pager e-mail of security violations and attacks, but you still need to dump "unusual activity" into an ordinary e-mail box (separate from your usual mailbox) to check on occasion. Unfortunately, Logcheck cannot send an e-mail message to a different address, depending on the highest severity of the errors encountered.

  3. Each entry of a given severity also appears in each portion of the report that carries less severe but possibly important warnings. Consequently, I found myself ignoring the Active System Attack Alerts section of the e-mail because these items also are in the Security Violations section of the e-mail. Similarly, I rarely continued to the Unusual System Events because mostly this section repeated the security violations.

  4. Almost every entry is substantially longer than 80 characters, causing lines to wrap around. This makes it very hard to scan the e-mail rapidly for things that concern me, such as attacks to particular IP addresses or ports.

  5. Each line contains data that usually is not important, such as the system name and kernel: Packet log: when it was obvious by the context that this was an IP Chains entry. Also, various TCP flags of limited interest are reproduced. Logcheck makes no effort to "boil down" common errors such as packet dropping. In this case, usually your interest would be in the source and destination systems' IP addresses, ports, and protocol used. There always is the option to grep in the original log file for the additional fields in the rare case when one is interested.

  6. The SysAdmin is expected to remember protocol numbers, such as 6 for TCP, 17 for UDP, and 2 for DDP. While I know these after analyzing network traffic for over a decade, it still takes longer to interpret the raw numbers.

  7. Logcheck includes PortSentry's log of what command it used to lock out each attacker. I know what I told PortSentry to do, so I do not want to see this same long line dozens of time a day.

  8. Logcheck does not allow altering the From: line to indicate which computer it came from. If Pentacorp has a firewall in each of, say, eight offices, an e-mail address of root@pentacorp.com is not informative, especially when other e-mail comes from root, too.

I have corrected these problems in Logcheck and included my enhanced version on the CD-ROM. Alternatively, if you e-mail a request for it to book@verysecurelinux.com, I will provide you with a copy of it. Psionic's original may be obtained by starting at

www.psionic.com/products/logsentry.html

and following their instructions to do this and "sign" that and try to make some builds. The resulting tarball also may be obtained from the CD-ROM under the original name of logcheck-1.1.1.tar.gz. My enhanced version is under the book subdirectory as logcheck-1.1.1bob.tar.gz.

Some typical output follows, with the first item being the date and time in the format MM/DD-HH:MM:SS.

 
 Active System Attack Alerts (dada) =-=-=-=-=-=-=-=-=-=-=-=-=-= 04/16-16:39:46 portsentry: ATTACK: riddle.com/212.223.7.6 TCP 80 Security Violations (dada) =-=-=-=-=-=-=-=-=-= 05/15-19:42:44 C in  DENY e0 T 151.27.1.37:1 126.193.251.197:80 S #21 05/20-03:93:15 T I= O=e0 126.193.251.187 126.193.251.197 DF ICMP=8:0 05/21-00:21:51 T D I=e1 O=e0 230.83.97.91:1944 76.121.32.45:6346 DF TCP 

The Active System Attack was generated when PortSentry detected the riddle.com system scanning our system for a Web server (they probably did this to topple IIS). The fact that we are not running a Web server on the firewall system caused PortSentry to consider this an attack and locked riddle.com out. (Specifically, PortSentry determined this by seeing a packet sent to a service port where no service was listening.) The line beginning with C is what IP Chains lines look like. This packet was on the input chain. The e0 indicates that the packet came in on eth0, and the T indicates that the protocol is TCP. The source and destination IPs and ports are next, followed by S to indicate that the SYN bit was set (TCP protocol only). Lastly, the #21 indicates that this is rule number 21 in the input chain.

The second violation, starting with T, indicates an IP Tables log entry. (Normally you will not see both IP Tables and IP Chains on the same system as only the IP Tables module or the IP Chains module may be running at any particular time and most people use one or the other.) The I= would be the input interface if applicable. The O=e0 is short for OUT=eth0. Next are the source and destination IPs. The DF was set by the Division of Mysteries.[3] Lastly, the ICMP=8:0 indicates that this is an ICMP message of type 8 (ping) and subtype 0 (there are no other ping subtypes).

[3] DF means that the Don't Fragment flag was set. It took my using grep on the kernel source tree to figure this out as it is not documented.

The last violation is another IP Tables entry line but was generated from rules generated by SuSE's firewall2 script, available on recent SuSE releases. The initial D (for DEFAULT) is my shortened form of SuSE's verbose SuSE-FW-DROP-DEFAULT.[4] I know that it is an SuSE system, that this is a firewall message, and that probably it would not be logged if it was not being dropped. Therefore, I use T (for IP Tables) to indicate all of this. Realistically, one learns to read these very quickly and these shortened forms can be read much faster than muddling through the wrapped-line raw output from IP Tables or IP Chains. This is the opinion of my Beta testers as well.

[4] Under IP Tables, you can specify that certain text be prepended to the log file message that is generated. SuSE's firewall2 script, standard on its recent releases, prepends SuSE-FW-stuff, which makes for long lines with repetitive text. This also violates one of the most important rules of programming I learned in school. That is, the varying part of a space-separate word of an error message should be at the beginning of the word (and at the beginning of the line) or it likely will not be noticed.

There is a good lesson here for anyone writing error messages for human consumption, including log messages (applicable to firewall2, PortSentry, etc.):

work very hard not to include obvious, repetitive, or rarely important information in error messages. This unnecessary information serves mostly to make understanding the error message more difficult. This puts the intended reader of the message at risk for not understanding its critical information. Also, processed log messages should not exceed 80 characters, to avoid line wrapping. By following this lesson, the person reading the messages can skim columns of interest far more quickly and not spend time reading unnecessary words.

Let us now walk through an installation. First, ensure that syslog is running; it should be running on any Linux system or BSD-like UNIX system. The log files probably will be under /var/log, with the messages file getting a copy of all messages except debugging and mail logs. It is recommended that the permissions of all log files be changed to mode 600 and be owned by root to stop crackers from using ordinary accounts to get system information from them. Next, mount the CD-ROM as /mnt/cdrom and issue the commands

 
 zcat /mnt/cdrom/book/logcheck-1.1.1bob.tar.gz | tar -xovf - cd logcheck-1.1.1bob 

Now, read the text files LICENSE, README*, CREDITS, INSTALL, and crontab.

For those who use Sendmail, edit /etc/mail/aliases and create an alias of alert for the e-mail of whoever should be notified of security problems. Preferably, this should be a separate mailbox from your regular mailbox to avoid cross-contamination. Then, create an alias, alertpage, for e-mail-capable pagers to receive security violation and attack e-mail. If no pager e-mail is desired, use

 
 alertpage: /dev/null 

Now update the aliases database with

 
 newaliases 

For those using a different mail delivery program, such as Postfix, qmail, or smail, either update its aliases file or resort to editing the systems/linux/logcheck.sh file.

While the original logcheck.sh sent e-mail to "root," I find the use of the alert and alertpage aliases more configurable. This is both because there is no rootpage account and because the person to receive Logcheck e-mail may be different than the person who receives the occasional error message from root. Psionic intended each SysAdmin to edit the logcheck.sh script itself, but I don't like customizing scripts for each system as it makes installing an updated version of the script on each of many systems an inefficient ordeal.

Now, su to root and install the logcheck.sh script in /usr/local/etc and compile and install the logtail program with the command

 
 make linux 

Before going to "production," test Logcheck by invoking it directly with the invocation

 
 /usr/local/etc/logcheck.sh 

If the system has been running for a long time with no trimming of the log files then it could take 10 60 seconds to run the first time. Logcheck tracks how large each log file is when it is being processed, remembers the next time, and does the equivalent of a tail command using its logtail program. The implementation is very efficient: if 10 lines of new text have been added to a 10MB log file then it does a seek to the start of those 10 new lines rather than reading the entire 10MB. Invoke Logcheck a second time to convince yourself of this. Then check your e-mail and pager and ensure that it worked. If it complained in each e-mail about the same missing log file that does not exist in your distribution of Linux, simply create the zero-length file with the touch command and chmod it to mode 600.

If you have been running IP Tables or IP Chains without Logcheck or other monitoring, you probably will be surprised at the large number of dropped packets. If you have Windows systems and DHCP on your network then there will be a lot of chatter generated that will not be of interest. This can be dealt with by editing your firewall script and taking logging off of the rules that drop these packets. It could take some time to tune the rules to perfection to avoid all of this chatter, but this will save time over the long run. You likely will discover that you have been the subject of numerous attacks that did not work (if they had worked, you already would know). You may discover that your ISP is wasting your bandwidth with all sorts of pings, idents, and similar requests to your firewall or systems.[5]

[5] In configuring client systems around the world, I have noticed about half of the ISPs unintentionally waste their clients' bandwidth and the time of those involved in configuring firewalls with such things as undesired routing information protocol (RIP) packets (on UPD port 520) and Ident (auth) requests (on TCP port 113) in response to DNS or SMTP requests. These can and should be dropped or rejected.

The UDP requests certainly should not be trusted, as UDP packets are spoofable and most ISPs do not use the simple antispoofing techniques discussed in this book. The Ident requests most likely are due to someone at the ISP not thinking about bandwidth issues during configuration.

Finally, from the logcheck-1.1.1bob directory add Logcheck's invocation to root's crontab entry with the following commands

 
 crontab -l > foo2 tail -1 crontab >> foo2 

Edit foo2's last line if you are not paid enough to be paged every hour in the middle of the night and all weekend. The unedited last line will look like

 
 00 * * * * /usr/local/etc/logcheck.sh 

The second space-separated field is a list of what hours the command should be run. 9-17 works for anyone who can leave this behind at 5 P.M.. For an additional check at 8 P.M., use 9-17,20. The fifth field specifies the days on which the script should be run, with 1-5 indicating during the week. For those who want additional checks on the weekends, duplicate the line and edit each one appropriately. The resulting Logcheck lines output by crontab -l then might look like

 
 00 9-17,20     * * 1-5 /usr/local/etc/logcheck.sh 00 10,14,18,22 * * 6,0 /usr/local/etc/logcheck.sh 

   
Top


Real World Linux Security Prentice Hall Ptr Open Source Technology Series
Real World Linux Security Prentice Hall Ptr Open Source Technology Series
ISBN: N/A
EAN: N/A
Year: 2002
Pages: 260

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net