16.3 Using Logcheck to Check Log Files You Never CheckThe Logcheck program analyzes a system's often massive log files, extracts anything that might indicate (1) an attack, (2) a security violation, or (3) other abnormality (listed in order of severity), generates a report summarizing these events, and informs the SysAdmin via e-mail. For those that do not check their log files on a daily or hourly basis (which is almost everyone), it does this task well. It is invoked from each system's root crontab, typically either hourly or daily. It is important to recognize that this is an Intrusion Detection System (IDS) that does not operate in real time.[1] Even if it is invoked hourly, it still gives a cracker an average of 30 minutes to do damage under the best circumstances and an entire weekend under the worst circumstance.
In its standard form from Psionic, it provides a "general feel" for what type of activity your static firewall has been triggered by, and it also shows what Portsentry[2] has triggered on and what sites it has blocked as a result. It also will show what failed SSH attempts have occurred, what mail relaying attempts occurred, etc. If a breach has occurred, it may enable you to discover this through the logging of failed attempts by a cracker to elevate her privileges. It makes clever use of pattern matching to recognize ominous errors that have attack, violation, and similar words that indicatespecific errors for major subsystems. Logcheck has a list of phrases to ignore so you will not be worried about the clones attacking because you already know about that.
While Psionic renamed Logcheck to LogSentry on their Web site, including the gzipped tarball, the name of the script that you run from the root crontab remains logcheck and otherwise is unchanged from earlier versions. In fact, the LogSentry tarball now offered by Psionic is identical to the previous one that was offered on the companion CD-ROM of the first edition of this book down to the last byte. While using Logcheck daily for monitoring client networks, each with dozens to hundreds of systems, and even for the systems on my own network, I have fixed a number of flaws:
I have corrected these problems in Logcheck and included my enhanced version on the CD-ROM. Alternatively, if you e-mail a request for it to book@verysecurelinux.com, I will provide you with a copy of it. Psionic's original may be obtained by starting at www.psionic.com/products/logsentry.html and following their instructions to do this and "sign" that and try to make some builds. The resulting tarball also may be obtained from the CD-ROM under the original name of logcheck-1.1.1.tar.gz. My enhanced version is under the book subdirectory as logcheck-1.1.1bob.tar.gz. Some typical output follows, with the first item being the date and time in the format MM/DD-HH:MM:SS. Active System Attack Alerts (dada) =-=-=-=-=-=-=-=-=-=-=-=-=-= 04/16-16:39:46 portsentry: ATTACK: riddle.com/212.223.7.6 TCP 80 Security Violations (dada) =-=-=-=-=-=-=-=-=-= 05/15-19:42:44 C in DENY e0 T 151.27.1.37:1 126.193.251.197:80 S #21 05/20-03:93:15 T I= O=e0 126.193.251.187 126.193.251.197 DF ICMP=8:0 05/21-00:21:51 T D I=e1 O=e0 230.83.97.91:1944 76.121.32.45:6346 DF TCP The Active System Attack was generated when PortSentry detected the riddle.com system scanning our system for a Web server (they probably did this to topple IIS). The fact that we are not running a Web server on the firewall system caused PortSentry to consider this an attack and locked riddle.com out. (Specifically, PortSentry determined this by seeing a packet sent to a service port where no service was listening.) The line beginning with C is what IP Chains lines look like. This packet was on the input chain. The e0 indicates that the packet came in on eth0, and the T indicates that the protocol is TCP. The source and destination IPs and ports are next, followed by S to indicate that the SYN bit was set (TCP protocol only). Lastly, the #21 indicates that this is rule number 21 in the input chain. The second violation, starting with T, indicates an IP Tables log entry. (Normally you will not see both IP Tables and IP Chains on the same system as only the IP Tables module or the IP Chains module may be running at any particular time and most people use one or the other.) The I= would be the input interface if applicable. The O=e0 is short for OUT=eth0. Next are the source and destination IPs. The DF was set by the Division of Mysteries.[3] Lastly, the ICMP=8:0 indicates that this is an ICMP message of type 8 (ping) and subtype 0 (there are no other ping subtypes).
The last violation is another IP Tables entry line but was generated from rules generated by SuSE's firewall2 script, available on recent SuSE releases. The initial D (for DEFAULT) is my shortened form of SuSE's verbose SuSE-FW-DROP-DEFAULT.[4] I know that it is an SuSE system, that this is a firewall message, and that probably it would not be logged if it was not being dropped. Therefore, I use T (for IP Tables) to indicate all of this. Realistically, one learns to read these very quickly and these shortened forms can be read much faster than muddling through the wrapped-line raw output from IP Tables or IP Chains. This is the opinion of my Beta testers as well.
Let us now walk through an installation. First, ensure that syslog is running; it should be running on any Linux system or BSD-like UNIX system. The log files probably will be under /var/log, with the messages file getting a copy of all messages except debugging and mail logs. It is recommended that the permissions of all log files be changed to mode 600 and be owned by root to stop crackers from using ordinary accounts to get system information from them. Next, mount the CD-ROM as /mnt/cdrom and issue the commands zcat /mnt/cdrom/book/logcheck-1.1.1bob.tar.gz | tar -xovf - cd logcheck-1.1.1bob Now, read the text files LICENSE, README*, CREDITS, INSTALL, and crontab. For those who use Sendmail, edit /etc/mail/aliases and create an alias of alert for the e-mail of whoever should be notified of security problems. Preferably, this should be a separate mailbox from your regular mailbox to avoid cross-contamination. Then, create an alias, alertpage, for e-mail-capable pagers to receive security violation and attack e-mail. If no pager e-mail is desired, use alertpage: /dev/null Now update the aliases database with newaliases For those using a different mail delivery program, such as Postfix, qmail, or smail, either update its aliases file or resort to editing the systems/linux/logcheck.sh file. While the original logcheck.sh sent e-mail to "root," I find the use of the alert and alertpage aliases more configurable. This is both because there is no rootpage account and because the person to receive Logcheck e-mail may be different than the person who receives the occasional error message from root. Psionic intended each SysAdmin to edit the logcheck.sh script itself, but I don't like customizing scripts for each system as it makes installing an updated version of the script on each of many systems an inefficient ordeal. Now, su to root and install the logcheck.sh script in /usr/local/etc and compile and install the logtail program with the command make linux Before going to "production," test Logcheck by invoking it directly with the invocation /usr/local/etc/logcheck.sh If the system has been running for a long time with no trimming of the log files then it could take 10 60 seconds to run the first time. Logcheck tracks how large each log file is when it is being processed, remembers the next time, and does the equivalent of a tail command using its logtail program. The implementation is very efficient: if 10 lines of new text have been added to a 10MB log file then it does a seek to the start of those 10 new lines rather than reading the entire 10MB. Invoke Logcheck a second time to convince yourself of this. Then check your e-mail and pager and ensure that it worked. If it complained in each e-mail about the same missing log file that does not exist in your distribution of Linux, simply create the zero-length file with the touch command and chmod it to mode 600. If you have been running IP Tables or IP Chains without Logcheck or other monitoring, you probably will be surprised at the large number of dropped packets. If you have Windows systems and DHCP on your network then there will be a lot of chatter generated that will not be of interest. This can be dealt with by editing your firewall script and taking logging off of the rules that drop these packets. It could take some time to tune the rules to perfection to avoid all of this chatter, but this will save time over the long run. You likely will discover that you have been the subject of numerous attacks that did not work (if they had worked, you already would know). You may discover that your ISP is wasting your bandwidth with all sorts of pings, idents, and similar requests to your firewall or systems.[5]
Finally, from the logcheck-1.1.1bob directory add Logcheck's invocation to root's crontab entry with the following commands crontab -l > foo2 tail -1 crontab >> foo2 Edit foo2's last line if you are not paid enough to be paged every hour in the middle of the night and all weekend. The unedited last line will look like 00 * * * * /usr/local/etc/logcheck.sh The second space-separated field is a list of what hours the command should be run. 9-17 works for anyone who can leave this behind at 5 P.M.. For an additional check at 8 P.M., use 9-17,20. The fifth field specifies the days on which the script should be run, with 1-5 indicating during the week. For those who want additional checks on the weekends, duplicate the line and edit each one appropriately. The resulting Logcheck lines output by crontab -l then might look like 00 9-17,20 * * 1-5 /usr/local/etc/logcheck.sh 00 10,14,18,22 * * 6,0 /usr/local/etc/logcheck.sh |
Top |