Chapter 13

Section: Part IV:  The Defender's Toolkit

Chapter 13. Logging and Auditing Tools

IN THIS CHAPTER

        Why Log?

        Logs from a Cracking Perspective

        Forming a Logging Strategy

        Network Monitoring and Data Collection

        Tools for Analyzing Log Files

        Specialized Logging Utilities

This chapter explains why logs are important, how to create a logging strategy, and how to avoid some of the common pitfalls. We will introduce the tools and techniques that will help you get the most from your logs without losing your mind.

 

Section: Chapter 13.  Logging and Auditing Tools

Why Log?

Logs are another set of double-edged swords that quietly lie behind the scenes. They can completely save your butt, or completely overwhelm you, depending on the situation. Their importance, however, is frequently underestimated.

Logs are useful for a number of things. They can help you troubleshoot problems. They can be used for tracking down network anomalies. They can help trace an intruder's steps, or help solidify your case in a court of law. However, if you don't have a logging strategy, rest assured you will eventually come to regret it.

 

Section: Chapter 13.  Logging and Auditing Tools

Logs from a Cracking Perspective

If your operating system already supports logging, you might be tempted to skip using additional logging tools. Try to resist that temptation. You can't always trust your logs. In fact, altering logs to cover one's tracks is one of the first things crackers learn. The practice has become so common that there are tools that automate the process. Here are a few:

        UTClean. UTClean is a utility that erases any evidence of your presence in wtmp, wtmpx, utmp, utmpx, and lastlog. Check out UTClean at http://packetstorm.securify.com/Exploit_Code_Archive/utclean.c.

        remove. remove will clean utmp, wtmp, and lastlog, erasing any evidence of your presence. Check out remove at http://packetstorm.securify.com/UNIX/penetration/log-wipers/remove.c.

        marry. marry is a tool for editing utmp, wtmp, and lastlog entries. Check out marry at http://packetstorm.securify.com/UNIX/penetration/log-wipers/marry.c.

Note

wtmp, wtmpx, utmpx, and lastlog record and report user information, including what time this or that user accessed the system. For example, grepping for a last entry on root will produce output like this:

root  console  Fri Jun 19 17:01 - down  (00:01)

root  console  Fri Jun 12 12:26 - down  (4+02:16)

root  console  Tue May 19 10:45 - down  (01:50)

root  console  Fri May  1 11:23 - down  (00:02)

root  console  Fri Apr 24 09:56 - 09:56  (00:00)

root  console  Mon Mar 23 02:53 - down  (00:01)

root  console  Mon Mar 23 02:43 - down  (00:01)

When an intrusion occurs, system administrators turn to these logs to determine who accessed the machine and when.

 

It should also be noted that "rootkits," packages designed to cover an intruder's tracks and provide back doors into the system, usually contain log cleansers as well. One of the largest collections of rootkits I've seen can be found at http://packetstorm.securify.com/UNIX/penetration/rootkits/.

 

Section: Chapter 13.  Logging and Auditing Tools

Forming a Logging Strategy

To hedge your bets against crackers tampering with your log entries you should create a logging strategy that is difficult to circumvent. The easiest way to achieve this is to write your logs to a one-way write-once device, or to copy your logs to a secured logging server. Some administrators have their UNIX machines write their logs to a serial port that is attached to a standalone machine. Although this is certainly quite secure, the model doesn't scale very well.

One model that is a little more scalable revolves around using the syslog protocol. Syslog is a native service on almost every UNIX platform, and recently add-on products have made it available on other platforms (such as Windows NT) as well. Although there are some more secure alternatives to syslog, syslog is now common across most router and firewall products. This ubiquity gives administrators a common denominator in which to centralize all logging. For example, administrators could configure all hosts to log to a protected and centralized syslog-based logging server giving security teams a single point in which to coordinate log data (see Figure 13.1).

Figure 13.1. Centralizing logging.

When configured properly, the only traffic allowed to the syslog server is traffic destined to UDP port 514 (the syslog port). By sending system logs to a separate, secure machine, you make it a LOT more difficult for intruders to clean their tracks.

Adiscon makes a great Windows NT-based utility called Event Reporter that enables you to send the Windows NT event logs to a syslog-based server. See http://www.eventreporter.com for more information.

Last year a program called SRS (Secure Remote Streaming) came onto the scene. SRS was written to replace syslog with security at the core of its design. It's not as frequently adopted as syslog, but it is certainly worth checking out as a more secure alternative: http://www.w00w00.org/files/SRS/ .

In addition to centralizing your logs, you might want to consider using at least one third-party logging or parsing tool. This approach has several advantages. First, although the cracker community is familiar with operating system-based logs, few crackers have the knowledge or the means to circumvent third-party logging software. Second, good third-party software packages derive their logs independently of the operating system logs. You'll know that intruders have penetrated your system when this information is compared, and there's a discrepancy between your third-party logs and your regular logs.

This is especially true if you insulate your third-party logs. For example, suppose you use a third-party logging tool to later verify the integrity of operating system-based logs. Although expensive, writing those third-party logs to write-once media guarantees you one set of reliable logs, and reliability is everything.

Note

Developers have been working on methods of preventing crackers from altering logs. For example, 4.4BSD introduced secure levels, a system by which the kernel and system files are protected from modification by intruders. (These secure levels can be set so that even root can't alter the data.) However, in June 1998, the secure levels scheme was cracked. (The problem is by no means confined to UNIX, either. Windows NT server logs can be corrupted and flooded with errors when attacked by a utility called coke.)

 

Using third-party products is prudent in case your out-of-the-box logging utilities fail. For example, on some versions of Solaris, the tmpx file will truncate incoming hostnames, rendering any data obtained via last erroneous and incomplete.

Coming at this from a different angle, it's now a pretty common procedure for crackers to kill your logging capabilities prior to launching a real attack. If the target is running an unpatched version of Solaris 2.5.x, for instance, you can kill syslogd simply by sending it an external message from a nonexistent IP address. Similarly, if syslogd accepts remote messages, anyone can make a false entry in the log.

You should consider an alternative logging system for all these reasons. The next section briefly covers several good ones.

 

Section: Chapter 13.  Logging and Auditing Tools

Network Monitoring and Data Collection

The following tools not only report data from logs, they also collect data from diverse sources. Note that some of these tools are starting to tread pretty close to the Intrusion Detection space, which we covered in detail in Chapter 12, "Intrusion Detection Systems (IDS)." It will be interesting to see whether the two types of utilities will interoperate over time.

SWATCH (The System Watcher)

The authors wrote SWATCH to supplement logging capabilities of out-of-the-box UNIX systems. SWATCH, consequently, has logging capabilities that far exceed your run-of-the-mill syslog. SWATCH provides real-time monitoring, logging, and reporting. Because SWATCH is written in Perl, it's both portable and extensible.

SWATCH has several unique features:

        A "backfinger" utility that attempts to grab finger information from an attacking host.

        Support for instant paging (so you can receive up-to-the-minute reports).

        Conditional execution of commands. (If this condition is found in a log file, do this.)

Lastly, SWATCH relies on local configuration files. Conveniently, multiple configuration files can exist on the same machine. Therefore, although originally intended only for system administrators, any local user with adequate privileges can use SWATCH.

Author: Stephen E. Hansen and E. Todd Atkins

Platform: UNIX (Perl is required)

URL: http://packetstorm.securify.com/UNIX/IDS/swatch-3.0b4.tar.gz

Watcher

Kenneth Ingham developed Watcher while at the University of New Mexico Computing Center. He explains that the Computing Center was being expanded at the time. As a result, the logging process they were then using was no longer adequate. Ingham was looking for a way to automate log scanning. Watcher was the result of his labors.

Watcher analyzes various logs and processes, looking for radically abnormal activity. (The author sufficiently fine-tuned this process so that Watcher can interpret the widely variable output of commands such as ps without setting off alarms.)

Watcher runs on UNIX systems and requires a C compiler.

Kenneth Ingham

Kenneth Ingham Consulting

1601 Rita Dr. NE

Albuquerque,NM 87106-1127

Phone: [phone]

Email: [e-mail]

URL: http://www.i-pi.com/watcher.html

lsof (List Open Files)

lsof version 4 traces not simply open files (including network connections, pipes, streams, and so on), but the processes that own them. lsof runs on many UNIX systems, including but not limited to the following:

        AIX

        BSDI BSD/OS

        NetBSD 1.[23] for Intel and SPARC-based systems

        FreeBSD

        Digital UNIX (DEC OSF/1)

        HP-UX

        IRIX

        Linux

        NEXTSTEP 3.1 for NEXTSTEP architectures

        SCO UnixWare

        Solaris and SUN OS

Author: Vic Abell

Platform: UNIX

URL: ftp://vic.cc.purdue.edu/pub/tools/unix/lsof/

Private-I

Private-I has two primary functions. First, it serves as a back-end log archiver for Cisco IOS-based routers, PIX and Checkpoint firewalls, and RedCreek VPN devices. Second, it is capable of generating real-time alerts based on known firewall and IOS event codes. Because Private-I has been designed to process the vendor-specific event codes piped to it via syslog, it can alert administrators of problems in real-time, as well as produce informative reports.

OpenSystems.com

55 West St.

Walpole,MA02081

USA Phone: [phone]

URL: http://www.opensystems.com

WebSense

Though WebSense is best known for its screening capabilities, the product also has powerful logging capabilities. (These have recently been enhanced as the product has been designed to work closely with PIX firewalls from Cisco.)

WebSense, Inc.

World Headquarters

10240 Sorrento Valley Rd.

San Diego,CA 92121

Phone: [phone]

Fax: [phone]

Email: [e-mail]

URL: http://www.websense.com/

Win-Log version 1

Win-Log is a very simple utility for Windows NT. It logs when, how often, and how long Windows NT is used. (You can use this utility to ascertain whether someone has been rebooting your box, even if they somehow circumvent Event Logger.)

iNFINITY Software

Email: [e-mail]

URL: http://www.isoft.demon.co.uk/

NOCOL/NetConsole v4

NOCOL/NetConsole v4.0 is a suite of standalone applications that perform a wide variety of monitoring tasks. This suite offers a Curses interface, which is great for running on a wide range of terminals. (It does not require X to work.) It is extensible, has support for a Perl interface, and operates on networks running AppleTalk and Novell.

NOCOL/NetConsole v4

Location: ftp://ftp.navya.com/pub/

 

Section: Chapter 13.  Logging and Auditing Tools

Tools for Analyzing Log Files

The following tools examine log files, extract the data, and generate reports.

NestWatch

NetWatch can import log files from all major Web servers and several firewalls. NestWatch runs on Windows NT and can output reports in HTML and distribute these to servers of your choice.

Scandinavian Security Center

Hermodsvej 5A, 2.

DK-8230 Aabyhoej

Denmark

Phone: + 45-8744-7800

Fax: + 45-8744-7820

Email: [e-mail]

URL: http://www.sscnet.com/nestwatch.html

NetTracker

NetTracker analyzes both wall and proxy files. The product has extensive filtering and reporting, and can export data to Excel and Access file formats. (The product also can analyze general access logs and format custom reports suitable for graphing.) NetTracker runs on Windows 95/Windows NT; a 30-day evaluation is available on the Web.

Sane Solutions, LLC

35 Belver Ave., Suite 230

North Kingstown, RI 02852

Phone: [phone]

Email: [e-mail]

URL: http://www.sane.com/products/NetTracker/

LogSurfer

LogSurfer is a comprehensive log analysis tool. The program examines plain text log files and based on what it finds (and the rules you provide), it can perform various actions. These might include creating an alert, executing an external program, or even taking portions of the log data and feeding that to external commands or processes. LogSurfer requires a C compiler.

Univ. Hamburg, Dept. of Computer Science

DFN-CERT

Vogt-Koelln-Strasse 30

22527 Hamburg, Germany

Location: ftp://ftp.cert.dfn.de/pub/tools/audit/logsurfer/

WebTrends for Firewalls and VPNs

WebTrends for Firewalls and VPNs combines Web link, usage, and traffic analysis with log analysis for the following firewall:

        Firewall-1

        NAI/TIS Gauntlet

        Raptor

        Cisco PIX

        Lucent Managed Firewall

        IBM eNetwork Firewall

        Novell Proxy Server

        Netscape Proxy Server

        Microsoft Proxy

WebTrends can pull some very impressive statistics and writes to a wide variety of database report formats. (This product runs on Windows NT and Windows 95.)

WebTrends Corporation

851 SW 6th Ave.

Suite 1200

Portland, OR 97204

Phone: [phone]

Fax: [phone]

Email: [e-mail]

URL: http://www.webtrends.com

Analog

Analog is probably the only truly cross-platform log file analyzer. Analog currently runs on the following operating systems:

Macintosh	VAX/VMS
OS/2	RISC/OS
Windows 95/NT	BeOS
UNIX	BS2000/OSD

Not only is Analog cross-platform, it also has built-in support for a wide variety of languages, including English, Portuguese, French, German, Swedish, Czech, Slovak, Slovene, Romanian, and Hungarian.

Analog also does reverse DNS lookups (slowly), has a built-in scripting language (similar to the shell languages), and has at least minimal support for AppleScript.

Lastly, Analog supports most of the well-known Web server log formats, including Apache, NCSA, WebStar, IIS, W3 Extended, Netscape/iPlanet, and Netpresenz.

Author: Stephen Turner

University of Cambridge Statistical Laboratory

URL: http://www.statslab.cam.ac.uk/~sret1/analog/

 

Section: Chapter 13.  Logging and Auditing Tools

Specialized Logging Utilities

These utilities are included more for academic reasons then anything else. Although still useful, their utility is quickly being absorbed by that of network-based intrusion detection systems. However, readers are still encouraged to download them and try them out.

Courtney

Courtney is a Perl script designed to detect and log SATAN attacks. As described in the Courtney documentation:

Courtney receives input from tcpdump counting the number of new services a machine originates within a certain time window. If one machine connects to numerous services within that time window, Courtney identifies that machine as a potential SATAN host.

System requirements include libpcap-0.0, tcpdump-3.0, and perl5.

Author: Marvin J. Christensen

URL: http://packetstorm.securify.com/UNIX/audit/courtney-1.3.tar.Z

Note

Tools like NMAP and STROBE (port scanners) by default open many socket connections in a short period of time. This behavior is highly unusual and is easily distinguished from legitimate user activity. Tools like Courtney rely more on the behavior of incoming hosts (and their control loop) than they do on the type of data being transmitted.

 

Gabriel

Gabriel serves the same purpose as Courtney to log and warn of SATAN attacks. However, Gabriel is designed very differently and relies on one server and a series of clients to constantly distribute status reports. These status reports indicate various patterns of resource usage by remote hosts. When a host appears to be eating an inordinate number of resources (or requesting an abnormal number of connections), that host is flagged at a high priority. (Note: Gabriel relies largely on syslog.)

You need a generic UNIX system, a C compiler, and network include files to run Gabriel.

Los Altos Technologies, Inc.

01 First Street, Suite 790

Los Altos, CA 94022

Phone: 800.999.UNIX

Fax: [phone]

Email: [e-mail]

URL: http://www.lat.com/gabe.htm

 

Section: Chapter 13.  Logging and Auditing Tools

Summary

Never underestimate the importance of keeping detailed logs. Not only are logs essential when you're investigating a network intrusion, they're also a requisite for bringing charges against an attacker. Sparse logs simply won't do.

In recent years, many criminal cracking cases have ended in plea bargains. One of the primary reasons for this is because perpetrators were often kids kids who were just "having a little fun." But, plea bargains are becoming less prevalent as real criminal elements migrate to the Net. Real criminals know that proving a case before a judge or jury is very difficult (especially if the prosecution has little Internet experience). When judges and jurors are asked to send a human being to prison, they need substantial proof. The only way you can offer substantial proof is by having several fail-safe methods of logging.

Crimes perpetrated over the Internet are unlike most other crimes. For example, in a robbery case, crooks are placed in a lineup so the victim can identify the culprit. In burglary cases, fingerprints will generally reveal the identity of the perpetrator. On the Internet, however, you have neither a physical description nor fingerprints. Therefore, without logs, making a case against a cracker is almost impossible.