11.2. Forensics on BSD

Once you have determined a host has been compromised, you may want to perform an analysis of the host to figure out what the attacker really did. This process, known as forensic analysis, can be a highly involved activity. Keeping track of evidence, performing highly technical tests against a host, and staying current on the latest attack trends can be a difficult job. However, the roots of forensic analysis are really simple common sense and investigation. Your detective work, ultimately, needs to be educational to you and help your enterprise learn from the compromise. Keep that goal in mind as you actually examine exploited hosts.

11.2.1. How Serious Are You?

The first thing you need to ask yourself is: what are you planning on doing with the data you gather from your analysis? If the break-in was severe and legal action may be taken against the attacker, you must perform a dramatically more detail-oriented analysis than if you are pursuing a minor-script driven attack that you are unlikely to follow up on. In some cases, you may find during the course of your incident response procedure that the vector of attack is obvious and your time is better spent bringing the host back online than actually performing forensics work. Just because a host was broken into does not mean you have to perform analysis on it, but if you are going to simply bring the host back into operation without determining how the attacker got in and how to stop him from doing it again, be prepared to rebuild the host every time it is compromised.

Quick Forensics

Sometimes, the avenue an attacker uses to gain access to a host is made obvious through a very cursory incident response procedure. In these cases, lengthy paperwork-driven procedures are more trouble than they are worth.

One of the authors was once involved in an incident where a Solaris FTP server appeared to have been compromised and was being used to distribute French pornography. Upon initial inspection of the FTP process that was running, it was clear that the compromise came as a result of flags that allowed anonymous write access to the public FTP directories. This excessive access was what the attackers were using to store and retrieve their pornography.

The system administrator who was responsible for the server was asked why he was allowing anonymous write access to the FTP server. As it turns out, the admin was a BSD administrator and used the flags appropriate for the BSD-based FTP server. Unfortunately, the same flags on the FTP server on the Solaris host allowed for world writable access. No further forensic analysis was performed.

11.2.2. Online and Offline Analysis

Depending on the state of the machine you're dealing with and the urgency with which you need to analyze the compromise, there are different types of analysis you may choose to do. At a high level, there are two types of forensic analysis;:online and offline.

Online analysis is performed against a running system. Using some manner of trusted media with forensic tools such as a CD-ROM, online analysis allows for the examination of the system as is operates. The advantage of online analysis is that you can examine what processes are running on a box and the characteristics of the running system. This type of analysis may result in a deep knowledge of how the box was compromised and what types of hacker tools are running on the host. Poking around after a reboot, or simply examining the static data on disk, may not provide a complete picture of what is going on.

The disadvantage of online analysis is that you may be disturbing and changing the host. Some attack tools detect when they are being "prodded" and may start deleting data or performing other malicious activities. Further, if you are investigating a host to which the attacker still has access (you have not unplugged the network cable) the attacker may begin changing things as you look at them.

Offline analysis is the act of performing forensics activities against a "dead" system. Rather than acting on a running host and examining processes and memory, an offline analysis deals with disk images and static files. The advantage of an offline analysis is that the data on the disk is relatively safe. Disks can be copied and write protected and all binaries can be made nonexecutable. This prevents some manner of logic bomb left by the attacker from changing the data on the compromised host. Also, multiple copies can be created allowing multiple individuals to analyze the data concurrently. If you want to do offline analysis but also want to examine what processes were running on the host, you can shut the host down using halt -d. This will force a crash dump and provide access to process tables and other kernel-level information.

11.2.3. Things to Look For

Forensic analysis is not a magic wand. There is no single tool that you can run that immediately returns "Host owned through OpenSSH exploit by Billy Bob with Social Security Number 123-45-6789."

11.2.3.1 Changed files

One of the first thing you should look for are changed files. An attacker may modify configuration files such as /etc/inetd.conf to start new services or install a completely different binary specifically designed to hide her tracks. netstat(1) is a binary utility that is commonly modified by an attacker to hide new network connections. The Trojaned netstat will only show non-attacker connections to give the illusion that the network status is normal.

You may find changes in a variety of ways. Visual inspection of inetd.conf may help you find footprints of an attack. However, manual comparison of a binary is basically impossible. Ideally, you will have a HIDS like Osiris or Tripwire (covered in Chapter 9) running that will allow you to compare checksums of existing files and the original files on the host.

When performing an online analysis of a potentially compromised host, assume that all utilities on the machine are not trustworthy. Every utility you run should be launched from a trusted read-only media such as a CD-ROM. When viewing inetd.conf, for instance, use an editor or viewer from the CD-ROM, not the native less utility on the host.

11.2.3.2 Added users

Another popular maneuver by an attacker is to add new users to the system to allow remote logins and processes to be executed. Again, if you don't have a known good copy of the passwd file, you're at a disadvantage, especially if you have a large number of users on the system. Hopefully there are uncompromised hosts that have a similar user set that you can compare against.

When creating users, attackers like to create accounts with high privileges. A root-level account will allow an attacker to perform superuser activities on a host without having to do any other trickery. The dead giveaway is a user with UID of zero, the superuser UID. Any user with UID of zero that is not root or toor is likely not an authorized account.

Also, an attacker may attempt to create an account that looks like a system account but is really his gateway account into a box. For example, a typical FreeBSD installation has a number of system accounts preinstalled that are used for a variety of system functions. In Example 11-1, there's a user that shouldn't be there.

Example 11-1. /etc/passwd on a FreeBSD host

root:*:0:0:Charlie &:/root:/bin/csh toor:*:0:0:Bourne-again Superuser:/root: daemon:*:1:1:Owner of many system processes:/root:/sbin/nologin operator:*:2:5:System &:/:/sbin/nologin bin:*:3:7:Binaries Commands and Source:/:/sbin/nologin tty:*:4:65533:Tty Sandbox:/:/sbin/nologin kmem:*:5:65533:KMem Sandbox:/:/sbin/nologin games:*:7:13:Games pseudo-user:/usr/games:/sbin/nologin news:*:8:8:News Subsystem:/:/sbin/nologin man:*:9:9:Mister Man Pages:/usr/share/man:/sbin/nologin cron:*:10:10:Cron Admin User:/:/bin/csh sshd:*:22:22:Secure Shell Daemon:/var/empty:/sbin/nologin smmsp:*:25:25:Sendmail Submission User:/var/spool/clientmqueue:/sbin/nologin

Without looking carefully, you could easily overlook the illegitimate cron user with a shell of /bin/csh.

11.2.3.3 Strange directories

Other tidbits to look for are oddly named directories. Attackers may try and hide data and tools in directories that are difficult to find or cd into. A directory named ... may blend in with the typical ls -al listing, but may contain traces of an attack.

% ls -al total 636 drwxr-xr-x   6 root  wheel     512 May 16  2004 . drwxr-xr-x  21 root  wheel     512 Jun 24 16:11 .. -rwx------   1 root  wheel    6750 Oct 23 13:49 ... -r--r--r--   1 root  wheel    8192 Feb 23  2004 boot

Also, directories with control characters or other nonprintable characters may be difficult to see or access. If you have problems accessing a file or directory (i.e., you try and change into a directory and get an error that there is no directory by that name) use ls -q to print the contents of the directory. The -q flag turns all nonprintable characters into question marks allowing you do determine that something really is amiss. In this example, ls -al returns a normal looking output, but ls -lq paints a different story.

% ls -al total 2942 drwxr-xr-x   5 root  wheel      512 Nov 18 19:52 . drwxr-xr-x  21 root  wheel      512 Jun 24 16:11 .. drwx------   1 root  wheel     1024 Nov 17 17:21 testdirectory drwxr-xr-x   3 501   501        512 Nov  4 12:04 osiris-4.0.6 -rw-r--r--   1 root  wheel  1882069 Nov  4 12:05 osiris-4.0.6.tar.gz % ls -lq total 2942 drwxr-xr-x   5 root  wheel      512 Nov 18 19:52 . drwxr-xr-x  21 root  wheel      512 Jun 24 16:11 .. drwx------   1 root  wheel     1024 Nov 17 17:21 test??directory drwxr-xr-x   3 501   501        512 Nov  4 12:04 osiris-4.0.6 -rw-r--r--   1 root  wheel  1882069 Nov  4 12:05 osiris-4.0.6.tar.gz

11.2.3.4 Unknown processes and LKMs

Attackers may start new processes on a host or inject kernel modules to cause the host to perform unwanted activities such as network scanning, password cracking, and file sharing. A standard ps -auxww that is common in BSD administration can reveal if strange new processes are afoot. On a busy system, there may be quite a long process list and it may be difficult to determine what is legitimate and what is not. Take your time and do not jump to conclusions.

You may recall from Chapter 2 that running in a securelevel of 1 or greater will help ensure that kernel modules cannot be loaded.

Kernel modules can be checked with kldstat(8). Again, if you don't know what your baseline kernel modules are, it may be difficult to determine what is legitimate and what is the result of an attack. Osiris, discussed in Chapter 9, has the ability to monitor kernel modules and track when new modules are brought online.

In Example 11-2, kldstat shows a suspicious kernel module at ID 4. This is worth investigating.

Example 11-2. Running kldstat to see loaded modules

% kldstat Id Refs Address    Size     Name  1    7 0xc0400000 5e74fc   kernel  2    1 0xc09e8000 51ac8    acpi.ko  3    1 0xc27ac000 19000    linux.ko  4    1 0xc4200000 27e12    rootkit.ko

11.2.3.5 Known rootkits and hacker tools

The tools installed by an attacker who has broken into your system are generally not something specifically written by the attacker. There are a common set of tools used by attackers, and many times they are bundled into rootkits. The Wikipedia definition from http://www.wikipedia.org/wiki/Rootkit is as follows:

A rootkit is a set of tools used after cracking a computer system that hide logins, processes, and logs as well as usually sniff terminals, connections, and the keyboard.

Called "root" kit after the fact that originally it referred to a set of recompiled Unix tools such as "ps" "netstat" "w" "passwd" that would carefully hide any trace of the cracker that those commands would normally display, thus the cracker could maintain "root" on the system without the system administrator even seeing them.

That sounds daunting. Rootkits are specifically designed to evade detection and hide an attackers tracks. Thankfully, rootkits leave fingerprints on a system much like viruses do. Like viruses, there are tools designed to track down rootkits and help users identify what the rootkit does.

One of the better tools is called Rootkit Hunter and is available from http://www.rootkit.nl/projects/rootkit_hunter.html. Rootkit Hunter examines the md5 checksums of files on the system to see if they match known rootkits, looks for odd binary permissions, searches for hidden files, and tries to determine malicious kernel modules. Sounds convenient, huh?

Rootkit Hunter is straightforward to download and install. When you run Rootkit Hunter, you probably want to scan the entire host. It's hard to say where an attacker may or may not have left tools on the host. But using the -checkall flag, all Rootkit Hunter tests are run. Further, with -skip-keypress Rootkit Hunter will run non-interactively allowing you to run the tool and not have to babysit it.

When it's done, Rootkit Hunter will summarize the results. In Example 11-3, two vulnerable applications were found that should be patched. However, no rootkits were found. This does not necessarily mean there are no rootkits on the host, but simply that this tool didn't find any.

Example 11-3. Rootkit Hunter sample results

---------------------------- Scan results ---------------------------- MD5 MD5 compared: 33 Incorrect MD5 checksums: 0 File scan Scanned files: 328 Possible infected files: 0 Application scan Vulnerable applications: 2 Scanning took 97 seconds