Performing Forensics with the Coroner s Toolkit | Hacking Knoppix (ExtremeTech)

Performing Forensics with the Coroner's Toolkit

One of the most popular shows on TV right now is CSI: Crime Scene Investigation. Every week, an intrepid team of investigators uses forensic evidence and the latest scientific tools to plumb the depths of human crime and malevolence. Using clues such as blood spatters, footprints, teeth impressions, and gunpowder residue, the past can be reconstructed and crimes can be solved.

In the computer world, forensics is a well-known aspect of computer security. When a machine is compromised, it's important for a qualified, knowledgeable person to use tools such as the Coroner's Toolkit (known as TCT) to enable the reconstruction of the computer to learn the following:

The exact causes of the break-in can be reasonably ascertained
The extent of the damage can be cataloged
The evidence can be preserved in a way that can be used in legal proceedings

Caution

That last point is especially important. If a court case is to be made, it is imperative that the evidence is gathered and preserved in a way that does not allow for an argument to be made that the gathering and preservation itself changed or manipulated any of the evidence. For that reason, I do not encourage you to intrude yourself if the legal system is going to be involved. Instead, engage the services of a security professional (who may very well run TCT).

The Coroner's Toolkit is actually a collection of software packages, some of which can be run independently, that work together to give you a snapshot of the activities a computer was engaged in at a specific time. It includes the following:

Grave-robber: Runs several other tools that gather together as much information about a computer system as possible.
Mactime: One of the tools used by grave-robber, which you can also run independently, that looks at the last time the following occurred:
- A file was modified (the file's mtime)
- A file was accessed for reading (by ls or cat, for instance) or writing (the file's atime)
- A file's inode was changed, by changing permissions or ownership, for instance (the file's ctime)
Unrm: Copies all free disk space
Lazarus: Analyzes data gathered by unrm

Note

Don't know what an inode is? Wikipedia tells you at http://en.wikipedia.org/wiki/Inode. Wondering why creation time isn't listed? Because UNIX-based systems don't keep track of a file or an inode's creation time!

Using Grave-robber

You can run grave-robber against either a live system or a corpse. A live system is just that: one that hasn't been rebooted and is still the site of a suspected break-in. A corpse, on the other hand, is a backup copy of the problematic disk (usually created using the dd command) or a problematic disk that has been mounted on a system.

Insofar as this book is concerned, you're running TCT against a corpse because you would have to reboot anyway to use the software on a Knoppix disk. That mean using the dd command to image the suspected hard drive while it was still alive, booting with Knoppix, mounting the image using the loopback device, and then running TCT against the mounted image, now a corpse (be aware that the last command involving grave-robber is usually run against an entire filesystem, so it can take hours and hours or even days). Here's the quick series of commands:

 # dd if=/dev/hda1 of=/mnt/hda2/corpse.img

Then reboot with Knoppix and continue:

 $ sudo mkdir /mnt/corpse $ sudo mount -o loop,ro /mnt/hda2/corpse.img /mnt/corpse $ sudo grave-robber -c /mnt/corpse -o LINUX2

Note

For the commands used on a live system, see man grave-robber.

These commands are what you would normally do. While you're learning, though, run graverobber against the /KNOPPIX/ directory, which should give you more than enough data to play with (this is kind of like using a corpse because /KNOPPIX is on the CD-ROM and is therefore mounted read-only like the corpse in the preceding list of commands). To start TCT, run the following and leave your computer for a while:

 $ sudo grave-robber /KNOPPIX/

For the full list of options that you can use when running grave-robber, check out man graverobber. By not explicitly choosing an option, grave-robber runs with the following options as default: -i, -m, -M, -P, -s, -t, -l, -I, -O, -F, -S, and -V. Use your free time to find out what all of those do!

Eventually, the process finishes (it can take quite a while depending upon the speed of your processor, your CD drive, and other factors), and then you'll see something like this as output on the terminal:

 Starting preprocessing paths and filenames on Knoppix... Processing $PATH elements... /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /bin /usr/X11R6/bin         Processing dir /         Processing dir /etc         Processing dir /bin         Processing dir /sbin         Processing dir /dev Finished preprocessing your toolkit... you may now use programs or examine files in the above directories

TCT places two very important log files — coroner.log and error.log — in your current directory. The first tells you exactly what commands grave-robber ran, and the dates and times those commands ran, as it did its work; the second lets you know — surprise! — about any errors that grave-robber encountered. Both can help you learn exactly what grave-robber does and explain why things don't work as you might have expected them to.

TCT places its output files at /var/cache/tct/data, but you need to be root to see them (which makes sense because you run grave-robber as root). The easiest way to view the files is to run Konqueror as root. From the K menu, select System → Root Terminal, logging in as root, and then entering konqueror to run that program.

You should have already set a root password to open the Root Terminal. If you haven't, open the Terminal Program on the panel, enter sudo passwd, and then assign a new password for root.

Once Konqueror is open, navigate to /var/cache/tct/data and start looking at the treasure trove of data that grave-robber has made available to you. Here's an overview of some of the files and folders you should see:

body: The mactime database, used by the mactime program
body.S: File attributes of all suid files. The same data is in the body file, but it's called out here.
command_out/: Output of programs grave-robber executes to find out information about the system, such as top, dmesg, arp, and netstat
conf_vault/: Copies of files that grave-robber found that looked interesting, such as configurations and other critical files
proc/: Images of running processes gathered from the /proc filesystem
removed_but_running/: Deleted files that were still running
trust/: Files presenting information about trust relations between computers and users, such as cryptographic keys from GPG and SSH, command histories, and .forward files

As an intrepid forensics investigator, your job is to look over these files and try to reconstruct the machine's state when grave-robber was run against it. This is going to take experience and time, so jump right in. Look at everything, try to figure out what it signifies, and glean all that you can from the data you have. No one said it was going to be easy!

Running Mactime

The mactime command is a bit easier to get a handle on. Remember that it can be run independently of grave-robber, which means that you could start with it and master that tool before moving on to the full glut of data provided by grave-robber. For a quick look at how to utilize mactime, run it against the /etc/init.d directory, which holds a lot of the programs run as services on your Knoppix machine:

 $ sudo mactime -y -R -d /etc/init.d 1/1/1970

The options used in this command mean the following (for the full list of options, use man mactime):

y: Display the date as YY MM DD, instead of MM DD YY, which makes for better sorting.
R: Process the current directory, and all subdirectories, recursively.
d: Use the following directory instead of a database (/etc/init.d in this example).
1/1/1970: Show all files modified, accessed, or changed after this date (in this example, the UNIX epoch, seen by UNIX systems as the start of time); it's also possible to use a range, such as 4/1/2005–4/8/2005.

For more information on the UNIX epoch, see the Jargon File at http://clueless.com/jargon3.0.0/epoch.html or Wikipedia's in-depth "Unix time" article at http://en.wikipedia.org/wiki/Unix_epoch.

The example command should produce output containing at least some of the following:

 99 Nov 07 10:35:31     2213 m.. -rwxr-xr-x root     root     /etc/init.d/rc 99 Nov 13 11:24:46     1909 m.. -rwxr-xr-x root     root /etc/init.d/bootmisc.sh 00 Feb 11 05:52:19     1478 m.. -rwxr-xr-x root     root /etc/init.d/mountnfs.sh 00 Apr 03 07:57:46     1074 m.. -rwxr-xr-x root     root     /etc/init.d/atd 00 Jun 05 12:36:19      853 m.. -rw-r--r-- root     root     /etc/init.d/README 00 Jun 06 04:42:13      420 m.. -rwxr-xr-x root     root     /etc/init.d/umountnfs.sh 00 Jun 06 04:42:14      430 m.. -rwxr-xr-x root     root /etc/init.d/umountfs                         601 m.. -rwxr-xr-x root     root /etc/init.d/mountall.sh 00 Jul 27 23:05:51      927 m.. -rwxr-xr-x root     root     /etc/init.d/ipx 01 Jan 28 01:29:37      800 m.. -rwxr-xr-x root     root     /etc/init.d/mt-st 01 May 11 20:03:54     1007 m.. -rwxr-xr-x root     root     /etc/init.d/kerneld                         197 m.. -rwxr-xr-x root     root     /etc/init.d/reboot

Notice that files are first listed in order from oldest to newest using the date format specified, and then by the time. The next column provides the size of the file in bytes, and then the action on the file (m for modified, a for accessed, and c for changed). Following that are permissions and ownership, similar to running ls -l.

In this listing, it's very easy to determine whether files were perhaps altered within a specific time span. If you see a file that has been altered, and you don't remember changing it, tampering may be indicated. The mactime command can help you identify files that should get your attention. With that in mind, it might be nice to run it regularly with a cron job and have the output emailed to you for possible follow up later.

Looking at Unrm and Lazarus

The other two major programs in TCT — unrm and lazarus — are run separately from grave-robber. This discussion won't walk you through all the details of these programs, but it provides an overview so you'll know what they are and how to use them.

Caution

Be aware that unrm has one very large limitation: At this time, it works only on Linux systems formatted with the ext2 filesystem. Because more and more Linux distros ship with journaled filesystems such as ext3, reiserfs, and even XFS by default, this greatly limits the usefulness of unrm.

Suppose that you accidentally delete a file on your computer, or that you're trying to see whether any bad guys took over a machine and attempted to delete evidence of their presence. In those cases, the combination of unrm and lazarus will help. Run unrm first; it makes a copy of all free disk space. Then lazarus analyzes the data in the copy created by unrm and tries to figure out what the data is.

Understanding Unrm

Think about what unrm does for a second before you run it. It makes a copy of all free disk space. In other words, if you have a 60GB filesystem drive, and 19GB are used on that filesystem, unrm is going to make a bit-by-bit copy of everything not occupied by those 19GB (and then lazarus is going to process those files, and in so doing create another copy). Hmmm …

This means you need to consider two things before running unrm. First, unrm is going to copy 41GB — everything not used by the 19GB of data — and lazarus is going to generate another 41GB. Second, just where are you going to put those 82 (41 + 41) GB? You can't put them on the same filesystem you're running unrm against, or you'll be busily overwriting the very things you're trying to recover! You must send unrm's output to a different filesystem.

To check the deleted space on /mnt/hda2 using Knoppix, for example, you boot Knoppix and run the following:

 $ df -h

The output is something like the following (your results will obviously differ):

 Filesystem         Size Used Avail Use% Mounted on /dev/root          3.4M  13K  3.4M   1% / /dev/hdc           695M 695M     0 100% /cdrom /dev/cloop         1.9G 1.9G     0 100% /KNOPPIX /ramdisk           392M  75M  318M  19% /ramdisk /UNIONFS           6.1G 5.6G  477M  93% /UNIONFS /dev/hda2           27G  25G  1.7G  94% /mnt/hda2 /dev/hda1          9.8G 5.2G  4.7G  53% /mnt/hda1

You can see that hda2 is a 27GB file system, with 1.7GB free. Fortunately, hda1 has 4.7GB free, so it's going to be close but not a problem. You can send the unrm output to hda1 for safekeeping.

Before running unrm, you must umount the filesystem against which you want to run unrm:

 $ umount /mnt/hda2

Check that you've mounted hda1 with writing allowed, or this process won't work. Now run the unrm command, sending the output to a file on the still-mounted hda1:

 $ unrm /dev/hda2 > /mnt/hda1/hda2_unrm_results

The unrm command can take a while to run, depending on the amount of free space on a device. Once it finishes, it's time to go to work with lazarus.

Understanding Lazarus

Lazarus analyzes the data dug up by unrm and tries to bring anything it can find back to life. Be warned: lazarus can take twice as long to complete its job as it took unrm to work, so this may be one of those commands that you start and then go to bed. Here's how to run it:

 $ lazarus -h /mnt/hda1/hda2_unrm_results

The -h option tells lazarus to produce a report in HTML that you can load in a browser and use. That report is composed of three files: hda2_unrm_results.html, hda2_ unrm_results.menu.html, and hda2_unrm_results.frame.html; the last one is the one you need to open in your Web browser, and it takes care of loading the others.

Lazarus tries — emphasis on the word "tries" — to figure out whether recovered data is binary or text, and then it tries to figure out what kind of text or binary data it is: HTML, mail, logs, executable, sound, and so on. A key is available in the HTML files that lazarus displays in your browser. Most of the Web page is taken up with a list of those keys, one for each file that lazarus recovered. Click the link, and you can see the contents of the file.

This is very similar to looking for a needle in a haystack, of course, so you may want to speed things up by using grep. For example, if you're looking for a file having to do with the McGillicuddy account, you could try grep McGillicuddy /mnt/hda1/hda2_unrm_results/*. Try long enough, and you may just find that file you so desperately need.

Note

For further info about grep, try man grep. For more information about TCT, see Clarke L. Jeffris' "The Coroner's Toolkit — In depth," a PDF available at http://www.sans.org/rr/whitepapers/incident/651.php. For an excellent overview of unrm and lazarus, see http://www.antioffline.com/TCT/help-recovering-file, written by the folks behind the two programs, or CERT's "Using The Coroner's Toolkit: Rescuing files with lazarus" at http://www.cert.org/security-improvement/implementations/i046.03.html.

For more on computer forensics, see Timothy E. Wright's "A Method for Forensic Previews" at http://securityfocus.com/infocus/1825. Another good piece, which also contains some great info on TCT, is Derek Cheng's "Freeware Forensics Tools for Unix," at http://securityfocus.com/infocus/1503. There's a massive list of whitepapers, FAQs, and other great links at http://forensics.nl/links, and a nice list of forensics software at http://forinsect.de/forensics/forensics-tools.html.