Advanced Digital Forensic Tools

We have explored the basic forensic process and methodology that should be used, the basic tools to image a disk drive or partition, and the more advanced technique of using Foremost for recovering data. Now we should mention some more advanced tools and techniques that you might need in your forensic analysis.

A common problem when trying to lift data from a recovered system is an issue with reading the Master Boot Record (MBR) of a disk, which stops the recovery workstation from reading the partition information. This may happen because the disk/system was intentionally or accidentally damaged and it stops many inexperienced forensic investigators in their tracks. What happens if there is damage to the physical hardware and dd won't make a disk image? Is it still detected by the hardware at boot-time? There may be ways to recover it.

dd_rescue to the Rescue

If dd won't make a disk image because it is not designed to deal with hardware disk errors, another option is a utility appropriately named dd_rescue, written by Kurt Garloff. It should also be noted that there is another program with a similar name, called ddrescue, and released by GNU directlyboth are licensed under the GNU General Public License (GPL). Both of these programs do basically the same thing, but we prefer the dd_rescue program as it has a slightly more elegant user interface and appears to be a bit faster, with some additional features, which we'll get to. Disk Dump Rescue, as it name would imply, is designed to deal with hardware disk errors and can create disk images, in a similar fashion as dd, because it does not abort when an error occurs, or always truncate the output as dd will. There are several options and methods for handling these errors and ways of dealing with them. First off, dd_rescue's command line options are very similar to dd but with a slightly simpler syntaxyou don't need to specify the if =< file > or of =< file > options. The basic execution and options are as follows :

  dd_rescue [options] <infile><outfile>  

• -s < n > Starting position (offset in bytes from the beginning of the file: default is zero)

• -b < n > Block size for copy operation (default is 16,384 bytes)

• -e < n > Exit after errors (default is zero, never exit because of errors)

• -l < file > Log file to log error messages to

• -v Verbose output (useful to monitor progress)

For example, the following listing would execute dd_rescue using the disk device, assuming some partitions can't be read, and create a complete disk image where any errors that are found will be written with zeroed blocks instead of truncated ones (not written).

 # dd_rescue -v -1 /data/disk_images/target1-hda-dd_rescue.log \          -b 16384 -B 512 -A /dev/hda /data/disk_images/target1-hda.dd_rescue_image 

Also, if an error is encountered , the block size will be reduced from the default (16,384 bytes) to a much smaller size (512 bytes), to improve the likelihood of capturing data that is not readable (because of the errors). Some additional more advanced options are listed here:

• -B < n > Fallback blocksize when errors are encountered (default is 512)

• -A Always write blocks; zeroed if an error occurs (default is not to write zero length blocks, but to truncate)

• -r Read the input file in reverse. This is extemely useful if the disk partitions are so damaged that dd_rescue can't read large areas of the drive. Reading damaged areas of a disk with dd_rescue is not a quick process. Be patient.

As mentioned above, the -r option is used to read the input file in reverse. This is extremely useful for advanced recovery, as you can start at the end of the disk, read the disk in reverse, including the option of starting at particular offsets to read portions of the disk you are trying to recover, if that is preferable. Remember, the output will then be in reverse, so you will need to reverse the disk image created to perform further analysis. An example recovery using these options is provided below:

 # dd_rescue -A  -r  /dev/hda /data/disk_images/target1-hda-2.dd_rescue_image_reverse # dd_rescue  -r  /data/disk_images/target1-hda-2.dd_rescue_image_reverse \                      /data/disk_images/target1-hda-2.dd_rescue_image 

What other options are there? What type of filesystem is it?

disktype

The disktype utility, written by Christoph Pfisterer, is a useful program that works similar to the way the file utility works to read information from the disk and make a guess as to what type of partition and filesystem it contains. disktype contains signatures for almost every type of filesystem, partition type, and archive file format out there: Windows FAT, NTFS, and various UNIX file systems. It supports pretty much anything across the board: boot loaders; image files such as ISO9660, tar, and cpio; and compressed files such as compress (.Z), gzip, and bzip2. This is extremely useful if you don't know what type of filesystem, disk format, or partition type the data are that you are analyzing.

The listing below is the output from executing disktype on both a mounted and unmounted disk image.

 # disktype /dev/hda1 -- - /dev/hda1 Block device, size 18.63 GiB (20003848192 bytes) Ext3 file system   UUID 0FAE7EA1-E975-42D4-8DD1-28180C0E6595 (DCE, v4)   Volume size 18.63 GiB (20003848192 bytes, 4883752 blocks of 4 KiB) # disktype target1-hda1.dd_rescue_image -- - target1-hda1.dd_rescue_image Regular file, size 18.63 GiB (20003848192 bytes) Ext3 file system   UUID 0FAE7EA1-E975-42D4-8DD1-28180C0E6595 (DCE, v4)   Volume size 18.63 GiB (20003848192 bytes, 4883752 blocks of 4 KiB) 

Other tools such as gpart and TestDisk may also help in this process as well. These utilities can guess at the partition types of partitions, and restore/undelete partitionswe'll let you investigate these further on your own. Now that you have the disk image made, you can identify and mount the partition and determine what files were modified and when.

The Coroner's Toolkit (TCT)

TCT, written by Dan Farmer and Wietse Venema, is a collection of forensic tools and a framework to run them in. The tools automate the forensic processthey may be used either before or after a disk image has been made; however, some will only be useful before. TCT has been around since at least the late 1990s, and it is designed to assist you in the post-mortem analysis of a compromised running system, or from an offline system image. It automates the process of collecting the information, which can be a very time-consuming and error-prone process, some of which, if not done correctly, can further compromise or hinder your ability to recover the information. TCT doesn't do the analysis for you; it simply collects and organizes the data for your review. Written as a general front-end to many other useful utilities, it leverages their work. Some of the tools or procedures that TCT can leverage are

• ps The process list utility.

• Isof Displays a list of open files and associates them with the process by which they were opened. On the Windows platform, several programs exist that do a similar function. One of the most popular is called Process Explorer, by Sysinternals.

• Deleted Files Using a combination of ils and icat commands, some deleted files that are still open and in memory are often recoverable. In addition, the lazarus utility, which is a bit experimental, may be able to recover deleted files from temporary or swap file space. The unrm command may be useful as well, before using lazarus to extract data that was recently freed. These tools aren't for the faint of heart. You could waste a lot of time messing with these programs to produce nothing.

• pcat This utility can recover the environment of running processes. This is useful to look at what was in memoryas often there is history informationthat might reveal commands that were typed or IP addresses from which connections were made.

• Set-UID programs List of all the Set-UID programs that were found.

• Mactime TCT will also automate the process of computing a mactime database for later review. This is one of the most important steps, and the primary reason for utilizing TCT. MAC time is modified, access, and creation times on files, and almost any operation on a file changes this information, so (1) it must be done carefully and (2) quickly. The mactime utility can then be used to receive lists of files that correspond to the dates provided.

• Backup of Configuration Interesting configuration files to investigate.

• Users' Shell Histories Copies users' shell histories for investigation.

• Users' SSH Keys and Trusted Hosts Copies users' SSH keys and trusted hosts files for investigation.

Let's look further at the mactime database after running the general grave-robber script, which extracts all the information for our review. First, the grave-robber script was run using the following command-line:

 [root@target1 tct-1.15]# ./bin/grave-robber -v / 

Note 

This is not a quick processit may take several hours or more. Be patient! When complete, grave-robber should have created a directory for all the output it generated under the data directory, inside the package directory of TCT. The directory should contain the hostname and the date of execution. There is a symbolic link, with just the name of the host, for your convenience. Of course, if you run the program again it will generate a totally different set of output and place all of this data in a different, corresponding directory.

Once that process is complete, let's analyze the results, looking for unusual file accesses. Using the mactime utility, we can search by date for all accesses since that date. This is where having a timeframe of intrusion is important. Let's assume we were notified on December 31, 2004 that someone may have broken into the system. It is not clear how it occurred, or what they may have done. The command listing below is a simple search for MAC times since that date:

 [root@recovery target1]# ../../bin/mactime 12/31/2004  Dec 31 05 15:07:08  1160 m.. -rw-r--r-- root   root    /etc/ssh/sshd_config  Jan 05 05 09:48:28     4096 m.c drwx--- brett    brett    /home/brett Jan 05 05 23:50:00   197221 m.c -rw-r--r-- root   root     /var/log/sa/sa05 Jan 05 05 23:53:00   232830 mac -rw-r--r-- root   root     /var/log/sa/sar05                      197221 .a. -rw-r--r-- root   root     /var/log/sa/sa05 Jan 06 05 23:50:00   197221 m.c -rw-r--r-- root   root     /var/log/sa/sa06 Jan 06 05 23:53:00   197221 .a. -rw-r--r-- root   root     /var/log/sa/sa06                      232830 mac -rw-r--r-- root   root     /var/log/sa/sar06 

From the listing above, you can see that someone has accessed and modified the file /etc/ssh/sshd_config, which holds the Secure Shell server (daemon) configuration for this system. This would be immediate cause for concern. We want to see what was changed, and what purpose that change might have served . Did they modify this file so less restrictive authentication can be used with SSH loginsfor example, remote root loginswhich by default are not allowed on this server? On this server, we have SSH configured so you cannot login as root. Therefore, administrative users must use su or use the sudo command to execute root privileged commands, which is a fairly standard security practice on most UNIX systems these days.

This is a simple example, and further investigation would be warranted, including ascertaining how they modified this file, did they have root access, was it a user on the system, and which account might it have been, or was it through a vulnerability in a program we have running and aren't aware of (for example, a buffer overflow in a program that would have write access to this file). Details of these types of exploits are investigated further in Chapter 18.

Exploring the mactime a little further, this program has many options to help you refine your searches. These options are explained below:

• -time1 The start date to search for, where anything greater than it will be displayed.

• -time2 Optional data range, where dates between time1 and time2 will be displayed.

For example, to search for MAC times between December 30, 2004 and January 1, 2005 (inclusive), we would enter the following command:

 [root@recovery target1]$ ../../bin/mactime 12/30/2004-1/2/2005 Dec 30 04 04:08:16 559958 .a. -rw-r--r-- root     root /var/lib/tripwire/report/target1-20041230-040619.twr Dec 30 04 04:08:19 559958 m.c -rw-r--r-- root     root /var/lib/tripwire/report/target1-20041230-040619.twr Dec 31 04 04:07:07 559926 .a. -rw-r--r-- root     root /var/lib/tripwire/report/target1-20041231-040508.twr Dec 31 04 04:07:10 559926 m.c -rw-r--r-- root     root /var/lib/tripwire/report/target1-20041231-040508.twr Dec 31 04 14:51:10    48768 m. . -rw-rw-r-- root     utmp        /var/log/wtmp.1                       20753 m. . -rw--- - root      root       /var/log/messages.2 Jan 01 05 04:02:01    48768 . .c -rw-rw-r-- root      utmp       /var/log/wtmp.1 Jan 01 05 04:04:27    12434 m.. -rw-r--r-- root      root       /var/log/rpmpkgs.2 Jan 01 05 04:06:27   559918 .a. -rw-r--r-- root     root /var/lib/tripwire/report/target1-20050101-040431.twr Jan 01 05 04:06:30   559918 m.c -rw-r--r-- root     root /var/lib/tripwire/report/target1-20050101-040431.twr Jan 01 05 04:42:00        9 m.c -rw--- - root           root /var/spool/anacron/cron.monthly 

There are other options for using mactime that are helpful when using the program without the mactime database created using the grave-robber scripts, or for output of the data in formats that prepare the output for display in a web browser. Some of these options are

• -d < dir > Use a different directory for the body database.

• -b < file > Use a different body database file.

• -f < file > Flag files listed in this file with a different color helpful for printing reports .

• -h Print out HTML text for help with viewing in browserhelpful for printing reports.

• -p < file > Use an alternate password file for display of UIDs.

• -g < file > Use an alternate group file for display of GIDs.

• -S Flag SUID/SGID files in a different color (HTML only).

• -n Use normal date formatted dates instead of MM/DD/YYYY, for example, Thu Jan 13 11:17:56 MST 2005.

• -u < user > Print out files owned by user in a different color (HTML only).

• -y Print out date/times with year first to avoid European/U.S. data ambiguity, for example, YYYY/MM/DD instead of MM/DD/YYYY.

memdump

From the forensic analysis process proposed at the beginning of this chapter, there isn't much detail about investigating the system before placing it in the clean-room environment. This utitity, called memdump, created by Wietse Venema, can be useful for inspecting system memory on a target system before the system is brought offline and placed in single-user mode. What might you expect to find with memdump? With this utility, you can inspect bits from the operating system, programs that are running or have executed recently, and pieces of files that were accessed. This may be useful to see if a file was accessed, created, or modified and there isn't evidence of that anymorefor example, if it was deleted. This utility needs to be run soon after a system has been accessed (or compromised), and care must be taken with the output. The output should not be stored on the file system being analyzed , as modification of the file system will change system memory. The best way to use it is to redirect it to network socket, using the netcat utility, via the openssl client, or an established SSH tunnel.

Example commands to execute memdump and redirect its output are provided in the listing below:

 # memdump  nc host port # memdump  openssl s_client -connect host:port 

Remember, the data in memory may contain both readable and nonreadable binary information. When viewing this information, you must use tools that can handle this mixed data. The strings program is a useful program when looking for human-readable strings in a file. For example, the standard GNU version of grep (when it detects that the file is binary) by default produces a line like the following when searching for some content in binary files:

 # grep "Some String" /data/memory/target1.memdump Binary file /data/memory/target1.memdump matches 

This really isn't all that useful. What you want is either to use the strings program and then pipe () the output to grep, or tell grep that the input is ASCII or at least process it as if it were. Of course, remember, if you do this, you need to be careful with the output again as this may disrupt your terminal because binary output will be presented. A hex or binary file viewer may be needed at this point. Examples of these approaches are as follows:

 # strings /data/memory/target1.memdump  grep "PuTTY" SSH-2.0-PuTTY-Release-0.55 PuTTY6cPuTTYPuTTYPuTTYPuTTY6c6c6cPuTTY6c6cPuTTYPuTTY6cPuTTYPuTTY6c6c6c6cPuTTY6c6 c6cPuTTYPuTTYPuTTY6cPuTTYPuTTYPuTTYPuTTY ... # grep -a "PuTTY" /data/memory/target1.memdump  od -ax  more 0000000   ;   1   0 etx   -   *   .   6   ~   > stx   p bel   \ enq   C         313b 83b0 2aad 362e befe f002 dc07 c305 0000020   = nak  si   Y       {  K  nl  us   p      7 ack   w dc2   O   {         953d d98f fbfc 8a4b 701f 0637 1277 7bcf 0000040   1   #   /   _   W   a   z   g   ~   d      [ stx   /   M   ] dc3 ... # hexdump -C /data/memory/target1.memdump  more 00000740  00 00 00 00 00 00 00 00  72 20 6c 6f 61 64 69 6e  ........r loadin 00000750  67 20 6f 70 65 72 61 74  69 6e 67 20 73 79 73 74  g operating syst 00000760  65 6d 00 4d 69 73 73 69  6e 67 20 6f 70 65 72 61  em.Missing opera 00000770  74 69 6e 67 20 73 79 73  74 65 6d 00 00 00 00 00  ting system..... 00000780  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ................ ... 

tcpflow

Tcpflow is a TCP flow recorder, written by Jeremy Elson. Unlike programs like tcpdump, tcpflow is designed to capture data as part of a TCP connectionthe flowsand store the data in a format convenient for subsequent protocol analysis, or debugging the application protocol. Again, the same rules that applied for the memdump utility apply for tcpflow; however, since you are running tcpflow on the packet capture/logger system, not on the host itself, you don't have to worry about where you store the data. If you do decide to run tcpflow on the target itself, the data should not be stored on the local filesystem of the target host. It should be sent to a remote workstation via the nc, openssl client, or established SSH tunnel.

Commands to capture the traffic are similar to those that use the BPF driver, or libpcap. BPF syntax may be used to specify what data to capture and what to filter. For example, the following commands cause traffic captured on the target host to be written in tcpflow format to the standard output (console).

 capture: ~# tcpflow -c -r data/capture/target1.pcap 

The following is obviously a bit of a contrived example, as it isn't a real protocol, but hopefully it will show you the results of seeing the raw data along with the TCP sequence information. The binary information, of course, is more useful for protocol debugging. For forensic analysis, sometimes that is overkill.

 XXX.XXX.XXX.101.03283-192.168.XXX.050.05000: [ [c.4 ... ... ...>K. ._fh' ... .X. . [G. .y... I .x.t-... . .#... 192.168.XXX.050.05000-XXX.XXX.XXX.101.56554: c. .yd.'{.".s. .b) ... .' "... .Ac... .\Ex . .7R.V. .8o... .IG 192.168.XXX.002.00023-192.168.XXX.056.05000: are you there? ERROR: Cannot enable software for full application mode - no key specified. Please try again, or contact your sales representative, or visit http://www.example.com. You may also call [phone] for software licensing information.   Thank you. 

If tcpdump is used, or if packet capture is available, one program that can be extremely useful in a simulation environment is tcpreplay, along with its close sibling, flowreplay, both written by Aaron Turner. Tcpreplay is capable of replaying a BPF-style packet capture file to send packets at the same targets as seen in the capture. Alternatively, the targets may be changed using command line options, including network address translation (NAT) features. Complete explanation of how to use these tools is beyond this scope of this chapter; however, there are many options and a lot of uses, especially in a clean lab environment where you need to prove how someone broke in. The following is an example of re-executing a capture file against a test host with the same vulnerability to prove that it did in fact have the vulnerability, and was exploited in the way the packet capture would indicate .

 dev: ~# tcpreplay -i eth0 -F -r .5 /data/capture/target1.pcap 

From the example above, you can see that traffic captured from tcpdump can be replayed via tcpreplay utility.

One minor augmentation to this process is to replay the client portion of the connections only, instead of both sides, as would be interesting in a simulated network environment. To achieve this, the flowreplay command is used. Flowreplay operates in a similar way to tcpreplay, except only the client side of the connections is replayedthe server has to respond appropriately for this type of tool to work properly. An example output of this process is listed here:

 dev: ~# flowreplay -i /data/capture/target1.pcap -t <lab_ip> -p tcp/5000 -V 

For safety purposes, this should also be done in a lab environment, and not against the production server. These tools provide an excellent means of testing captured traffic against firewall and IDS configurations to ascertain why a particular flow was or, as is often the case, was not filtered.