10.3 Forensic Tracks

Now that you are reasonably sure ^[2] that there are no traces of your attack in the logfiles, it is time to take concealment to the next level.

^[2] Reasonably sure implies that the level of effort you apply to hiding exceeds the effort (and investment) the investigators are willing and able to make to find you.

10.3.1 File Traces

Even if you are sure that the OS audit trail is clear, the shell histories, source files, and logfiles you erased and even your keystrokes might hide in many places on the system. The vigor with which you pursue these traces depends on what's at stake as well as the skill of your adversaries. Uncovering erased data is simple on Windows and only slightly more difficult on Unix filesystems. However, you can be sure that there is always a chance that a file subjected to the wrath of /bin/rm will come to life again (as a zombie). The research (such as the famous paper "Secure Deletion of Data from Magnetic and Solid-State Memory," by Peter Gutmann) indicates that there is always a chance that data can be recovered, even if it has been overwritten many times. Many tools are written to "securely erase" or "wipe" the data from a hard drive, but nothing is flawless. However, these tools have a chance of foiling a forensics investigation. In fact, there are even tools "marketed" (in the underground ) as antiforensics. An example is the notorious Defiler's Toolkit, described in Phrack #59 (file #0x06, "Defeating Forensic Analysis on Unix"). It's rarely used and is usually overkill, but the kit demonstrates that advanced hackers may easily make forensics investigation onerous or even impossible . In fact, the author of the paper laments the poor state of computer forensics and the lack of advanced data discovery tools.

One of the main issues with secure deletion of data is that the filesystem works against the attacking side (which attempts to hide or remove data) and the defending side (which seeks to uncover the evidence). Often, Unix filesystems overwrite the drive area where the removed files were located (this is especially likely to happen to logfiles). On the other hand, the filesystem has an eerie tendency to keep bits and pieces of files where they can be found (swap, /tmp area, etc.). Overall, reliably removing everything beyond recovery is just as difficult as reliably recovering everything.

There are a lot of Unix tools that claim to reliably erase data. However, many of them use operating system disk-access methods that tend to change, since OS authors do not have to be concerned about preserving low-level access to the disk ”it goes unused by most applications. Such changes have a good chance of rendering a wiping tool ineffective . Thus, unlike other application software, a wiping tool that performs just fine on Red Hat Linux 7.1 might stop working for 7.2.

The simpler, more reliable way of erasing all host traces (without destroying the drive) requires your presence at the console. For example, the autoclave bootable floppy system (http://staff.washington.edu/jdlarios/autoclave/) allows you to remove all traces of data from the IDE hard disk (SCSI is not supported). In fact, it removes all traces of just about everything and leaves the disk completely filled with zeros or random patterns.

Unlike the programs that run from a regular Unix shell (such as many incarnations of wipe and shred ), autoclave has its own Linux kernel and wiping utility that ensures erased means gone . In this case, you can be sure the filesystem or OS does not play any tricks by inadvertently stashing bits of data somewhere. However, autoclave is not useful for remote attackers , since inserting a floppy into the machine might be problematic and removing everything with 38 specially crafted character passes , while extremely (in all senses extremely ) effective, might bring attention to an otherwise inconspicuous incident. The process is also painfully slow and might take days for a reasonably large hard drive. A single "zero out" pass takes at least 3 hours on a 20-GB drive with modern disk controllers. Many similar mini-OS bundles exist for reliably cleaning the disks.

Thus, in real life, under time pressure, you must rely on application-level deletion tools that use whatever disk access methods the OS provides and sometimes miss data. Even the best wiping tools (including those with their own kernels , such as autoclave) are not guaranteed against novel and clandestine forensics approaches that involve expensive custom hardware.

Here is an example of using GNU shred, the secure deletion utility that became standard on many Linux and *BSD distributions:

 #  shred -zu ~/.bash_history  

This command erases the above shell history file with 25 overwrite cycles, inspired by Gutmann's paper. Or, rather, it tries to erase the file. However, the user will likely have no idea whether it was erased or not. Many things can get in the way: filesystem code, caches, and so on. While the tool authors do take care to make sure that the erased bits are really erased, many factors beyond their control can intervene. For example, even if shred works for you with the ext2 filesystem on Linux, you still need to test it to know whether it works on ext3 or ReiserFS. As pointed out by one wiping tool's author (http://wipe. sourceforge .net), "if you're using LFS ^[3] or something like it, the only way to wipe the file's previous contents (from userspace) is to wipe the whole partition..."

^[3] For information on LinLogFS, see http://www.complang.tuwien.ac.at/czezatke/lfs.html.

You can test the behavior of your wiping tool on your specific system with the following sequence of commands. They check whether the tool actually wipes the data off the floppy disk:

  # mkfs -t ext2 /dev/fd0  

Create a fresh Linux ext2 filesystem on a floppy disk.

  # mount /mnt/floppy  

Mount the floppy to make the created filesystem available.

  # dd if=/dev/zero of=/mnt/floppy/oooo ; sync ; /bin/rm /mnt/floppy/oooo ; sync  

Zero the disk using the dd command in order to remove prior data.

  # echo "some data" > /mnt/floppy/TEST  

Create a test file.

  # sync  

Make sure the file is in fact written to the disk.

  # strings /dev/fd0  grep data  

Confirm that the data is indeed written to disk.

  # shred -vuz /mnt/floppy/TEST  

Remove the file using (in this case) the GNU shred utility.

  # umount /mnt/floppy  

Unmount the filesystem to make absolutely sure the file is indeed wiped.

  # strings /dev/fd0  grep data  

Try to look for the file data on disk (should fail ”i.e., nothing should be seen).

You should see nothing in response to the last command. If you see some data, the secure wipe utility fails the test. The GNU shred utility passes it just fine. However, the test is not conclusive, since the floppy often has a different filesystem from the hard drive; thus, the tool might not pass the test for the real hard drive. Additionally, sometimes the drive hardware plays its own games and doesn't actually write the data, even if synced. In this case, the data might be retained in the drive's internal memory.

In many cases, even makeshift solutions such as this will help. Suppose you are erasing the file .bash_history from the directory /home/user1 . The following commands attempt to make recovery problematic:

 #  /bin/rm ~user1/.bash_history  #  cat /dev/zero > /home/user1/big_file  (until file system overflows and "cat" command exits) # sync #  /bin/rm /home/user1/big_file  

The Unix dd command may be used in place of cat , as in the floppy example above.

The trick is to remove the file and then make the system allocate all the disk space on the same partition for big_file with zeros, just as in our floppy test above. Even though the sync command is supposed to copy all the memory buffers to disk, the operation has a chance of not working due to caches, buffers, and various filesystem and drive firmware idiosyncrasies.

These steps make it more difficult to recover erased data. It makes sense to deal similarly with swap, which can contain pieces of your "secret" data. The procedure to do this for a Linux swap partition (swap can also be a file, which makes cleaning it easier) is straightforward. It involves disabling swap, usually with swapoff , and then writing data (such as zeros or special characters ) to a raw partition starting from a swap file header. The Sswap utility from the THC secure_delete kit automates the process ”except that turning off swap should be done manually. The utility handles Linux swap files by default and might be able to clean other Unix swap files.

Placing the data on a disk to specifically foil forensic tools sounds like overkill for almost any attack. However, the methods to do so are available (see, for example, "Defeating Forensic Analysis on Unix" in Section 10.5). Certain tools can clean up filesystem data that is used by forensic tools to uncover evidence. A good example is cleaning inodes data on the ext2 Linux filesystem ”this data is used by forensic tools (such as TCT and TASK) to find deleted files.

In some cases, even the hardware might revolt against the attacker. Certain disk controllers combine the write operations, thus decreasing the number of passes applied. Basically, the disk drive controller firmware sees that you are trying to write zeros, say, five times, and will just write them once, assuming that is what you want. Similarly, the OS built-in sync command might have an affect on the drive's built-in memory cache, thus also thwarting attempts to wipe the data.

10.3.2 Timestamps

Another critical forensics trace, and one that will always be left on the system, is timestamps. Consumer operating systems such as Windows 9x/Me keep track of changes to files by adjusting the file timestamp; i.e., the modification time. Other OSs record much more.

Most Unix filesystems record not only when the file was changed (change time, or ctime ) and when its properties (such as permissions) were changed (modified time, or mtime ), but when the file was last accessed for reading (access time, or atime ). Together, these timestamps are referred to as MAC times (Modify-Access-Change times).

Here is how Linux ext2 stores the times for each inode (filesystem unit in ext2):

 struct ext2_inode { ...other fields...         _  _u32        i_atime;        /* access time - reading */         _  _u32        i_ctime;        /* change time - permissions  */         _  _u32        i_mtime;   /* modification time - contents */         _  _u32        i_dtime;        /* deletion time - or 0 for non-deleted files*/ ...other fields... 

For each inode, four times are stored by the filesystem as 32-bit unsigned integers.

Here is an example excerpt from the MAC-robber tool (by Brian Carrier; see http://www.sleuthkit.org/mac-robber/desc.php), which collects all such timestamps from Unix files. The first line shows the format of the file (MAC times are in bold).

 md5filest_devst_inost_modest_lsst_nlinkst_uidst_gidst_rdevst_size  st_ atimest_mtimest_ctime  st_blksizest_blocks 0/usr/local/bin7694816716877drwxr-xr- x20056324096  105791175310509355761050935576  40968 0/usr/local/bin/a2p7694843533261-rwxr-xr- x1002816107759  010188883131050509378  4096224 0/usr/local/bin/argusarchive7694843733261-rwxr-xr- x10028163214  105791071510228481351050509378  40968 0/usr/local/bin/argusbug7694843833133-r-xr-xr- x10028169328  105791071510228481351050509378  409624 0/usr/local/bin/c2ph7694843933261-rwxr-xr- x200281636365  010188883131050509379  409672 

The timestamps, such as "1050935576", show as numbers of seconds since January 1970, the standard time notation on Unix systems ("Unix epoch time"). The above number actually stands for "Monday, April 21, 2003 2:32:56".

Many conversion tools are available (e.g., http://dan.drydog.com/unixdatetime.html or http://www.onlineconversion.com/unix_time.htm). A Google query for "1970 Unix time convert" provides numerous examples.

The critical issue of timestamps is that collecting them on a running filesystem changes the atime, since the file has to be accessed in order to check the timestamp. That is exactly the reason why forensics manuals recommend working with a read-only copy of the evidence.

For any running program under Unix, many libraries and system files are usually called. Thus, a running program leaves a wake of running waves of changing atimes. Such changes may be detected . Obviously, the changed files will have their ctimes reset as well.

10.3.2.1 Countermeasures

There are two main methods to try to stop these information leaks about your activities on a system. One is to remount the filesystem in such a way that no atime timestamps are collected. It may be accomplished under Linux using the command:

 #  mount -o noatime, remount /dev/hda1 /usr  

This prevents the atime analysis, while doing nothing to ctime and mtime changes. Even more effective is mounting the filesystem as read-only, as follows :

 #  mount -o ro, remount /dev/hda1 /usr  

This effectively prevents all timestamp changes, but it might be impractical if changes to the partition are needed.

Timestamps in Unix can also be changed manually using the touch command; e.g., touch -a /tmp/test changes the atime of a file /tmp/test , while touch -m /tmp/test affects the mtime. The command may also be used to set the time needed on a file and to copy the timestamp from a different file. touch is an effective tool to influence time stamps. Just keep in mind that running the touch command creates the usual atime wake.

Yet another method is to go ahead and access all the files, so that all timestamps are changed. This can be done via the touch command or other means. For example, you can loop through all the files to touch them and thus distort all accessible timestamps, so that forensic investigators see all files as modified.

Going to such lengths to thwart host forensics might be futile if the data resides on network devices or other machines. Network devices (such as routers) and security devices (firewalls, IDSs) might still remember you and remain out of your reach.