| < Day Day Up > |
|
Similar to a Windows machine, a Unix machine that falls victim to an incident can be examined using a live response technique. The live response allows you to obtain volatile data (such as contents of RAM) for incidents that occurred before the machine was powered down for a forensic duplication. (In fact, your company may not allow you to shut down this machine, which is required to perform a forensic duplication.) A live response will allow you to perform a sound investigation and remediate the attack situation. This section discusses the tools used in a successful live response and ties it together with a real case study.
All the tools mentioned in this chapter are combined in a trusted live response tool kit that you copy to your live response media. Because all the command-line examples in this chapter assume that your current directory is the live response media (the CD-ROM or floppy that contains your tool kit), they will be prepended with the ./path. The ./path will run the tool from the current directory and eliminate the chance of your running a tool of the same name in the user’s path (from the untrusted victim system).
To conduct live response activity, you must be logged in as root. Most of these commands cannot produce output unless you have root access to the objects they were designed to analyze.
The output of all commands run in these responses will be delivered to a destination workstation for storage and analysis. You do not want to write the information to the local victim’s hard drive because it could destroy potential evidence. The process of transmitting the information across the network can be performed with Netcat (or Cryptcat), just as was done earlier in this chapter (in the “Windows Live Response Tool Kit” section).
Unix works differently from Windows in that you cannot just copy the associated DLL files to the response media when dynamic library calls are being made. Instead, because most of the tools are open source (that is, the source code is available to you), you must recompile them statically. Because the explanation of a static compilation for the tools in this chapter is beyond the scope of this book, it will not be iterated here. A basic rule of thumb, however, is to change the makefile so that the lines CFLAGS or LDFLAGS contain the token -static. With some packages, you must run the configure script to generate the makefile first. If you cannot compile a static version, you must copy all of the DLLs to the CD-ROM and change the environment variable LD_LIBRARY_PATH to where the CD will be mounted during the response (typically something like /mnt/cdrom). You can determine which libraries are needed for an executable by typing the following:
forensic# ldd /usr/local/sbin/lsof libkvm.so.2 = /usr/lib/libkvm.so.2 (0x2807d000) libc.so.4 = /usr/lib/libc.so.4 (0x28083000)
This command will show that the lsof command needs libkvm.so.2 and libc.so.4 to run if we do not recompile it statically.
Imagine for a minute that you are an attacker and you want to maintain access to a system you have just fully compromised. You want to modify some of the system commands, such as the command shell, to hide your presence. If you can modify a command shell to do what you need, you could place it on the compromised system.
Now, switch your viewpoint back to the investigator. As a responder, you want to make sure that you are not executing this nefarious shell; instead, you want to execute a shell you compiled, so you know it’s trusted. The shells are usually found in the /bin directory on most Unix systems.
Type the following command to execute a trusted shell:
victim# ./bash
Note | The bash shell is one of the author’s favorites, but sh, tcsh, csh, or another shell can be used as an alternative as long as they are copied from a trusted system first. |
The netstat command on a Unix system is similar to the Windows command. It will list all the network connections and listening TCP/UDP ports on the system. This tool provides data that will be useful in tracking down backdoors and the endpoints of network connections associated with the victim system. Netstat is typically found in the /bin or /usr/bin directory, depending on the type of Unix you’re using.
The following command is used in the live response. Notice how it is exactly the same as its Windows counterpart:
victim# ./netstat -an
The following output is the result of the netstat command. It is assumed that the compromised system has an IP address of 192.168.1.104.
Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:4375 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:98 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:79 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:513 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:514 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:23 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:515 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:113 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:1024 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN udp 0 0 0.0.0.0:518 0.0.0.0:* udp 0 0 0.0.0.0:517 0.0.0.0:* udp 0 0 0.0.0.0:513 0.0.0.0:* udp 0 0 0.0.0.0:1026 0.0.0.0:* udp 0 0 0.0.0.0:1025 0.0.0.0:* udp 0 0 0.0.0.0:704 0.0.0.0:* udp 0 0 0.0.0.0:689 0.0.0.0:* udp 0 0 0.0.0.0:1024 0.0.0.0:* udp 0 0 0.0.0.0:111 0.0.0.0:* raw 0 0 0.0.0.0:1 0.0.0.0:* 7 raw 0 0 0.0.0.0:6 0.0.0.0:* 7 Active UNIX domain sockets (servers and established) Proto RefCnt Flags Type State I-Node Path unix 0 [ ACC ] STREAM LISTENING 517 /dev/printer unix 7 [ ] DGRAM 422 /dev/log unix 0 [ ACC ] STREAM LISTENING 682 /tmp/.font-unix/fs-1 unix 0 [ ACC ] STREAM LISTENING 652 /dev/gpmctl unix 0 [ ] STREAM CONNECTED 169 @00000014 unix 0 [ ] DGRAM 853 unix 0 [ ] DGRAM 720 unix 0 [ ] DGRAM 685 unix 0 [ ] DGRAM 636 unix 0 [ ] DGRAM 511 unix 0 [ ] DGRAM 446 unix 0 [ ] DGRAM 434
We see that port 4375 is open and listening for connections. We know that this port was not open earlier (because we are the system administrators) and therefore we should investigate this further! No other ports in netstat’s results require our attention.
The version of netstat with Linux, specifically, allows a -p switch, which will map the listening ports to the binary files on the disk that opened them. The command looks like this:
victim# ./netstat -anp
Because most flavors of Unix do not support the -p flag, we choose not to use it. Instead, we use the tool lsof (in an upcoming section) to take care of the mapping between open ports and their parent processes.
Tip | If you are interested in acquiring the host’s routing table, you can do so with the -r switch to netstat. |
The Address Resolution Protocol (ARP) table maps the physical machine—the Media Access Control (MAC)—addresses of the Ethernet cards to the associated IP addresses in the subnet. Because most networks do not secure the local subnet by binding a specific MAC address to an IP address using switches, anyone can modify his or her ARP table or IP address and cause havoc. This occurs, for example, when one employee masquerades as another on the internal network. By using the arp command, you can see within the last few minutes which MAC address was mapped to which IP address, and this may help you to track down a rogue user.
The ARP program is typically located in the /sbin or /usr/sbin directory, depending on the version of Unix you are using.
To see the ARP table of the victim machine, the command is very similar to that used in the Windows live response:
victim# ./arp -an
The ARP table returned from the victim system is as follows:
? (192.168.1.1) at 00:BD:81:43:07:03 [ether] on eth0
The results show that the machine at IP address 192.168.1.1 has a MAC address of 00:BD:81:43:07:03. This additional piece of information helps us in the case of tracking down 192.168.1.1 on our network when we do not regulate the IP addresses. We could examine every machine until we found the MAC address of 00:BD:81:43:07:03.
Caution | A user with sufficient privileges can change his or her own MAC (and IP) address in many operating systems. It is possible to do this with a Windows or Unix machine. |
The ls command in Unix is similar to the dir command in the Windows live response. We can use it to collect the last accessed and the last modified times of the files on the system. It is a good idea to run this command so that none of the timestamps are lost on the system if a mistake was made during the live response. The ls command is usually located in the /bin directory.
Tip | For the same reasons as we cited in the “Windows Live Response Tool Kit” section, run your ls command to acquire the time and date stamps at the beginning of your response. |
To collect the last accessed times, execute the following command:
victim# ls -alR --time=atime /
The relevant files returned from this command are observed in the following fragments. First, we look at the files in the /etc directory:
-rw-r--r-- 1 root root 7470 Mar 21 06:32 mime.types -rw-r--r-- 1 root root 1048 Mar 7 2000 minicom.users -rw-r--r-- 1 root kjohnson 196 Mar 22 00:17 motd -rw-r--r-- 1 root root 90 Mar 22 00:23 mtab -rw-r--r-- 1 root root 1925 Feb 9 2000 mtools.conf
Then we view the files in the /kjohnson home directory:
/home/kjohnson: total 240 drwx------ 2 root kjohnson 4096 Mar 22 00:30 . drwxr-xr-x 7 root root 4096 Mar 22 00:30 .. -rw------- 1 root kjohnson 216 Mar 22 00:18 .bash_history -rw-r--r-- 1 root kjohnson 24 Mar 22 00:18 .bash_logout -rw-r--r-- 1 root kjohnson 230 Mar 21 23:40 .bash_profile -rw-r--r-- 1 root kjohnson 124 Mar 21 23:40 .bashrc -rw-r--r-- 1 root kjohnson 3394 Mar 21 23:39 .screenrc -rwxr-xr-x 1 root kjohnson 210096 Mar 22 00:13 1
To collect the last modification time, we execute the following command:
victim# ls -alR --time=mtime /
If this command did not work, it may be that modified times are already displayed by default (this is the case in Linux).
If the last modified times are shown by default, you could use the following command instead:
victim# ls -alR /
The following relevant files were observed from this command’s results. First, the suspicious files in the /etc directory:
-rw-r--r-- 1 root root 7470 Mar 21 06:32 mime.types -rw-r--r-- 1 root root 1048 Mar 7 2000 minicom.users -rw-r--r-- 1 root kjohnson 196 Mar 22 00:17 motd -rw-r--r-- 1 root root 90 Mar 22 00:23 mtab -rw-r--r-- 1 root root 1925 Feb 9 2000 mtools.conf
Then we view the /kjohnson home directory:
/home/kjohnson: total 240 drwx------ 2 root kjohnson 4096 Mar 22 00:18 . drwxr-xr-x 7 root root 4096 Mar 21 23:39 .. -rw------- 1 root kjohnson 216 Mar 22 00:18 .bash_history -rw-r--r-- 1 root kjohnson 24 Mar 21 23:39 .bash_logout -rw-r--r-- 1 root kjohnson 230 Mar 21 23:39 .bash_profile -rw-r--r-- 1 root kjohnson 124 Mar 21 23:39 .bashrc -rw-r--r-- 1 root kjohnson 3394 Mar 21 23:39 .screenrc -rwxr-xr-x 1 root kjohnson 210096 Mar 21 23:43 1
To collect when any of the information within the inode (the data structure containing file permissions, where on the disk the rest of the file can be found, and time and date stamps) of a file has changed, the following command is executed:
victim# ls -alR --time=ctime /
This command is not much different than the “last modified” command we examined at the beginning of this section. The only time this command would be different is if a file’s properties would be changed without the contents of the file itself changing.
In the /kjohnson home directory, we see a suspicious file named 1. It may not mean much to us now, but it will play an important role later in the chapter in the Case Study at the end of the “Unix Live Response Tool Kit” section.
Note | ctime is often confused with "Creation Time." Be sure to keep it straight when analyzing a Unix system. |
Similar to the loggedon command we used with Windows, a command called w exists for Unix. This command (discussed further in Chapter 5) displays all the currently logged-on users and their originating IP addresses. This command is useful for examining unauthorized use of accounts on a Unix system.
The w command is located in the /usr/bin directory.
To use w, type the following command:
victim# w
The output from w on our victim machine looks like this:
12:24am up 1:38, 1 user, load average: 0.02, 0.02, 0.00 USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT root tty1 - 10:44pm 0.00s 0.71s 0.03s w
Because we are the root user and logged in from the console (indicated by the - character in the FROM column), we do not see any suspicious activity.
Caution | Even though we do not see any unauthorized accounts in the output generated by w, we can’t be sure that an attacker is not currently on the system. A user must have completed the valid login process and the logs must be intact on the system for their information to show up in the output. A backdoor does not typically call the login facility and therefore would not show up here. |
To view the last logins for users on a particular Unix system, the last and lastb commands are available. The last and lastb commands display the last successful and failed logins, respectively. The last command helps generate evidence in the case of an unauthorized use of an account on our system, while the lastb command may show evidence of a brute-force attack against a machine.
The last and lastb commands are typically located in the /usr/bin directory.
Note | Most versions of Unix do not initiate the lastb facility by default. In Linux, in particular, the file /var/log/btmp must have been created with the touch command in order for lastb to work. |
In a response scenario, you will want to dump the login attempts for all the users on the system. This is accomplished with the following command (for the lastb command, change last to lastb):
victim# last
The command outputted the following results from our victim machine:
mpepe pts/0 192.168.1.1 Thu Mar 21 23:37 - 00:24 (00:46) root tty1 Thu Mar 21 22:44 still logged in reboot system boot 2.2.14-5.0 Fri Mar 22 03:42 (-3:-6) root tty1 Fri Mar 22 03:39 - down (00:00) reboot system boot 2.2.14-5.0 Fri Mar 22 03:09 (00:30) reboot system boot 2.2.14-5.0 Thu Mar 21 09:04 (18:36) wtmp begins Thu Mar 21 09:04:17 2002
As we can see, the user kjohnson has not been observed logging in. We could draw a couple of different conclusions: either the logs were tampered with by the attacker or the attacker also compromised the mpepe account and switched users, using the su command, to the kjohnson account.
Because most flavors of Unix systems do not offer a version of netstat that supports the -p flag, we use a tool called lsof to map the network sockets open to executables on the file system. In this respect, lsof is similar to fport from the “Windows Live Response Tool Kit” section. Additionally, lsof will show us all the open files on the system. Lsof is freely available to the public and has been ported to nearly all the Unix flavors. Although lsof is a tool with many options useful to the general system administrator, this chapter discusses only those options useful to a live response scenario. If you are interested in using other options, check out the lsof man page, which provides a great discussion of the tool.
Lsof has been repackaged by many Unix OS vendors and can be found anywhere. The “official” home of lsof is on the maintainer’s web site: http://www-rcd.cc.purdue.edu/~abe/.
Use the following command to list all the open sockets and files on the system:
victim# ./lsof -n
The -n option in the command tells lsof not to perform the DNS reverse lookup for any IP addresses listed within the results. For response purposes, we tend not to rely on the fully qualified domain names because they can change; rather, we use the actual IP addresses. The relevant output to the command is shown here:
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME inetd 721 root cwd DIR 3,2 4096 2 / inetd 721 root rtd DIR 3,2 4096 2 / inetd 721 root txt REG 3,2 21552 35319 /usr/sbin/inetd inetd 721 root mem REG 3,2 340663 146606 /lib/ld-2.1.3.so inetd 721 root mem REG 3,2 4101324 146613 /lib/libc-2.1.3.so inetd 721 root mem REG 3,2 246652 146644 /lib/libnss_files-2.1.3.so inetd 721 root 0u CHR 1,3 65387 /dev/null inetd 721 root 1u CHR 1,3 65387 /dev/null inetd 721 root 2u CHR 1,3 65387 /dev/null inetd 721 root 3u IPv4 745 TCP *:39168 (LISTEN) inetd 721 root 4u IPv4 746 TCP 192.168.1.104:39168-192.168.1.1:2028 (CLOSE_WAIT) inetd 721 root 5u unix 0xc2a99980 853 socket inetd 721 root 6u IPv4 748 TCP *:ftp (LISTEN) inetd 721 root 7u IPv4 749 TCP *:telnet (LISTEN) inetd 721 root 8u IPv4 750 TCP *:shell (LISTEN) inetd 721 root 9u IPv4 751 TCP *:login (LISTEN) inetd 721 root 10u IPv4 752 UDP *:talk inetd 721 root 11u IPv4 753 UDP *:ntalk inetd 721 root 12u IPv4 754 TCP *:finger (LISTEN) inetd 721 root 13u IPv4 755 TCP *:linuxconf (LISTEN) inetd 721 root 14u IPv4 756 TCP *:4375 (LISTEN) 1 881 root cwd DIR 3,2 4096 83456 /home/kjohnson 1 881 root rtd DIR 3,2 4096 2 / 1 881 root txt REG 3,2 210096 83461 /home/kjohnson/1 1 881 root mem REG 3,2 340663 146606 /lib/ld-2.1.3.so 1 881 root mem REG 3,2 4101324 146613 /lib/libc-2.1.3.so 1 881 root mem REG 3,2 246652 146644 /lib/ /libnss_files-2.1.3.so 1 881 root 0u CHR 136,0 2 /dev/pts/0 1 881 root 1u CHR 136,0 2 /dev/pts/0 1 881 root 2u CHR 136,0 2 /dev/pts/0 1 881 root 3u sock 0,0 954 can't identify protocol 1 881 root 4w REG 3,2 36864 35934 /tmp/.net
We can see that the inetd opened the TCP 4375 port. Therefore, we need to investigate the /etc/inetd.conf file. Furthermore, we see the executable 1 opens a file named /tmp/.net, and this file will also have to be examined. The 1 executable also opens a raw socket, as seen in this line:
1 881 root 3u sock 0,0 954 can't identify protocol
If we ascertain that this executable opens a raw socket and a regular file, we can assume that 1 may be a sniffer program (sniffers are discussed in Chapter 16).
To obtain a listing of the currently running processes on the system, the ps command is used. This command is similar to the Pslist program we discussed in the “Windows Live Response Tool Kit” section. We use the ps command to view rogue processes, such as sniffers, backdoors, distributed denial-of-service (DDoS) zombies, and password crackers, running on the victim machine.
The ps command is typically located in the /bin directory.
When we collect the process information, we want to see all the processes currently executing on the system and which user ran them. This can be accomplished using the following command:
victim# ./ps -aux
The process list from our victim machine is shown here. Notice that we can see the start times of each process, which may indicate which processes were run shortly after the system was compromised.
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.1 0.7 1120 476 ? S Mar21 0:06 init [3] root 2 0.0 0.0 0 0 ? SW Mar21 0:00 [kflushd] root 3 0.0 0.0 0 0 ? SW Mar21 0:01 [kupdate] root 4 0.0 0.0 0 0 ? SW Mar21 0:00 [kpiod] root 5 0.0 0.0 0 0 ? SW Mar21 0:00 [kswapd] root 6 0.0 0.0 0 0 ? SW Mar21 0:00 [mdrecoveryd] bin 319 0.0 0.7 1212 496 ? S Mar21 0:00 portmap root 334 0.0 0.0 0 0 ? SW Mar21 0:00 [lockd] root 335 0.0 0.0 0 0 ? SW Mar21 0:00 [rpciod] root 358 0.0 0.7 1104 480 ? S Mar21 0:00 /usr/sbin/apmd -p root 409 0.0 0.8 1172 552 ? S Mar21 0:00 syslogd -m 0 root 418 0.0 1.2 1440 768 ? S Mar21 0:00 klogd nobody 432 0.0 0.9 1292 628 ? S Mar21 0:00 identd -e -o nobody 435 0.0 0.9 1292 628 ? S Mar21 0:00 identd -e -o nobody 436 0.0 0.9 1292 628 ? S Mar21 0:00 identd -e -o nobody 438 0.0 0.9 1292 628 ? S Mar21 0:00 identd -e -o nobody 439 0.0 0.9 1292 628 ? S Mar21 0:00 identd -e -o daemon 450 0.0 0.7 1144 496 ? S Mar21 0:00 /usr/sbin/atd root 464 0.0 0.9 1328 620 ? S Mar21 0:00 crond root 496 0.0 0.8 1204 532 ? S Mar21 0:00 lpd root 510 0.0 0.8 1156 532 ? S Mar21 0:00 rpc.rstatd root 526 0.0 0.6 1140 408 ? S Mar21 0:00 rpc.rusersd nobody 540 0.0 0.9 1316 612 ? S Mar21 0:00 rpc.rwalld root 554 0.0 0.8 1132 552 ? S Mar21 0:00 rwhod root 598 0.0 1.7 2128 1124 ? S Mar21 0:00 sendmail: accepti root 613 0.0 0.7 1144 456 ? S Mar21 0:00 gpm -t ps/2 xfs 647 0.0 1.2 1728 808 ? S Mar21 0:00 xfs -droppriv -da root 685 0.0 1.6 2224 1040 tty1 S Mar21 0:00 login -- root root 686 0.0 0.6 1092 408 tty2 S Mar21 0:00 /sbin/mingetty tt root 687 0.0 0.6 1092 408 tty3 S Mar21 0:00 /sbin/mingetty tt root 688 0.0 0.6 1092 408 tty4 S Mar21 0:00 /sbin/mingetty tt root 689 0.0 0.6 1092 408 tty5 S Mar21 0:00 /sbin/mingetty tt root 690 0.0 0.6 1092 408 tty6 S Mar21 0:00 /sbin/mingetty tt root 693 0.0 1.5 1716 976 tty1 S Mar21 0:00 -bash root 721 0.0 0.8 1156 520 ? S Mar21 0:00 /usr/sbin/inetd root 881 0.0 1.2 1964 776 ? S 00:14 0:00 ./1 -s 65535 -n - root 975 0.0 1.1 2332 700 tty1 R 00:34 0:00 ps aux
If we examine the process resulting from the command 1, we see it is running as process ID 721. The process was executed by root and was started at 00:14 the same day. Therefore, if we know that kjohnson, with the 1 binary, was an unauthorized user on the system and this process is running as root, the attacker probably has root privileges.
The examples in this book show a format of output that is useful to the responder in most situations. However, the ps command can output in a variety of formats. The format from the first ps command (ps -aux) truncates some of the enumerated process’s command-line syntax. To solve this truncation problem, we can reformat the output of the ps command as follows:
victim# ./ps -axo <var1>,<var2>,…
The <var> tokens could be any one of the parameters listed in the following table. The Code column contains the code you will input as <var1>, <var2>, and so on; the Header column contains the name of the header seen in the output from the ps command. With these codes, nearly any imaginable aspect from the process table can be enumerated.
Code | Header |
---|---|
%cpu | %CPU |
%mem | %MEM |
alarm | ALARM |
args | COMMAND |
blocked | BLOCKED |
bsdstart | START |
bsdtime | TIME |
c | C |
caught | CAUGHT |
cmd | CMD |
comm | COMMAND |
command | COMMAND |
cputime | TIME |
drs | DRS |
dsiz | DSIZ |
egid | EGID |
egroup | EGROUP |
eip | EIP |
esp | ESP |
etime | ELAPSED |
euid | EUID |
euser | EUSER |
f | F |
fgid | FGID |
fgroup | FGROUP |
flag | F |
flags | F |
fname | COMMAND |
fsgid | FSGID |
fsgroup | FSGROUP |
fsuid | FSUID |
fsuser | FSUSER |
fuid | FUID |
fuser | FUSER |
gid | GID |
group | GROUP |
ignored | IGNORED |
intpri | PRI |
lim | LIM |
longtname | TTY |
lstart | STARTED |
m_drs | DRS |
m_trs | TRS |
maj_flt | MAJFL |
majflt | MAJFLT |
min_flt | MINFL |
minflt | MINFLT |
ni | NI |
nice | NI |
nwchan | WCHAN |
opri | PRI |
pagein | PAGEIN |
pcpu | %CPU |
pending | PENDING |
pgid | PGID |
pgrp | PGRP |
pid | PID |
pmem | %MEM |
ppid | PPID |
pri | PRI |
rgid | RGID |
rgroup | RGROUP |
rss | RSS |
rssize | RSS |
rsz | RSZ |
ruid | RUID |
ruser | RUSER |
s | S |
sess | SESS |
session | SESS |
sgi_p | P |
sgi_rss | RSS |
sgid | SGID |
sgroup | SGROUP |
sid | SID |
sig | PENDING |
sig_block | BLOCKED |
sig_catch | CATCHED |
sig_ignore | IGNORED |
sig_pend | SIGNAL |
sigcatch | CAUGHT |
sigignore | IGNORED |
sigmask | BLOCKED |
stackp | STACKP |
start | STARTED |
start_stack | STACKP |
start_time | START |
stat | STAT |
state | S |
stime | STIME |
suid | SUID |
suser | SUSER |
svgid | SVGID |
svgroup | SVGROUP |
svuid | SVUID |
svuser | SVUSER |
sz | SZ |
time | TIME |
timeout | TMOUT |
tmout | TMOUT |
tname | TTY |
tpgid | TPGID |
trs | TRS |
trss | TRSS |
tsiz | TSIZ |
tt | TT |
tty | TT |
tty4 | TTY |
tty8 | TTY |
ucomm | COMMAND |
uid | UID |
uid_hack | UID |
uname | USER |
user | USER |
vsize | VSZ |
vsz | VSZ |
wchan | WCHAN |
While many of these fields may not be of interest to you, you may find some of them helpful. For instance, if you desired to view only the process ID (PID), the user who created the process, the start time, and the full command line, the following command will work nicely:
victim# ./ps -axo pid,uid,start,command
This command line would list more of the command-line syntax of the enumerated processes than the ps -aux version would.
If we are asked by the “powers that be” to remediate a situation immediately, we might choose to kill the offending process (ID 721). This can be accomplished using the kill command. The kill command is installed by default on Unix operating systems and can be found in /bin.
The following command will kill a process with ID of PID:
victim# ./kill -9 PID
Note | We are not necessarily recommending this course of action to remediate such a situation. We mention it only because it is a possible option to remediate this kind of situation and has been used successfully in the past. |
After all the information has been transferred to the destination (forensic) workstation, it is a good idea for you to get an MD5 checksum of the output. Because the forensic workstation could be a Unix system (instead of a Windows system), this section offers the correct Unix notation.
Md5sum is distributed with the base Linux operating system and a similar version, md5, is distributed with FreeBSD.
The following command will calculate the MD5 checksum of the output files and save them in a file called md5sums.txt:
forensic# md5sum -b * > md5sums.txt
Note | On a *BSD system, the command is not md5sum but rather md5, and the command does not require use of the -b switch. |
At any point, md5sum can verify the MD5 checksums of any files if you supply a list. The following command will verify the MD5 checksums for a list of files and report any altered content:
forensic# md5sum -c md5sums.txt
Note | md5sums.txt will not have its correct MD5 checksum reported within itself and will always be mismatched from its true MD5 checksum. This is because the file is being written to as md5sum is calculating the checksum. |
Carbonite was developed by Keith J. Jones and Kevin Mandia at Foundstone, Inc., as an answer to loadable kernel module (LKM) root kits, specifically Knark (see Chapter 10). It can be downloaded from http://www.foundstone.com and runs on most Linux v2.2 kernels (though it was developed on RedHat and has shown the best results on a similar system).
Because processes can be running without an associated binary accessible on the file system, the traditional action of forcefully powering off the machine would destroy evidence. Therefore, hidden processes with Knark’s kill -31 command could be retrieved if it were possible to get into the kernel and examine the process table. Therefore, one answer seemed to be to fight LKM root kits with LKM solutions, and Carbonite was incarnated.
Carbonite must be compiled on a system with the same kernel as the victim machine. The kernel version can be observed, typically, with the following command:
victim# uname -a
After a trusted machine with the same version of your victim machine is available, untar the contents of the Carbonite package and change directory into it. Type the following command to make the package:
forensic# make
The package compiles, and a carbonite.o file is created. Copy this directory to the victim machine in a trusted manner (via disk or CD). Then, Carbonite will need to be installed into the kernel using the following command:
victim# ./carbonite.sh
Note | The carbonite.sh file may need to be edited to suit your specific needs. For instance, if you are using Carbonite in a live response, you will want to point the script at a trusted version of the module loader (insmod) so it does not use the copy on the compromised machine. |
When Carbonite is inserted into the kernel, it temporarily freezes the system until it has accomplished its mission. It creates a directory, /tmp/CARBONITE, that will contain an image of every process running on the machine. The name of the process images will be CARBONITE.command.PID, where command is the process name and PID is the process ID.
An additional file, CARBONITE.html, is created and can be loaded in a web browser. This file is similar to the file created using the ps command (reviewed earlier), but because it is acquired by entering the kernel directly, it is more reliable and will show all the processes, even if they are hidden by Knark.
Caution | Carbonite will write to the host media. Keep this in mind if you plan on leaving the option open for forensic duplication! |
The system we analyze for this case study is a RedHat v6.2 Linux machine. This host is a standard RedHat installation without the unnecessary services disabled and removed.
The attacker gained access through the rpc.statd service, which was an exploit rampant a few years back. Once the attacker gained access, he added new users so that he could telnet in any time he wanted. This method of attack leaves fingerprints in the live response we will perform.
We, as the system administrator, logged into the system in the morning and noticed that the message of the day (the /etc/motd file) was modified to say the following:
You site be lame. I hacked it. I plan to remove files unless you pay be me good $4000 US dollars. This no jokey. I am one mean guy! Your momma wears combat boots! Signed, Vladimir Dorkchov
We, of course, rolled our eyes, thinking it was an overzealous kid trying to have fun on our system.
The “powers that be,” once again, would like confirmation of the accuracy of this message before they dedicate resources to the investigation. Armed with our incident response media, we begin our live response.
The examples throughout the “Unix Live Response Tool Kit” section were captured using this victim machine. You may want to refer back to these examples to refresh your memory about the results we found.
lsof The lsof command unearths two suspicious processes: the 1 process that writes to a file and opened a raw socket, and the inetd processes that opened TCP port 4375. Upon further analysis of the inetd.conf file, we see that the daemon opened a TCP port that is bound to a root shell. In short, it provides a prompt command needing no login credentials (therefore, it doesn't show up in the output of last or w) to gain root access. There should never be a legitimate reason for this to happen. The following line is from the inetd.conf file:
4375 stream tcp nowait root /bin/sh -h
ls The ls command provides us with a new user account (kjohnson) that was not on the system before the machine was compromised. This directory contains the 1 executable, and we can see when it was probably created (from the modified times) and executed (from the last accessed times).
last The last command supplies us with the last key piece of information, because we never saw kjohnson log in (kjohnson was the owner of the 1 process and altered the /etc/motd file). What we can assume, if the logs were not altered, is that the user account mpepe was used as a backdoor into the system after it was compromised and switched users to kjohnson via the su command.
| < Day Day Up > |
|