6.7. Analyzing DDoS tools | Internet Denial of Service: Attack and Defense Mechanisms

This section deals with the analysis of hosts that are suspected of being involved in a DDoS attack, including identification and analysis of malware artifacts associated with these DDoS attacks. We will start by looking at some past DDoS tool analyses that have been made public, and then move on to guiding someone in performing a similar analysis on his own. In general, one should follow the same kind of who, where, what, when, why, and how that a reporter would. Some of the goals are to explain, in sufficient detail for someone to act on this information, how the DDoS network looks and behaves, how it was set up, how an attacker uses it, and how to detect its constituent parts and tear it down.

Take a moment to study Figure 6.4. It diagrams the flow of command and control traffic, as well as DDoS attack traffic. Malware artifacts associated with the DDoS attack network are at sites A, B, C, and D, among other places.

Figure 6.4. Illustration of all DDoS network traffic across all involved sites

During the attack, the victim site can detect only a huge flood of traffic or disruption in their servers' activities. The victim site can capture and analyze the attack traffic but has no access to the control traffic exchanged by DDoS network members. Further, the attack traffic cannot be captured in full (the monitoring system's hard drives would fill in minutes, if not seconds, at greater than gigabit per second speeds) but must be carefully sampled in short intervals. The compelling reason for capturing attack traffic is to understand and characterize the attack and devise appropriate selective responses. Another reason is to preserve some evidence of the attack for possible law enforcement activity. Analysis of traffic logs can prove bandwidth consumption rates, identify some of the attacking agents to do traceback, and can help verify that the attack tools the FBI may find on a suspect's system actually match the traffic that was seen coming into the victim's site. It will also help your own network engineers and incident response team or outside consultants in understanding what is going on and getting you back online quickly at a reasonable cost. There is nothing worse than paying a consultant hundreds of dollars an hour to try to guess what was happening on the network during the attack, which has stopped two hours before he arrived at your site, especially if the attack starts up again two hours, two days, or two weeks after he leaves.

Networks that host DDoS agents or handlers are in a much better position to gather evidence and identify the path leading to the attacker, since they can capture control traffic and malware artifacts.

6.7.1. Historical DDoS Analyses

The first public analyses of DDoS tools were performed by David Dittrich in 1999 and early 2000. Dittrich later collaborated with others, including Sven Dietrich, and Neil Long. While admittedly very technical, these analyses provide examples of all of the concepts that will be discussed below, along with references to the tools used and other related information. Anyone wanting to prepare to respond to the use of DDoS attack tools, whether at a victim site or a site hosting a DDoS handler or agent, should refer to documents on forensics, malware analysis, incident response, "Active Defense" and cyberwarfare, etc., found on Dave Dittrich's Web site [Ditd], in addition to the books referenced in Section 6.1.

The motivation for analyzing these tools was to (1) provide detailed information for peers in the incident response community that would allow detection, identification, and mitigation to occur faster; (2) facilitate law enforcement and military efforts to investigate attacks using these tools, should they be directed against critical infrastructure hosts or networks; and (3) try to anticipate what would come next and make recommendations that would prevent more severe attacks in the future. While some of these goals were accomplished, the fact that DDoS is still a growing problem five years after its appearance on the scene, with many victims just learning about it for the first time, means that this success is by no means complete.

A by-product of these analyses was the development of IDS signatures and host- and network-oriented scanners. The prototypes included in the analyses of trinoo [Ditf], TFN [Dith], Stacheldraht [Ditg] and Shaft [DLD00] were rewritten in C by Dave Dittrich, with assistance from Marcus Ranum, David Brumley, and others, and new features such as scanning by CIDR block were added. Another network-based scanner was David Brumley's RID.^[8] Those scanners send packets to potential remote DDoS agents or handlers and exploit vulnerabilities in DDoS tools to make them signal, "Hi, I'm here!," thus identifying compromised machines. Mark "Simple Nomad" Loveless also developed a tool called Zombie Zapper,^[9] which not only allows detection, but also issues commands to shut down agent machines. Of course, some of the DDoS tools used passwords, and all of them could be reprogrammed to use another port, making it more difficult to find and control them.

^[8] RID is no longer maintained but is available from Packetstorm Security at http://packetstormsecurity.nl/distributed/rid-1_0.tgz.

^[9] Zombie Zapper is available from BINDView's Web site at http://www.bindview.com/Support/RAZOR/Utilities/Unix_Linux/ZombieZapper_form.cfm.

Another way of detecting DDoS tools is to scan the file system from within the hosts themselves. In 2000, Mitre was contracted by the National Infrastructure Protection Center (NIPC) to write a signature-based detection tool for scanning file systems for several known DDoS tools (samples of which were provided by DDoS researchers and incident response teams). This tool, called find_ddos [NIP01], worked very well for identifying several DDoS handlers and agents, even if the original source code was modified prior to compiling. It must be run locally on a suspected compromised host using the system administrator account, and it is susceptible to being fooled by kernel level rootkits, which prevent it from seeing the malware in the first place. This may require that someone physically visit each potentially compromised host and boot first from trusted media to get a good scan. Using external network traffic analysis techniques, as described in this chapter, would help confirm beforehand if the host needed to be scanned or not. Network failures, reports from a victim of a DDoS attack that has identified the host, etc., can also trigger a more manually intensive internal file system scan.

6.7.2. Full Disclosure versus Nondisclosure

Before getting into details of analyzing a DDoS attack, let us digress into a brief discussion of the major points surrounding the disclosure and open discussion of computer security issues, including vulnerabilities, attack tools, and techniques. The debate involves two main positions surrounding computer security information: full disclosure (pro) versus nondisclosure (con). Without making any judgments, here are some of the main pros and cons regarding disclosure.

Con: You do not want to tip off the attackers.
By not disclosing anything publicly, the attackers who are using a program have no idea that anyone knows of their tools, where and how they are using them, and whether an investigation may be under way.
This assumes that the attackers are monitoring whatever channels you report to, and that they can correlate the information you provide with their own tools and activities. It is not clear when this assumption is valid or invalid.
Regardless, there is still some necessity to use discretion in what gets published, when, and where. In some circumstances, where a new attack tool is seeing use in the underground, and no known defenses exist, notification of the CERT Coordination Center, and perhaps the FBI, is warranted to provide some lead time to law enforcement investigators and incident response organizations.
Pro: Without knowledge of how DDoS tools work, how can you protect yourself or your network?
System administrators need to know how DDoS tools work, and how they appear and behave on their networks and computer systems, in order to find them and remove them. Without this knowledge, they are helpless to stop DDoS attacks or assist law enforcement.
Vendors of computer security products and services often provide just enough information to scare potential customers into buying their products, but not enough for anyone to take self-help actions. Some of the harshest criticism of partial disclosure comes from other vendors or those in the Open Source community, who may be excluded from the details that would allow them to produce response tools. There is a market advantage to control of information, and that market advantage can cloud the issues and create tremendous controversy.
Some individuals believe that the problem will not, and perhaps cannot, be solved by a small handful of vendors (even if they own the majority of the computer security market) because their products are not pervasive enough to provide a complete solution. Those who cannot afford the commercial products, or sites that cannot use these commercial products (like universities and ISPs), are left in a position of knowing there is a problem, but not knowing how to respond to it. These parties believe that the more widely the techniques and skills of DDoS defense are spread, the faster the overall response will be in mitigating emerging DDoS attacks.
Con: You do not want to help the attackers learn how to use DDoS tools or learn to build better ones.
Sharing analyses of DDoS tools would enable novice attackers to learn where to find DDoS tools, understand how to use them, make them more effective and less buggy, and assist them in not getting caught.
This argument assumes that attackers do not already share information about how DDoS tools work, and that they are not capable of finding bugs and improving them on their own. It is probably true that the highest-skilled attackers have both information sharing and software engineering skills at their disposal, and it is probably also true that some unskilled attackers would not be able to learn how to use these tools if analyses were not public.
Counterarguments to this include mentioning all the thousands of "hacker" Web sites and chat channels where information is shared, as well as places where pirated software, credit cards, and shell accounts are traded, which allow even unskilled attackers to gain enough systems to perform DDoS attacks without understanding the technical details of what they are doing. Those putting forward this argument try to balance the availability of information in the attacker community (which cannot readily be assessed anyway, since some of it is truly in the underground) with the benefit to defenders having this same information for use in protecting their networks. This balance cannot be accurately measured.
Pro: Software vendors will never discover and patch the vulnerable software or improve software writing skills if security problems are not understood.
Vendors of commercial software products for desktop computers, and server software for Web sites, etc., are not perfect. They produce code that contains bugs. There is no way around this, and every software product has bugs (some of which cause security problems). Unless vendors know where these security flaws are and how they can be exploited, they cannot fix them.
This is another claim that is hard to quantify. It is true that statistics show an increase in the number of vulnerabilities that are made public each year, and there are varying frequencies of patch and security advisory releases for various software and hardware products.
Con: Users do not pay attention to details of attacker tools, so they do not take appropriate actions to secure their systems.
Very few users of computer programs take the time to study hacker tools and read hacker Web sites as a way to make purchasing decisions, or to learn how to properly configure their computers. Security researchers, who produce detailed analyses of things like DDoS tools, are not helping the average user in a meaningful way.
The sheer number of hosts compromised (to this day!) by worms like Nimda, Code Red, Slammer, W32/Blaster, and Nachi; and hosts infected by viruses like W32/Sobig, MyDoom, and others that require the user to click on attachments show that there is a large and growing population of users who do not have sufficient skills to protect or clean up their own computers.
Of course, detailed technical analyses of DDoS tools will not help this audience, and in fact this is very rarely the audience targeted by these analyses. The goal here is getting the technical information to the technical people, and then having organizational administrators and public policy makers understand the threat and develop policies that address the problems before they become critical.
This is a classic problem in diffusion of information and organizational behavior that will probably be around as long as humans survive as a species (providing these problems do not hasten the end by themselves.) How exactly does knowledge about security flaws get from the researchers who document the vulnerabilities to executives who make business decisions about how products are manufactured, to engineers who design these products, and finally to those who purchase and use them? In this case, the researchers are documenting the vulnerabilities being exploited by attackers (e.g., the problems of TCP/IP that allow denial of service). Consumers will continue to purchase products that have these vulnerabilities. They do not know any better to demand that vendors produce products that do not have these vulnerabilities, which means engineers will not design them differently. But at the same time, CEOs will not read the analyses produced by computer security researchers, so they will not be making decisions about hiring, training, or resource allocation toward better engineering and testing to minimize the problems.
But a question still remains. Will not discussing these issues change the situation any faster?
Pro: Vendors and engineers of commercial products cannot design more robust products if they do not understand the failure modes.
Engineers themselves cannot design more secure products if they do not understand the vulnerabilities that exist in today's products. This may be one of the biggest benefits to sharing of technical information about DDoS, especially to router manufacturers, network engineers, software engineers, protocol writers, and application designers.
The issue of robust software engineering may be the real key. Should consumers of software systems have to suffer for a sufficiently long and painful time with viruses, worms, computers that crash, or Web sites that cannot be reached, all in order to motivate them to change their own purchasing behavior? Do they have to be forced to change the way they configure and use their computers through legal liability, which may result from what some would argue is their own negligence? Or do millions of consumers have to have their identities stolen and bank accounts emptied before they start to demand more secure software products?

As noted above, many of these individual pro and con points are hard to fully evaluate, so obviously the argument on both sides will continue.

6.7.3. How to Analyze Malware Artifacts

The term malware artifact means a program or file associated with malicious software. This includes compiled executable programs, the source code that produced them, scripts, log files, libraries, or any other file associated with an intrusion being used by the attacker. DDoS tools found on a compromised computer are one form of malware artifact, as are lists of handlers/agents and other files these DDoS tools rely on to function.

We will now take a look at the strategic and tactical issues you should consider when analyzing DDoS tools (or any malware artifacts, for that matter). To follow this material or perform the recommended analysis steps, you will need moderate Unix and Windows system administration experience and familiarity with TCP/IP networking concepts.

It is important to note that attack tools can be customized to run only on a particular host, using tools such as Burneye and Shiva [Meh03]. In essence the tools are quasilicensed to that particular host (by "fingerprinting the host,") and will not run on any other host. This has some consequences for DDoS tool analysis and forensics, as it slows down the analysis process. On the positive side for defenders, these tools are not in widespread use, partly because of the same problems that plague all uses of cryptography. These are not trivial. The encryption of attack tools leads to a typical key management problem: If the attacker uses the same password for all his agents, then the compromise of one agent key will compromise the entire network, but if different passwords are used, then many such keys need to be remembered. At the time of writing this book, no large deployment of such anti-analysis and anti-forensics tools was publicly known. Progress has been made in breaking tools such as Shiva [Eag03]. The recent (2003) follow-on to Burneye, objobf [Tes], shows that attackers are well aware of code obfuscation techniques such as the ones outlined in [Wan00] and [Wro02].

The discussion below assumes that there is no anti-forensics technique being used, such as binary encryption, advanced rootkits and kernel level concealment, log alteration, etc. If such techniques are being used, this significantly increases the difficulty and skill level required of the analyst. Always remember: Just because something is not seen, that does not mean it is not there. David Dittrich has a FAQ file on rootkits that can help you understand them, and some ways to get around them at http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq.

There are two things to consider at the start:

Preserve evidence to avoid altering it. Dominique Brezinski and Tom Killalea authored RFC3227, the Guidelines for Evidence Collection and Archiving [BK01] (also known as Best Current Practice number 55) to define the order and methods by which one should collect and preserve evidence. Other texts cover these issues as well, e.g., "Basic Steps in Forensic Analysis of Unix Systems" [Ditb, IAA, CIPS], and several examples of detailed analyses of a compromised honeypot [Spi02, Pro04] were performed as part of the Honeynet Project "Forensic Challenge" [Pro].
Be very careful when analyzing malware. You should never run malware you find on a system unless you are using a secure, isolated, and contained environment. You should either use a newly installed system that you plan to wipe and reinstall after analysis, or use a virtual machine environment that does not allow write operations to the guest operating system's file system. This way you do not run the risk of the malware getting loose (in the case of worms), or damaging your system if the attacker has booby-trapped it to do damage when run by someone other than the attacker. Having the host be isolated from your production network will prevent possible disruption from flooding, etc.

How Does the Malware Function?

In order to classify the malware, prioritize the response, and focus energies, it is helpful to know its purpose and function. Is it a scanner? Is it a remote exploit? How about a rootkit component or a log cleaner? If it is a DDoS tool, are you analyzing a handler, or an agent? Does it include features of a blended threat, where it can do DDoS, scanning, remote shell, self-upgrade, and serve you up the latest first-run movies in VCD (Video Compact Disc) format on an IRC channel platter?

The Internal Signature of the Malware

How can someone identify this malware artifact? It may have several different signatures. Try to identify and provide examples of as many of them as possible. These include:

File System Signature What directory was the malware artifact found in? What file name was used? When were the files created? What are the checksums or fingerprints of all components? If it is still running, what open file handles does it hold?

On Unix, use lsof (LiSt Open Files), which shows open file handles, sockets, current working directory, program that is running, etc. A Stacheldraht agent, seen with lsof, would look like this:

    # lsof -c ttymon    COMMAND  PID  USER  FD   TYPE  DEVICE  SIZE  NODE   NAME    ttymon   437  root  cwd  DIR   3,1     1024  37208  /usr/lib/libx/...    ttymon   437  root  rtd  DIR   3,1     1024      2  /    ttymon   437  root  txt  REG   3,1   324436  37112  /usr/lib/libx/.../ttymon    ttymon   437  root  mem  REG   3,1   243964  29140  /lib/libnss_files-2.1.1.so    ttymon   437  root  mem  REG   3,1  4016683  29115  /lib/libc-2.1.1.so    ttymon   437  root  mem  REG   3,1   342206  28976  /lib/ld-2.1.1.so    ttymon   437  root   3u  sock  0,0             779  can't identify protocol    ttymon   449  root  cwd  DIR   3,1     1024  37208  /usr/lib/libx/...    ttymon   449  root  rtd  DIR   3,1     1024      2  /    ttymon   449  root  txt  REG   3,1   324436  37112  /usr/lib/libx/.../ttymon    ttymon   449  root   0u  inet  811             TCP  *:32222 (LISTEN)    ttymon   449  root   3u  sock  0,0             779  can't identify protocol

In this example, the name of the program is ttymon, although it may not show up that way in a process listing (programs can change their apparent name in Unix). One can identify the current working directory of the process (FD column, or file descriptor, with a value of cwd), which is /usr/lib/libx/.../. (Note the use of the directory name with three dots. This creates a hidden directory in Unix, which is a tactic intended to fool novice Unix administrators into not noticing the attacker's files.) There may be other files related to the malware stored in this same directory, including perhaps a list of agents, if this node is serving as a DDoS handler. The example also indicates that the process is listening on port 32222/tcp (line with TCP *:32222 (LISTEN)), and that it has a raw socket open for sniffing (the column with TYPE of sock and unidentifiable protocol). The program itself is identified as ttymon (FD value of txt).

A somewhat similar Windows program to lsof would be Foundstone's FPort program. A Power bot as seen with FPort would look like this:

    FPort v1.33 - TCP/IP Process to Port Mapper Copyright 2000 by    Foundstone, Inc. http://www.foundstone.com    Pid   Process           Port   Proto Path    884   inetinfo   ->      21    TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    884   inetinfo   ->      25    TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    884   inetinfo   ->      80    TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    1400  winnt      ->     100    TCP   C:\winnt.exe    444   svchost    ->     135    TCP   C:\WINNT\system32\svchost.exe    884   inetinfo   ->     443    TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    8     System     ->     445    TCP    736   MSTask     ->     1044   TCP   C:\WINNT\system32\MSTask.exe    884   inetinfo   ->     1052   TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    660   sqlservr   ->     1056   TCP   C:\MSSQL7\binn\sqlservr.exe    8     System     ->     1067   TCP    660   sqlservr   ->     1433   TCP   C:\MSSQL7\binn\sqlservr.exe    1400  winnt      ->     2350   TCP   C:\winnt.exe    1400  winnt      ->     2351   TCP   C:\winnt.exe    1400  winnt      ->     2352   TCP   C:\winnt.exe    1400  winnt      ->     2353   TCP   C:\winnt.exe     [hundreds of lines removed . . .]    1400  winnt      ->     2646   TCP   C:\winnt.exe    1400  winnt      ->     2647   TCP   C:\winnt.exe    1400  winnt      ->     2648   TCP   C:\winnt.exe    772   termsrv    ->     3389   TCP   C:\WINNT\System32\termsrv.exe    884   inetinfo   ->     4700   TCP   C:\WINNT\System32\inetsrv\inetinfo.exe    1152  nt         ->     4836   TCP   C:\inetpub\scripts\nt.exe    1152  nt         ->    12624   TCP   C:\inetpub\scripts\nt.exe    444   svchost    ->      135   UDP   C:\WINNT\system32\svchost.exe    8     System     ->      445   UDP    260   lsass      ->     1027   UDP   C:\WINNT\system32\lsass.exe    220   winlogon   ->     1046   UDP   C:\WINNT\system32\winlogon.exe    248   services   ->     1051   UDP   C:\WINNT\system32\services.exe    884   inetinfo   ->     1064   UDP   C:\WINNT\System32\inetsrv\inetinfo.exe    564   llssrv     ->     1087   UDP   C:\WINNT\System32\llssrv.exe    464   spoolsv    ->     1217   UDP   C:\WINNT\system32\spoolsv.exe    884   inetinfo   ->     3456   UDP   C:\WINNT\System32\inetsrv\inetinfo.exe    1152  nt         ->    12623   UDP   C:\inetpub\scripts\nt.exe

The lines with the names winnt and nt in the Process column are associated with the bot's functions. To identify what files were created around the same time, one can use forensics tools that reveal file Modify, Access, and Change (MAC times) time stamp history. For Unix there are free forensics tools such as The Coroner's Toolkit (http://www.porcupine.org/forensics/) or Sleuthkit (http://www.sleuthkit.org/sleuthkit/docs/ref_timeline.html). Sleuthkit was written by Brian Carrier, originally as a research project that helped win him second place in the Honeynet Project "Forensic Challenge" [Pro]. This toolkit will help you construct a time line of file system modifications, recover deleted files, and otherwise do static analysis of a copy of a compromised host's file system. To illustrate this, here is a time stamp listing from a host running the mstream [CER01b, DWDL] DDoS tool, generated by Sleuthkit (see instructions about creating file activity timelines with Sleuthkit at http://www.sleuthkit.org/sleuthkit/docs/ref_timeline.html):

 Apr 13 2000 16:02:42  12060 .a. -rwxr-xr-x root/www root  /bin/chown                       12660 m.. -r-sr-xr-x root/www bin   /bin/login Apr 13 2000 16:02:43   2048 m.c drwxr-xr-x root/www root  /bin                       12660 ..c -r-sr-xr-x root/www bin   /bin/login                      168748 .a. -rwxr-xr-x root/www root  /usr/bin/as                       64796 .a. -rwxr-xr-x root/www root  /usr/bin/egcs                       64796 .a. -rwxr-xr-x root/www root  /usr/bin/gcc                       64796 .a. -rwxr-xr-x root/www root  /usr/bin/i386-redhat-linux-gcc                      168496 .a. -rwxr-xr-x root/www root  /usr/bin/ld                       12656 m.c -rws--x--x root/www root  /usr/bin/old                       12656 m.c -r-xr-xr-x root/www bin   /usr/bin/xstat                        2315 .a. -rw-r--r-- root/www root  /usr/include/_G_config.h                        1313 .a. -rw-r--r-- root/www root  /usr/include/alloca.h [ many lines deleted ]                         178 .a. -rw-r--r-- root/www root  /usr/lib/libc.so                       69638 .a. -rw-r--r-- root/www root  /usr/lib/libc_nonshared.a                        6162 .a. -rw-r--r-- 1046     squid /usr/src/linux/include/asm-i386 /errno.h                        1492 .a. -rw-r--r-- 1046     squid /usr/src/linux/include/asm-i386 /socket.h                         277 .a. -rw-r--r-- 1046     squid /usr/src/linux/include/asm-i386 /sockios.h                         305 .a. -rw-r--r-- 1046     squid /usr/src/linux/include/linux/errno.h Apr 13 100 16:02:44     702 m.c -rwxr-xr-x root/www root  /etc/rc.d/rc.local                        1024 m.c drwxr-xr-x root/www root  /root/.ncftp                           9 m.c lrwxrwxrwx root/www root  /root/.ncftp/history                           9 m.c lrwxrwxrwx root/www root  /root/.ncftp/log                           9 m.c lrwxrwxrwx root/www root  /root/.ncftp/trace                       29696 m.c drwxr-xr-x root/www root  /usr/bin                       17016 m.c -rwxr-xr-x root/www root  /usr/bin/rpc.wall                        8460 .a. -rw-r--r-- root/www root  /usr/lib/crt1.o                        1124 .a. -rw-r--r-- root/www root  /usr/lib/crti.o                        1892 .a. -rw-r--r-- root/www root  /usr/lib/gcc-lib/i386-redhat-linux /egcs-2.91.66/crtbegin.o   [ many lines deleted ] Apr 15 2000 05:55:09   1024 m.c drwxr-xr-x root/www root  /var/named                        1024 m.c drwxr-xr-x root/www root  /var/named/ADMROCKS Apr 15 2000 05:56:19  20437 .a. -rwxr-xr-x root/www root  /usr/sbin/tcpd Apr 15 2000 05:56:20     34 .a. -rw-r--r-- root/www root  /usr/libexec/awk/addy.awk                       35628 .a. -rwxr-xr-x root/www root  /usr/sbin/in.telnetd Apr 15 2000 05:56:26 159576 .a. -rwxr-xr-x root/www root  /usr/bin/pico                         975 .a. -rw-r--r-- root/www root  /usr/share/terminfo/v/vt200                         975 .a. -rw-r--r-- root/www root  /usr/share/terminfo/v/vt220

From this listing, the following observations can be made:

On April 13 at 16:02, /bin/chown was run (note .a. for access time stamp.)

    Apr 13 2000 16:02:42 12060 .a. -rwxr-xr-x root/www root /bin/chown

At the same time, /bin/login was modified (note m.. for modify time stamp).

                         12660 m.. -r-sr-xr-x root/www bin /bin/login

Also at the same time, the compiler egcs and gcc was run and /bin/old and /bin/xstat were created.

            168748 .a. -rwxr-xr-x root/www root  /usr/bin/as             64796 .a. -rwxr-xr-x root/www root  /usr/bin/egcs             64796 .a. -rwxr-xr-x root/www root  /usr/bin/gcc             64796 .a. -rwxr-xr-x root/www root  /usr/bin/i386-redhat-linux-gcc            168496 .a. -rwxr-xr-x root/www root  /usr/bin/ld             12656 m.c -rws--x--x root/www root  /usr/bin/old             12656 m.c -r-xr-xr-x root/www bin   /usr/bin/xstat

Simultaneous access to .h files in the system include directory /usr/src/linux/include/, indicates that the program being compiled uses network libraries.

     6162 .a. -rw-r--r-- 1046   squid  /usr/src/linux/include/asm-i386/errno.h     1492 .a. -rw-r--r-- 1046   squid  /usr/src/linux/include/asm-i386/socket.h      277 .a. -rw-r--r-- 1046   squid  /usr/src/linux/include/asm-i386/sockios.h      305 .a. -rw-r--r-- 1046   squid  /usr/src/linux/include/linux/errno.h

Next, the /etc/rc.d/rc.local file is replaced (note m.c for modify and i-node change time stamps).
```
    Apr 13 100 16:02:44 702 m.c -rwxr-xr-x root/www root /etc/rc.d/rc.local 
```
Further examination of this file (not shown in the example) revealed that the change was to include the line /usr/bin/rpc.wall at the end of the file, thus restarting the agent on each reboot.

ncftp logging files were also replaced. Further analysis showed that they were deleted and turned into links to /dev/null to disable logging of file transfers over the network using ncftp program.

                    1024 m.c drwxr-xr-x root/www root   /root/.ncftp                       9 m.c lrwxrwxrwx root/www root   /root/.ncftp/history                       9 m.c lrwxrwxrwx root/www root   /root/.ncftp/log                       9 m.c lrwxrwxrwx root/www root   /root/.ncftp/trace

The program /usr/bin/rpc.wall was modified, and C runtime libraries were loaded.

            8460 .a. -rw-r--r-- root/www root   /usr/lib/crt1.o            1124 .a. -rw-r--r-- root/www root   /usr/lib/crti.o            1892 .a. -rw-r--r-- root/www root   /usr/lib/gcc-lib/i386-redhat-linux  /egcs-2.91.66/crtbegin.o

If anti-forensic techniques such as sophisticated rootkits or kernel changes are being employed, some of this file system information would not be available and it would not be as easy to infer what exactly occurred and in what order. The same holds for the situation in which the system is not examined right away, but instead used for a long time, thus overwriting file access information.

Surface Analysis Surface analysis of malware artifacts attempts to understand the purpose and behavior of the program by examining the files installed on the system by an attacker.

For example, strings embedded in the malware artifact or file types containing malware code (e.g., shell scripts, executable objects, libraries, as reported by programs such as file on Unix) contain useful information. Knowing the file type helps narrow the search while investigating files on the system. Scripts are easy to read, and may reveal how a toolkit works, how it was installed, and how it is hidden. Keep in mind that simply looking at executable files with a pager or editor is not a very good idea, as it can affect your terminal screen.

Also keep in mind that surface analysis alone is very unreliable. It can be easily defeated using simple techniques, and sometimes strings are embedded to trick the unwary analyst into thinking a program is something other than what it really is, or to waste the analyst's time going down a blind alley.

For example, running a file for a malware artifact associated with the ramen worm produces the following report:

`1:`	`ASCII text`
`2:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`asp:`	`ASCII text`
`asp62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`asp7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`bd62.sh:`	`Bourne shell script text`
`bd7.sh:`	`Bourne shell script text`
`getip.sh:`	`Bourne shell script text`
`hackl.sh:`	`Bourne shell script text`
`hackw.sh:`	`Bourne shell script text`
`l62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`l7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`lh.sh:`	`Bourne shell script text`
`login:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`net.tgz:`	`gzip compressed data, deflated, last modified: Thu Jan 25 00:39:04 2001, os: Unix`
`ps.tgz:`	`gzip compressed data, deflated, last modified:Tue Jan 23 23:37:26 2001, os: Unix`
`ramen.tgz:`	`gzip compressed data, deflated, last modified: Thu Jan 25 13:37:47 2001, os: Unix`
`randb62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`randb7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`s62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`s7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`scan.sh:`	`Bourne shell script text`
`start.sh:`	`Bourne shell script text`
`start62.sh:`	`Bourne shell script text`
`start7.sh:`	`Bourne shell script text`
`synscan62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped`
`synscan7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped`
`TD:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped`
`update.c:`	`ASCII text`
`w62:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`w7:`	`ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), not stripped`
`wh.sh:`	`Bourne shell script text`

The above output indicates that there are several Bourne shell scripts that make up the core of the worm (e.g., scan.sh, start.sh, start62.sh, and start7.sh) as well as a number of precompiled programs (the items identified as ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked (uses shared libs), stripped). Further examination of the executable files shows that they perform the scanning, the exploitation of remote vulnerabilities to spread the worm. Closer analysis of this worm kit revealed that the file td was a Stacheldraht agent!

Network State Signature What network connections does the malware create? Noticing network connections will give you information about the "last hop" stepping stone from which the attacker broke into your machine, or if your machine is the stepping stone, it will also tell you where the attacker bounced to. Sockets in the ESTABLISHED state are active. The LISTEN state means that a server process (or peer-to-peer application) is waiting for new connections to be initiated.

On Unix, you can use either netstat (which will list all network sockets) or lsof. Here is what the output of netstat on a Linux system shows you on a sample agent machine:

    # netstat -an --inet    Active Internet connections (servers and established)    Proto Recv-Q Send-Q Local Address   Foreign Address        State    tcp     0       0   0.0.0.0:12345       0.0.0.0:*          LISTEN    tcp     0       0   10.10.10.3:2222     10.10.10.10:32965  ESTABLISHED    tcp     0       0   10.10.10.3:2222     0.0.0.0:*          LISTEN    tcp     0       0   10.10.10.3:22       10.10.10.2:33354   ESTABLISHED    tcp     0       0   0.0.0.0:22          0.0.0.0:*          LISTEN    raw     0       0   0.0.0.0:1           0.0.0.0:*          7    raw     0       0   0.0.0.0:6           0.0.0.0:*          7

In this case, one can tell by the lines showing Proto values of raw that some process has opened a raw socket. This by itself is suspicious as raw sockets bypass operating system handling of network communication and are frequently misused to send spoofed packets. There is also a process listening on port 12345/tcp (Proto equal to tcp and Local Address equal to 0.0.0.0:12345), and two established connections to ports 22/tcp and 2222/tcp. (Foreign Address of 10.10.10.10:32965 and 10.10.10.2:33354.) Further examination of suspicious processes on a given machine has indicated that there is an exploited SSH daemon, detailed in [Dit01]. The listening port on 12345/tcp resulted from the buffer overflow exploit on the vulnerable server.

Another suspicious sign is the program placing a network interface in promiscuous mode. Promiscuous mode allows a program to see all packets on the LAN segment, not just those that are supposed to be for the Network Interface Card (or NIC) of this particular computer, and is used for sniffing other users' communications. One can use the ifconfig command on Unix to see if the NIC is in promiscuous mode. (Be aware that rootkits that replace the ifconfig command, or replace kernel-level process calls, will "lie" to you about the promiscuous mode flag in the NIC's driver. You may still see system log messages showing the NIC entering/exiting promiscuous mode, or you may have to use other means to get around the rootkit.)

Process State Signature What processes are created by the malware, and how do they show up in process listings? Sometimes blended threats will include several programs that all run in the background: a backdoor shell, a file transfer server, an IRC bot, an IRC bouncer, a sniffer, and perhaps a DDoS agent. The parent child relationships between these processes can tell you something about how they were started on the system (perhaps from a buffer overflow in a Web server, which might make them all share a common httpd parent process).

On Unix, you can use the ps (simple process listing), pstree (a tree version, showing parent child grandchild relationships between processes), and/or the top command (show processes continually, with those using the most resources at the top of the list).

System Log Signature What was logged by the system? On Unix, start with the syslog daemon and look at the files it keeps. This may (if the attacker has not wiped or deleted the log files) show some signs of his entry and activity. To illustrate this, here is a sample log file excerpt:

    XXX 11 15:26:11 XXXX in.fingerd[864]: connect from XXX-XXX-14.XXXXXXXXX.XXX    XXX 11 15:26:11 XXXX in.telnetd[865]: connect from XXX-XXX-14.XXXXXXXXX.XXX    XXX 11 15:26:11 XXXX telnetd[865]: ttloop: peer died: Try again    XXX 11 15:26:12 XXXX in.pop3d[866]: connect from XXX-XXX-14.XXXXXXXXX.XXX    XXX 11 15:26:13 XXXX in.telnetd[867]: connect from XXX-XXX-14.XXXXXXXXX.XXX      .      .      .    XXX 12 05:36:20 XXXX in.telnetd[1126]: connect from DDDDDD.XXXXXX.XXX      .      .      .    XXX 12 11:01:52 XXXX in.telnetd[1213]: connect from EEEEEEE.XXX.XXX    XXX 12 11:02:21 XXXX su: XXXXX on /dev/ttyp1      .      .      .    XXX 12 11:04:28 XXXX in.rlogind[1229]: connect from CCCCCCCC.XXXXXXXX.XXX    XXX 12 11:04:44 XXXX in.rlogind[1230]: connect from CCCCCCCC.XXXXXXXX.XXX      .      .      .    XXX 12 11:08:57 XXXX su: XXXXX on /dev/ttyp1    XXX 12 11:11:19 XXXX su: XXXXX on /dev/ttyp1      .      .      .    XXX 12 11:33:05 XXXX in.telnetd[1290]: connect from AAAAAA.XXXXXX.XXX    XXX 12 11:33:16 XXXX login: 1 LOGIN FAILURE FROM AAAAAA.XXXXXX.XXX, XXX    XXX 12 11:33:21 XXXX login: 2 LOGIN FAILURES FROM AAAAAA.XXXXXX.XXX, XXX      .      .      .    XXX 12 11:34:02 XXXX su: XXXXX on /dev/ttyp1    XXX 12 11:41:52 XXXX wu.ftpd[1327]: connect from BBBBBBB.XXXXXX.XXX    XXX 12 11:41:57 XXXX ftpd[1327]: USER XXXXX    XXX 12 11:41:59 XXXX ftpd[1327]: PASS password    XXX 12 11:42:00 XXXX ftpd[1327]: SYST    XXX 12 11:42:01 XXXX ftpd[1327]: CWD /tmp    XXX 12 11:42:06 XXXX ftpd[1327]: TYPE Image    XXX 12 11:42:06 XXXX ftpd[1327]: PORT    XXX 12 11:42:06 XXXX ftpd[1327]: STOR mountd    XXX 12 11:42:08 XXXX ftpd[1327]: QUIT    XXX 12 11:42:08 XXXX ftpd[1327]: FTP session closed    XXX 12 12:00:25 XXXX in.telnetd[1342]: connect from AAAAAA.XXXXXX.XXX

The entries from 15:26:11 to 15:26:13 show what appears to be a port scan. The lines with in.telnetd[1213] and in.telnetd[1290] show connections using telnet, followed by successful use of su to elevate privileges. The lines following the wu.ftpd[1327] FTP daemon connection show a login, and transfer of a file mountd (the attacker's mount daemon exploit program) to the /tmp directory. This leads you to both the last hop point of entry information for the attacker, which can be correlated with other logs to find other systems the attacker may control, as well as a directory in which to start looking for evidence.

An attacker's access attempts may also be logged in the application level logs, like Apache's access_log and referrer_log files, Microsoft's Internet Information Server (IIS), or the standard Windows Event Logs. Here is an example of a buffer overflow exploit attempt found in an Apache log file contained on a Unix system:

    ==> /var/log/apache/access_log <==    [deleted host ip] - - [07/Aug/2001:17:19:35 -0400] "GET    /scripts/..%c1%9c../winnt/system32/cmd.exe?/c+ping.exe+"-v"+igmp+    "-t"+"-l"+65000+[deleted target ip]+"-n"+7000+"-w"+0" 404 -

This shows the attempted exploitation of the Unicode directory traversal vulnerability in Windows IIS to run the ping.exe program to attack some site with a distributed ICMP Echo Request flood. It did not work in this case (note the 404HTTP failure error code) because Unix does not have a file called /winnt/system32/cmd.exe (or ping.exe for that matter), but the attacker blindly tried anyway.

The External Signature of the Malware

What TCP and UDP ports show up when scanned from the outside? Often, malware will include a backdoor, IRC bot, or other code, which will be listening on a TCP or UDP port. Performing a half-open scan (only sending SYN packets and seeing the response) is one way of detecting them.

You can use nmap, Nessus, or other port scanners to perform half-open scans from outside your network. Compare the information gathered this way with the information gathered from internal examination with programs like lsof on Unix, or FPort on Windows. Here is what a Windows host running the Power bot blended threat looks like when scanned with nmap:

    Starting nmap V. 2.53 by fyodor@insecure.org (www.insecure.org/nmap/)    Interesting ports on XXXXXXXXXXXX (192.168.1.225):    (The 65522 ports scanned but not shown below are in state: closed)    Port        State       Service    21/tcp      open        ftp    23/tcp      open        telnet    25/tcp      open        smtp    80/tcp      open        http    100/tcp     open        newacct    135/tcp     open        loc-srv    139/tcp     open        netbios-ssn    443/tcp     open        https    445/tcp     open        microsoft-ds    1025/tcp    open        listen    1026/tcp    open        nterm    4836/tcp    open        unknown    12624/tcp   open        unknown    TCP Sequence Prediction: Class=random positive increments                             Difficulty=17052 (Worthy challenge)    Remote operating system guess: Windows 2000 RC1 through final    release

In this case, the last line shows there was a listening backdoor on port 12624/tcp, which gives the prompt "Password:" if you connect to it. Note here that one should be very careful about just randomly connecting to listening ports on an infected host! You have no idea how the program works, what it might do if you connect to it, whether it reports connections to the attacker, and so on. It is best to try to get copies of the program and analyze them on a safe, isolated, throw-away system, or to passively capture network traffic first and analyze it to determine how the program may function from what it exposes to you on the network.

This is where the anti-analysis features mentioned earlier come into play. The malware may be programmed to prevent you from tracing it, or dumping its memory. It may employ per-block in-memory decryption, so even if you can dump memory, it will be hard to determine any strings, passwords, or embedded IP addresses or domain names. If the malware also encrypts traffic on the network, it becomes extremely difficult to do runtime, or dynamic, analysis. If you run across anything that goes beyond your abilities to analyze, ask around for help (start with organizations like the CERT Coordination Center). If you determine that you do have a very advanced tool on your hands, make sure you also report this immediately to your local FBI or Secret Service office, and do your best to preserve as much evidence as you can about the intrusion [BK01].

Some programs do not have listening ports, instead watching for special packets using embedded commands in ICMP Echo Reply, UDP, or crafted TCP packets that are not part of any established connections. These are a form of covert channels when they are used for command and control. If they eventually open up a port for the attacker to connect to, they are called a closed port backdoor.

The Department of Defense defines covert channel to be any communication channel that can be exploited by a process to transfer information in a manner that violates the systems security policy. In the context of DDoS, this means using some protocol in a way that was not intended, for the purpose of command and control of DDoS handlers or agents. For example, Stacheldraht used ICMP Echo Reply packets sent between handler and agents to communicate that the agent was still alive. You can see the words "skillz" and "ficken" in these packets (which somewhat defeats the covertness of the communication, but using ICMP packets nonetheless is not something that a network operator or system owner would typically ever see if they were not looking for it). Following is the illustration of some ICMP packet contents captured as part of a Stacheldraht agent's covert command and control channel:

    ICMP message id: 10.0.0.1 > 192.168.0.1 ICMP type: Echo reply     45 E 00 . 04 . 14 . 01 . 0F . 00 . 00 . 40 @ 01 . E9 . 53 S 0A . 00 . 00 . 01 .     C0 . A6 . 00 . 01 . 00 . 00 . B4 . 13 . 02 . 9A . 00 . 00 . 00 . 00 . 00 . 00 .     00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     73 s 6B k 69 i 6C l 6C l 7A z 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     .     . [60 lines of zeros deleted]     .     00 . 00 . 00 . 00 .    ICMP message id: 192.168.0.1 > 10.0.0.1 ICMP type: Echo reply     45 E 00 . 04 . 14 . 04 . F8 . 00 . 00 . 40 @ 01 . E5 . 6A j C0 . A6 . 00 . 01 .     0A . 00 . 00 . 01 . 00 . 00 . CE . 21 ! 02 . 9B . 00 . 00 . 00 . 00 . 00 . 00 .     00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     66 f 69 i 63 c 6B k 65 e 6E n 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 . 00 .     .     . [60 lines of zeros deleted]     .     00 . 00 . 00 . 00 .

A closed port backdoor might accept special UDP or ICMP packets, which tell it to "open up a listening TCP socket on 12345/tcp and bind a shell to it." The attacker then has a small window of time (perhaps just seconds) to connect to the server port; otherwise, it closes again and goes back to waiting for another covert command. This makes it much harder to detect the backdoor from the network, but still allows the attacker full access whenever it is needed.

The Network Traffic Signature of the Malware

Get full packet captures of network traffic to/from the infected host. In a case in which the attacker is using anti-forensic techniques that would prevent you from restarting the program yourself in a closed environment, these traces must be gathered while the attacker is using the tool.

It is also best to collect the traffic as close to the suspect computer as possible (i.e., on the same LAN segment), or at least at the border of your network so you can capture all traffic in and out of the suspect computer. One factor that will complicate things is the use of fully switched networks, especially ones that are centrally managed and for which subnet owners are not provided with port mirroring (known on Cisco hardware as Switched Port Analyzer, or SPAN). Without this ability, the switch will provide only a subset of traffic that is associated with the IP addresses of devices on the same switch port, plus any broadcast Ethernet traffic. (Critical Networks maintains a Web page with information on sniffers and network taps that may be of use: http://www.criticalnets.com/resources/index.html).

Start by getting an overall view of the "scene of the crime" by gathering high-level statistics of network flows. One public domain program for Linux that works well is tcpdstat [Cho00]. A version modified by Dave Dittrich at the University of Washington [bDD02] ports the code to Linux and adds some new protocols and features (like peak bandwidth use). Here is a view of a traffic capture with [bDD02] to/from a suspected DDoS agent:

    DumpFile:     suspect.dump    Id:           200203221735    StartTime:    Fri Mar 22 17:35:42 2002    EndTime:      Sat Mar 23 13:13:25 2002    TotalTime:    70663.53 seconds    TotalCapSize: 3414.13MB    CapLen:       1514 bytes    # of packets: 167707807 (15702.13MB)    AvgRate:      2.07Mbps      stddev: 2.89M     PeakRate: 12.16Mbps    ### IP flow (unique src/dst pair) Information ###    # of flows: 91 (avg. 1842942.93 pkts/flow)    Top 10 big flow size (bytes/total in %):     54.8% 9.1% 5.7% 5.2% 5.1% 5.1% 3.6% 3.5% 2.4% 2.2%    ### IP address Information ###    # of IPv4 addresses: 68    Top 10 bandwidth usage (bytes/total in %):     100.0% 54.8% 9.1% 5.8% 5.2% 5.2% 5.2% 3.6% 3.5% 2.5%    ### Packet Size Distribution (including MAC headers) ###    <<<<     [   32-   63]:     768292     [   64-  127]:  165385712     [  128-  255]:       3256     [  256-  511]:        310     [  512- 1023]:       3309     [ 1024- 2047]:    1546928    >>>>    ### Protocol Breakdown ###    <<<<    protocol       packets           bytes              bytes/pkt    -------------------------------------------------------------    [0] total   167707807 (100.00%) 16464877084 (100.00%)   98.18    [1] ip      167707807 (100.00%) 16464877084 (100.00%)   98.18    [2] tcp       2312006 (  1.38%)  2223505740 ( 13.50%)  961.72    [3] ftp            12 (  0.00%)         742 (  0.00%)   61.83    [3] http(s)         5 (  0.00%)         300 (  0.00%)   60.00    [3] http(c)         5 (  0.00%)         322 (  0.00%)   64.40    [3] sunrpc          4 (  0.00%)         268 (  0.00%)   67.00    [3] ident          40 (  0.00%)        2732 (  0.00%)   68.30    [3] socks          10 (  0.00%)         799 (  0.00%)   79.90    [3] squid           3 (  0.00%)         180 (  0.00%)   60.00    [3] icecast         2 (  0.00%)         120 (  0.00%)   60.00    [3] irc6667      6303 (  0.00%)      669118 (  0.00%)  106.16    [3] http-a          2 (  0.00%)         134 (  0.00%)   67.00    [3] other     2305620 (  1.37%)  2222831025 ( 13.50%)  964.09    [2] icmp       398486 (  0.24%)    27952528 (  0.17%)   70.15    [2] res_255 164997315 ( 98.38%) 14213418816 ( 86.33%)   86.14    >>>>

We can see from the StartTime: and EndTime: lines that the capture period was just under a day. The average traffic (AvgRate:) rate was 2.07 Mbps, with peaks ( PeakRate:) going up to 12.16 Mbps. This one host alone could take out a 10-Mbps network! Looking further, we see that 98% of the traffic ([2] res_255) is not even a valid protocol! It is a reserved IP protocol value that is not supposed to be used. This would slip past some tools that expect only TCP, UDP, and ICMP, but provides nice ground for formulation of filtering rules. As attack traffic is using an invalid protocol number, filtering traffic based on this signature will inflict no collateral damage.

Another program that may work better (although it is slightly harder to read at first) is Aguri [CKK]. The following example includes real IP addresses, but they come from the author's own published paper.

    %%!AGURI-1.0    %%StartTime: Thu Mar 01 00:00:00 2001 (2001/03/01 00:00:00)    %%EndTime: Sun Apr 01 00:00:00 2001 (2001/04/01 00:00:00)    %AvgRate: 14.91Mbps    [src address]  4992392109177 (100.00%)    0.0.0.0/0        87902964189 (1.76%/100.00%)     0.0.0.0/1       206637364377 (4.14%/14.78%)     0.0.0.0/2       205796877844 (4.12%/7.12%)       60.0.0.0/6    97928228974 (1.96%/3.00%)            62.52.0.0/16   51875058871 (1.04%/1.04%)       64.0.0.0/8    100831910967 (2.02%/3.51%)        64.0.0.0/9   74610984109 (1.49%/1.49%)     128.0.0.0/2     142349668983 (2.85%/13.33%)      128.0.0.0/3    197067746696 (3.95%/10.48%)       128.0.0.0/5   202911635757 (4.06%/5.45%)        133.0.0.0/8  69142535628 (1.38%/1.38%)                     150.65.136.91 54123094932 (1.08%)     192.0.0.0/4     212653628837 (4.26%/38.41%)      192.0.0.0/6    88855538654 (1.78%/1.78%)       202.0.0.0/7   235853368912 (4.72%/14.70%)         202.0.0.0/9         117196493427 (2.35%/6.77%)                     202.12.27.33    160473669718 (3.21%)                  202.30.143.128/25  60239291958 (1.21%/1.21%)                     203.178.143.127 94031811680 (1.88%)        204.0.0.0/6  228960094456 (4.59%/17.68%)         204.0.0.0/8 125458765333 (2.51%/7.58%)                     204.123.7.2     87103414877 (1.74%)                     204.152.184.75  165733431144 (3.32%)         206.0.0.0/7 164036959478 (3.29%/5.51%)          206.128.0.0/9      53526598302 (1.07%/1.07%)         207.0.0.0/8 57628266965 (1.15%/1.15%)       208.0.0.0/4   282590640975 (5.66%/31.72%)        208.0.0.0/6  116047154301 (2.32%/22.20%)         209.0.0.0/8 140888988219 (2.82%/11.78%)                     209.1.225.217   238192306019 (4.77%)                     209.1.225.218   209160635530 (4.19%)         210.0.0.0/7 154008321340 (3.08%/3.08%)         216.0.0.0/9         192899750315 (3.86%/3.86%)    %LRU hits: 86.82% (1021/1176)

A paper on Aguri [CKK] explains the summary output this way: "In the address profile, each row shows an address entry and is indented by the prefix length. The first column shows the address and the prefix length of the entries. The second column shows the cumulative byte counts. The third column shows the percentages of the entry and its subtrees."

This illustration shows that the "top talker" (by source address) is 209.1.225.217, with 4.77% of the total traffic. Aguri can read tcpdump files, so a packet capture taken using the libpcap format^[10] should quickly confirm which specific host may be involved in a DoS attack. (For example, if this host were sourcing a significant portion of the 14.91- Mbps average, that would be suspicious and may warrant further investigation.)

^[10] libpcap is the library that tcpdump uses for saving files.

It has also happened before that worms like Slammer, or DDoS tools like mstream, will cause router instability because the source addresses they forge are in multicast address space, or otherwise force the router CPU to process them as part of egress filtering, which effectively erases or disables router settings. The lesson here is to never assume hardware will not fail or that a parameter that was set at one time is still set the same way at the present time.

Once you have isolated a suspect system, and you can filter out the attack traffic, start looking at other flows for command and control traffic. Trinoo, for example, was very obvious and chatty on the network. In the following code we show some trinoo communication captured with ngrep program.

You can see the commands the attacker types in the right column, including the password to connect to the handler (betaalmostdone), the password between the handler and agents (l44adsl), the prompts from the handler (TRinoo>), and the commands the attacker types in (for example, msize 32000, dos 216.160.XX.Y and quit.) The leftmost column indicates the protocol (T for TCP, and U for UDP). The TCP stream is the command shell to the handler, and the UDP stream is the command and control channel between the handler and the agent(s).

    # ngrep -x ".*" tcp port 27665 or udp port 31335 or udp port    27444 interface: eth0 (192.168.0.200/255.255.255.0) filter: ip and    ( tcp port 27665 or udp port 31335 or udp port 27444 ) match: .*    U 192.168.0.1:32892 -> 10.0.0.1:31335      2a 48 45 4c 4c 4f 2a                                     *HELLO*    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      ff f4 ff fd 06                                            .....    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      62 65 74 61 61 6c 6d 6f       73 74 64 6f 6e 65 0d 0a     betaalmostdone..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      74 72 69 6e 6f 6f 20 76       31 2e 30 37 64 32 2b 66     trinoo v1.07d2+f      33 2b 63 2e 2e 5b 72 70       6d 38 64 2f 63 62 34 53     3+c..[rpm8d/cb4S      78 2f 5d 0a 0a 0a                                         x/]...    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      74 72 69 6e 6f 6f 3e 20                                   trinoo>    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      62 63 61 73 74 0d 0a                                      bcast..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      4c 69 73 74 69 6e 67 20       42 63 61 73 74 73 2e 0a     Listing Bcasts..      0a                                                        .    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      31 39 32 2e 31 36 38 2e       30 2e 31 2e 20 20 20 0a     192.168.0.1.      0a 45 6e 64 2e 20 31 20       42 63 61 73 74 73 20 74     .End. 1 Bcasts t      6f 74 61 6c 2e 0a 74 72       69 6e 6f 6f 3e 20           otal..trinoo>    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      6d 74 69 6d 65 72 20 31       30 30 30 0d 0a              mtimer 1000..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      6d 74 69 6d 65 72 3a 20       53 65 74 74 69 6e 67 20     mtimer: Setting      74 69 6d 65 72 20 6f 6e       20 62 63 61 73 74 20 74     timer on bcast t      6f 20 31 30 30 30 2e 0a                                   o 1000..    U 10.0.0.1:1025 -> 192.168.0.1:27444      62 62 62 20 6c 34 34 61       64 73 6c 20 31 30 30 30     bbb l44adsl 1000    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      6d 74 69 6d 65 72 3a 20       53 65 74 74 69 6e 67 20     mtimer: Setting      74 69 6d 65 72 20 6f 6e       20 62 63 61 73 74 20 74     timer on bcast t      6f 20 31 30 30 30 2e 0a                                   o 1000..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      74 72 69 6e 6f 6f 3e 20                                   trinoo>    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      6d 73 69 7a 65 20 33 32       30 30 30 0d 0a              msize 32000..    U 10.0.0.1:1025 -> 192.168.0.1:27444      72 73 7a 20 33 32 30 30       30                          rsz 32000    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]       74 72 69 6e 6f 6f 3e 20                                  trinoo>    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      64 6f 73 20 32 31 36 2e       31 36 30 2e 58 58 2e 59     dos 216.160.XX.Y      59 0d 0a                                                  Y..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      44 6f 53 3a 20 50 61 63       6b 65 74 69 6e 67 20 32     DoS: Packeting 2      31 36 2e 31 36 30 2e 58       58 2e 59 59 2e 0a           16.160.XX.YY..    U 10.0.0.1:1025 -> 192.168.0.1:27444      61 61 61 20 6c 34 34 61       64 73 6c 20 32 31 36 2e     aaa l44adsl 216.      31 36 30 2e 58 58 2e 59       59                          160.XX.YY    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      74 72 69 6e 6f 6f 3e 20                                   trinoo>    T 192.168.100.1:1074 -> 10.0.0.1:27665 [AP]      71 75 69 74 0d 0a                                         quit..    T 10.0.0.1:27665 -> 192.168.100.1:1074 [AP]      62 79 65 20 62 79 65 2e       0a                          bye bye..    T 192.168.100.1:1075 -> 10.0.0.1:27665 [AP]      62 65 74 61 61 6c 6d 6f       73 74 64 6f 6e 65 0d 0a     betaalmostdone..    T 10.0.0.1:27665 -> 192.168.100.1:1075 [AP]      74 72 69 6e 6f 6f 20 76       31 2e 30 37 64 32 2b 66     trinoo v1.07d2+f      33 2b 63 2e 2e 5b 72 70       6d 38 64 2f 63 62 34 53     3+c..[rpm8d/cb4S      78 2f 5d 0a 0a 0a                                         x/]...    T 10.0.0.1:27665 -> 192.168.100.1:1075 [AP]      74 72 69 6e 6f 6f 3e 20                                   trinoo>    T 192.168.100.1:1075 -> 10.0.0.1:27665 [AP]      6d 70 69 6e 67 0d 0a                                      mping..    T 10.0.0.1:27665 -> 192.168.100.1:1075 [AP]      6d 70 69 6e 67 3a 20 53       65 6e 64 69 6e 67 20 61     mping: Sending a      20 50 49 4e 47 20 74 6f       20 65 76 65 72 79 20 42     PING to every B      63 61 73 74 73 2e 0a                                      casts..    U 10.0.0.1:1025 -> 192.168.0.1:27444      70 6e 67 20 6c 34 34 61       64 73 6c                    png l44adsl    U 192.168.0.1:32894 -> 10.0.0.1:31335      50 4f 4e 47                                               PONG    T 10.0.0.1:27665 -> 192.168.100.1:1075 [AP]      74 72 69 6e 6f 6f 3e 20 50 4f 4e 47 20 31 20 52           trinoo> PONG 1 R      65 63 65 69 76 65 64 20 66 72 6f 6d 20 31 39 32           eceived from 192      2e 31 36 38 2e 30 2e 31 0a                                .168.0.1    T 192.168.100.1:1075 -> 10.0.0.1:27665 [AP]       71 75 69 74 0d 0a                                        quit..    T 10.0.0.1:27665 -> 192.168.100.1:1075 [AP]      62 79 65 20 62 79 65 2e 0a                                bye bye..

It was the simplicity of the protocol, and the lack of encryption or authentication of commands between the handler and agent, that allowed David Dittrich's scanner and Zombie Zapper to detect and/or control trinoo agents. Not all DDoS tools are this easy to detect and manipulate.

Real-time monitoring of network traffic can sometimes be a controversial topic for two reasons. First, you are capturing electronic communication in real time, which is covered by state and federal wiretap (or electronic communication privacy) statutes. Be careful to understand the exemptions defined in these laws to know if you are complying with or violating them. Not just anybody can monitor traffic on any given network. Second, you are leaving a known compromised host in operation, and thus in the control of an attacker. If you are a network operator, your primary goal is to get the system off the network to make the network regain stability and have it cleaned as soon as possible. However, if you are an incident responder, researcher, or law enforcement agent investigating a crime, your primary goal is to get as much network traffic as possible, which implies keeping the infected host on the network. These two contradictory goals must be balanced, hopefully getting as much traffic as is reasonable before shutting off wall ports. Another option to collect a sufficient amount of malicious traffic while minimizing its bad effects is to use devices like the Honeynet Project's "honeywall" (http://www.honeynet.org/tools/cdrom) or other layer two filtering bridges configured to block outbound traffic that exceeds specific bandwidth limits or uses specific ports and protocols. A device like the honeywall provides protection for the network infrastructure and makes logging traffic much easier.

The type of traffic that can be obtained by sniffing is determined by the role your site plays in the DDoS network. Does your site host a handler or an agent? Is it the victim of the DDoS attack? Are you just a transit provider, routing packets between other parties? Each of these situations provides a different perspective on the DDoS network and different opportunities to capture traffic for analysis or investigation by law enforcement authorities.

Figure 6.4 on page 184 illustrates a DDoS network in action. In this diagram, Site A is where the attacker is located. She uses a stepping stone at Site B, to connect to handlers at several locations, including Site C. There could be many, many different stepping stones used by the attacker. We depict only one stepping stone here. The handlers, in turn, relay attack commands to all of the agents they are controlling at numerous sites, one of which is shown as Site D. All of the agents then unleash a flood of packets toward the victim at Site E. That is the aggregation point for all traffic, and may be knocked out entirely by the flood.

As you can see, trying to relate the attack on the victim to the attacker requires tracing packet flows back from Site E, through all intermediary sites until you get back to Site A. If the attacker disconnects from the stepping stone at Site B, or switches to another stepping stone, the trail is broken. If she uses multiple stepping stones, instead of just the one depicted here, traceback gets harder. If logs are cleaned on each stepping stone, or they are used only once, traceback becomes even more challenging.

Now let us take a look at what traffic you can capture, based on your perspective in the network. In all of the figures in this section, the traffic that can be captured by sniffing at the network border is depicted by a circle with an "S" in it.

From the perspective of a network hosting agents, you can see the command and control traffic coming in from a handler (perhaps not all of them) to the agents at your site, and you can see the outbound DDoS attack traffic from this subset of agents going to the victim. This is shown in Figure 6.5.

Figure 6.5. Illustration of attack and control traffic seen from site hosting agents

Most sites will fall into this category, since there are typically far more compromised hosts used as agents than there are handlers, stepping stones, or victims (usually combined). A site hosting agents has a responsibility to identify and clean up the compromised hosts, lest they be used over and over for continued attacks. At sites with small incident response teams, or none at all, the typical cleanup is "wipe the drive, reinstall the OS, get back online." As discussed in Section 6.5, this has potential negative consequences. One consequence is that, unless you also patch the system to current levels, you will be placing the same vulnerability back on the system and it will be compromised again. Another is the destruction of evidence. At least some data collection and preservation is typically warranted in the case of large attacks. A better response is to restore the system from a recent, known clean backup that already has software patches installed. Alternatively, do the operating system installation and patching from CD-ROMs you have prepared while the system is disconnected from the network entirely. Ask the vendor or distributor of your operating system of choice how to produce a patch CD-ROM for such use. Windows users can try things like Microsoft's Security Readiness Kit (found at http://www.microsoft.com/technet/security/readiness/default.mspx), or if you are using Red Hat Linux, by downloading and creating CDs from the packages in their errata Web pages http://www.redhat.com/errata/. Doing less than that can expose newly cleaned machines to repeated intrusions, sometimes nearly immediately. (Note that if you have a PC on the network of a large Internet 2 educational site, and you try to install a Windows 2000 or XP system from the original CD-ROMs distributed with your computer dating prior to 2003, that system has a high probability of being infected with W32/Blaster, Nachi, or some other currently active worm or bot in less than one minute, that is, before the installation is even complete!)

In some cases, the command and control traffic would expose all of the agents in the DDoS network. In other cases, DDoS handlers keep track of all the agents within a file. Knowing that your site is hosting agents, you can identify the victim and assist them in identifying all of the agents in the DDoS network by monitoring traffic at their site, and you can assist the site hosting the handler in recovering information about a large percentage of the agents and perhaps take down the entire network. Working with the site hosting the handler also gets one hop closer to the attacker.

Moving now to the site hosting the handler, one can capture only command and control traffic. The traffic rate will be much lower, and the view of the agents in a DDoS network provides more information because the handler communicates with agents at numerous sites. The view from the site hosting the handler is depicted in Figure 6.6.

Figure 6.6. Illustration of control traffic seen from site hosting a handler

Note that now both parts of the command and control traffic are visible: the connection from the attacker to the handler's user interface, as well as the traffic to/from a set of agents. Capture all packets to and from the handler, including the data portion of packets (not just headers). Keep these logs secure.

If the attacker is not using an encrypted channel, you also get to see the passwords, user account names, and command syntax of the DDoS tool. This alone may give you enough information to determine what tool is in use, just from watching traffic from outside the handler. Host-level forensics are also an option here for recovering even more information, and preservation of evidence should be carefully done in the event that law enforcement is involved later in the process. Note that you should consider very, very carefully any actions that would take control of the DDoS network in any way. The best advice is to get law enforcement involved as soon as you understand how the DDoS network functions and provide a report to them.

You also want to attempt to identify the stepping stones used for incoming connections to the handler. Keep track of all of them, and determine if there is more than one IP address or network block that is favored by the attacker. What time of day are connections most often made? Identify the sites involved, and very carefully attempt to contact them, perhaps by going through their upstream providers, to make sure that you do not end up reporting the DDoS network to the attacker or one of her friends! If you get cooperation from the other sites, they can then attempt to identify other upstream stepping stones, and maybe even other handlers in the network (or networks) being used by the attacker.

Now we will look at the site that is hosting the stepping stone. Here we have only a partial view of the command and control traffic between the attacker and the handlers. There is no more visibility beyond the handlers to agents. This is illustrated in Figure 6.7.

Figure 6.7. Illustration of control traffic seen from site hosting a stepping stone

Monitoring at the stepping stone may reveal more handlers, and following this trail may ultimately lead to identification of the attacker's location, other stepping stones, or even more agents. Over time, a pattern of behavior of the attacker can be determined, and in some cases active connections to handlers can be identified during the point of an attack.

If law enforcement is involved, they are now able to use search warrants and subpoenas to get more information about accounts used by the attacker, perhaps even hitting upon the attacker's own computers and accounts. At worst, they are able to identify the previous stepping stone in use by the attacker and continue to follow the chain backward toward the attacker.

As you may be seeing by now, the process of traceback is slow, tedious, and labor intensive. It requires some pretty detailed understanding of network capture and analysis tools and techniques, preestablished mechanisms to capture traffic at the network borders, knowledge of what kind of DDoS tool is being used, how the tool functions and presents itself on hosts and the network, and the skills to put all of this together. Taking out an entire DDoS network is a hard problem, but it can be done.

While doing this level of tracing requires many skills and is labor intensive, it has proven successful in the real world. One of the authors, David Dittrich, has personally used the techniques described in this chapter to take down entire DDoS networks, some numbering in the hundreds of hosts (see the time line of DDoS activity leading up to the February 2000 attacks referenced in Chapter 3). Of course, Dittrich wrote the analyses of several of these early tools, and intimately knows how they work. On the other hand, these analyses are freely available on many sites in the Internet to allow anyone who is motivated and resourceful to know everything publicly available about these same tools. Anyone can use the techniques described in these analyses to do the same. Many people have done just that, including some talented members of federal law enforcement in several countries, and DDoS attackers have been brought to justice using these methods of investigation. Similar analyses of newly discovered DDoS tools continue to be written by people around the world. Of course this effort takes skill, discipline, and time and generally makes a profit only for security companies who resell this public information or use it in products they sell. The upside is the reward that comes from knowing that you have contributed to improving the state of computer security on a global scale. Luckily for all, there is a large supply of people with these qualifications and who share this belief.

Malware Command and Control

What ports or protocols are used for command and control? Does the program use password protection or encrypt communications or files? What are the passwords or cryptographic keys, and where can they be found? How are agents identified to handlers (and vice versa) and can you get a complete list of them all?

Earlier, we discussed several examples of looking at network traffic. The analyses of trinoo [Ditf], TFN [Dith], Stacheldraht [Ditg], Shaft [DLD00], and Power [Dita] all provide further examples of how to identify DDoS agents on the network.

This is some of the most sensitive information, and you should think very carefully about how to disclose this, to whom, and when. It would be very inappropriate, and perhaps very dangerous and a potential legal liability, to provide information that allows another attacker to take over and use a large DDoS network. For example, if you explain that someone has established a bot network on the Dalnet IRC channel #NotARealChannel, and he is using a particular bot with a password of notapassword, then anyone with basic knowledge of the bot in question could take over the entire botnet and use it for his own purposes.

The Source Code Lineage of the Malware

Attack programs are often shared in the computer underground, in source code form and in compiled binary form. When source code is shared, features of two or more programs may be exactly the same (e.g., how IP header checksums are computed or shell code is injected into overfilled buffers in order to force execution of code on the stack). By knowing how program code is shared and changed from program to program, it is sometimes possible to trace a program back to its author, or to use an existing signature detection method to identify code variants. An entire class of programs may share the same set of known signatures, but a new program that appears on the scene for the first time may go undetected or get packets past existing DDoS mitigation tools. These are all reasons to want to understand how a program developed over time.

You may have only a compiled binary, not the source code, for the particular malware artifact in hand. This makes it harder to distinguish lineage, but not entirely impossible in all cases. There are still symbols that may exist in the symbol table, strings used for prompts and messages to the user, etc., that can be identified and compared to some well-known artifacts.

Many programs borrow heavily from others, be it specific attack features, exploit shell code, cryptographic libraries, etc. Some (like knight.c) were coded originally in C for Unix systems, but have been compiled under portable C compiler environments like Cygwin on Microsoft Windows. Some people will simply assume that if the program was originally written for Unix, it will not work on Windows and can not be easily ported to different operating systems.

Who Is Using the Malware?

Going back to 1999, DDoS was most often used for a sort of "drive-by shooting on the Information Superhighway." Attackers were continuously fighting wars using DDoS to take down opponents' machines and IRC channels. However, DDoS is now becoming a component of crime in the form of acts like extortion or rigging online bids. Sometimes a disgruntled ex-employee will attack her former company to exact revenge for a believed false termination. One day a terrorist organization or nation state may use DDoS as a force multiplier in an attack on another state's critical infrastructure. Knowing who is attacking attribution, in legal and military terminology can be important.

This is another sensitive subject, though. While capturing traffic to and from suspected compromised hosts, you may get IP addresses of stepping stones, handlers/agents, or direct connections from the attacker's main systems. Beginners have been known to connect from their parents' PC on a DSL line that can be traced directly to a phone number by their ISP. Others have used stolen credit cards to dial in to "no charge" ISPs that do not even have accounts or caller-ID capable switches for their modem pools, resulting in near total anonymity from anywhere in the world. Knowing either of these things tells you something about the attacker (but even if she is connecting from an identifiable DSL or cable modem line, that still does not mean the attacker is sitting at that particular keyboard!).

To Whom Should You Provide the Analysis, and When?

Anyone who is involved in DDoS either victims or sites whose computers were compromised and used as stepping stones or DDoS handlers or agents has some responsibility to not only clean up his machines, but to report the incident to others and to help in mitigating the problem. As time goes on, and as DDoS attacks become more prevalent and more damaging, the legal situation may likely change in such a way that failing to act appropriately, not reporting participation, or not helping to stop DDoS attacks may incur liability on the part of those involved. Establishing policies and procedures before this becomes an issue is the best course of action, and the best time to start is right now.

In most cases, the DDoS attack will span multiple jurisdictions, within your own country or between several countries around the world. Many attacks these days involve the attackers spreading DDoS agents around the world, and using stepping stones in two or more countries before connecting to the DDoS handlers. This makes it much harder to attribute the attack and identify suspects. A good place to start is by contacting your local representatives of federal law enforcement agencies (in the United States that would include the FBI and the Secret Service). It is best to contact these agencies in advance, and get to know them (and let them get to know you) so that it is easier to reach out to the right person in a timely manner when an attack hits. Once law enforcement officers know who you are and understand your skill level and areas of expertise, they will trust your reports more and can prioritize their response appropriately.

Federal law enforcement agencies are (as Australians would say) "up to their armpits in alligators." The more complete and readable your analysis, with an executive summary and high-level overview of the attack itself, the easier their job will be. Getting bit-image copies of compromised hosts used to attack, and full packet traces of network attack and command and control traffic (time stamped, cryptographically fingerprinted, and kept secure for chain-of-custody purposes, etc.) all help make a potential case.

You should also report the incident, in similar detail (but perhaps without some of the more sensitive information from your own site) to national-level incident response centers, such as the CERT Coordination Center in the United States. These organizations often have liaisons with federal law enforcement and other organizations with a larger view of the attack landscape. By correlating events across a larger population, they can identify and help prioritize national-level responses to very large events. Organizations like the CERT Coordination Center also have liaisons to similar institutions abroad and can facilitate cross-border incident response. Contact information for the CERT Coordination Center is provided in Chapter 9.

If you are a network service provider, it would be good to notify your customers if they are victims, or sources, of DDoS attacks. Many ISPs see DDoS as being more of a bandwidth problem, opting to simply add capacity to prevent the network from becoming unstable. Their support staffing can be stretched thin, and it can be very time-consuming trying to explain to a customer (who may know very little about network administration) that his computer was compromised and is being used to attack others, and to assist in the cleaning process. Some ISPs gain notorious status for being unresponsive to DDoS problems.

Lastly, ISPs and NSPs should likewise report large DDoS events to law enforcement and incident response organizations, as described above. For the same reasons, but at a higher level, they have a responsibility to report events, especially if they see these events as being widespread through their association with other network providers. Consult your legal staff to establish policies and procedures for reporting.

What to Keep Private

As mentioned earlier, there are many pieces of information that you should think twice, or even five times, about releasing.

Some pieces of information, like your own internal IP addresses of compromised production hosts, you may never want to release. It may be best to anonymize these, but leave the times and source addresses of attacking hosts intact for investigators. If the event does end in court, you or your company may be required to release the information at that point for use in court. In order to accommodate this, the anonymization must be reversible with a key so that information is preserved and can be released during a trial.

Keep in mind, however, that IP addresses are not authenticated and can easily be spoofed, so they do not serve as a completely reliable source of information identifying a particular computer, let alone an individual. Attackers can, and do, control DNS servers that allow them to also map IP addresses to DNS names at will, so it is important to do reverse mappings to show how DNS names translate at a given point in time. (Provide both IP address and current DNS mapping in your reports, showing the time and time zone to which the mapping corresponds.) It is necessary to have several sources of information, preferably authentication logs used to obtain the IP address in the first place, to help correlate and ensure proper identification.

During your investigation, you may determine that one or more of your own computers are attacking someone, or you may obtain IRC traffic or command/control traffic from a DDoS handler or agent that identifies victims of DDoS attacks. You do not want to be responsible for providing information that opens other hosts up to abuse because you have essentially given account/password information allowing their remote control, or for them to be trivially taken over by someone else. Of course, someone else may find them and take them over anyway before you release information, but at least you could not be accused of providing the means. The IP addresses of victims are another thing that you want to consider carefully before releasing, and then you may want to talk to the victim sites and allow them to release their own information.

Names of individuals involved (especially if suspected of performing a DDoS attack) should be held closely, but provided to law enforcement. Releasing names of suspects, or making statements about them and their skill level publicly, can sometimes result in their attacking you in retaliation. Perhaps worse, if you cannot prove your allegations you might be sued for libel.