21.1 Case Study: Worm Mayhem

Right around lunchtime, a help desk operator at Example, Inc. (a medium- sized manufacturing company) received a frantic call from a user who was unable to use his PC: it was continually rebooting. The user also reported that strange items had appeared on his desktop. The help desk operator was not sure whom to contact about such issues, so he tried calling his boss, but his boss was not in at the moment. The operator then opened a case in his Remedy console, describing the user's problem and recording his machine's hostname. Unfortunately, other calls for unrelated support issues grabbed his attention and the rebooting desktop was forgotten.

Meanwhile, the worm ”which is what really caused the problems with the user's PC ” continued to spread in the company network. The malicious software was inadvertently brought in by one of the sales people who often had to plug their laptops into untrusted networks. However, most of the security-monitoring capabilities were deployed in the DMZ (or "demilitarized zone" ”a somewhat inaccurate term for a semi-exposed part of the network where you place publicly accessed servers such as web, FTP, and email servers) and on the outside network perimeter, which left the "soft, chewy center" unwatched. Thus, the company's security team was not yet aware of the developing problem.

The network traffic generated by the worm increased dramatically as more machines became infected and contributed to the flood. Only when many of the infected PCs began attempting to spread the worm out of the company network was the infection noticed by the security team, via the flood of pager alerts. Chaos ensued. Since the breach was not initiated from the outside, the standard escalation procedure the company had previously adopted for hacker attacks was ineffective . Several independent investigations, started by different people, were underway, but there was little or no communication. While some people were trying to install antivirus updates, others were applying firewall blocks (preventing not only the worm scanning but also the download of worm updates), and yet another group was trying to scan for vulnerable machines using their own tools (and contributing to the network-level denial-of-service condition).

After many hours, most of the worm-carrying machines were discovered and the reinfection rate was brought under control, if not eliminated. Due to a major loss in employee time, backend system outage , and unstable network connectivity, the management requested an investigation into who was responsible and how to prevent such incidents. The company hired a computer forensics consultant. Unfortunately, the initial infection evidence was either erased, overwritten on disk, or extremely difficult to find (nobody looked into the help desk system, where the initial call for help resided, since the help desk system was not deemed relevant for security information). The investigation concluded that the malicious software was brought in from outside the company, but the initial infection vector was not determined, since by then some of the machines had already been rebuilt by the IT department, overwriting the infected disk images. In addition, it was extremely difficult to track all the vulnerable and exploited machines, since there was no central point for such information.

This nightmare is what might happen to your company if it lacks a central organization for security monitoring and incident handling, as well as an incident response policy. Huge financial losses, dead-end investigation, an inability to accumulate experience and knowledge in order to improve, and many other problems are likely to result.

This chapter should help you to avoid the pitfalls of chaotic , ineffective incident response. As a first step toward our goal, let us clarify some important definitions. Then we'll build a foundation for an effective incident response policy based on the SANS Institute's six-step process. ^[1]

^[1] The SANS Institute's six-step incident response methodology was originally developed for the U.S. Department of Energy and was subsequently adopted elsewhere in the U.S. Government and then popularized by the SANS Institute (http://www.sans.org).