Incident Response

Incident response refers to the process of identifying, investigating, repairing, documenting, and adjusting procedures to prevent another incident. Simply, an incident is the occurrence of any event that endangers a system or network. Two types of incident responses need to be discussed: internal incidents and incidents involving law enforcement professionals. Figure 4.15 illustrates the interlocked relationship of these processes in an incident response. Notice that each of the steps, including the first step, is related. Incidents are facts of life, and you want both your organization and yourself to learn from them.

Figure 4.15: Incident response cycle

The first type of incident response will be discussed here. Bringing law enforcement into the picture will be discussed in Chapter 10, "Security Management." In either event, it is a good idea to have the procedures that you will generally follow included in an Incident Response Plan (IRP). The IRP will outline what steps are needed and who is responsible for deciding how to handle a situation.

Note 

Law enforcement personnel are governed by the rules of evidence, and their response to an incident will be largely out of your control.

You need to carefully consider involving law enforcement before you begin. There is no such thing as dropping charges. Once they begin, law enforcement professionals are required to pursue an investigation.

Words matter. The term incident has special meanings in different industries. The term incident in the banking and financial areas is very specific, and it involves something that includes the loss of money. You would not want to call a hacker attempt an incident if you were involved in a bank network. This would automatically trigger an entirely different type of investigation.

The next five sections deal with the phases of a typical incident response process. The steps are generic in this example. Each organization will have a specific set of procedures that will generally map to these steps.

Incident Identification

Incident identification is the first step in determining what has occurred in your organization. An internal or external attack may have been part of a larger attack that has just surfaced, or it may be a random probe or scan of your network.

Note 

An event is an IDS-triggered signal. Operations personnel will determine if an event becomes an incident.

Many Intrusion Detection Systems will trigger false positives when reporting incidents. False positives are simply events that are not really incidents. Remember that IDS is based on rules and attack signatures. If the rules are not set up properly, normal traffic may set off the analyzer and generate an event. You do not want to declare an emergency, unless you are sure you have one.

One of the problems that can occur with manual network monitoring is overload. Over time, a slow attack develops which slowly increases in intensity. Manual processes typically will adapt, and they may not notice the attack until it is too late to stop it.

In the manually monitored situation described above, a type of phenomenon occurs called the boiled frog effect. A frog that is put in a cold pan of water will stay in the pan if the temperature is slowly turned up. The frog could leave the pan at any time, but it does not. The frog will cook before it realizes what has happened. Like the frog, personnel tend to adapt to changing environments if the changes occur over a long enough period of time. An automated monitoring system, such as an IDS, will sound the alarm when a certain threshold or activity level occurs.

Once you have determined that you indeed have an incident on your hands, you need to consider how best to handle it. This process, called escalation, involves consulting policies, consulting appropriate management, and determining how best to conduct an investigation into the incident. You want to make sure that the methods you use to investigate the incident are consistent with corporate and legal requirements.

A key aspect, often overlooked by systems professionals, involves information control. When an incident occurs, who is responsible for managing the communications about the incident? Employees in the company may naturally be curious about a situation. A single spokesman needs to be designated. Remember, what one person knows, one hundred people know.

Investigating the Incident

The process of investigating an incident involves searching logs, files, and any other sources of data about the nature and scope of the incident. If possible, you want to determine if this is part of a larger attack, a random event, or a false positive. False positives are very common in an IDS environment, and they may be merely the result of unusual traffic in the network. It may be that your network is being pinged by a class of computer security students to demonstrate the return times, or it may be that an attack is being launched by an automated tool. You may find that the incident does not require a response if it cannot be successful. Your investigation may conclude that a change in policies is required to deal with a new type of threat. These types of decisions should be documented, and if necessary, reconfigurations should be made to deal with the change.

Real World Scenario: The E-Mail Incident

You are the network administrator of a small network. This network has an old mail server that is used for internal and external e-mail. You periodically investigate log and audit files to determine the status of your systems and servers. Recently, you noticed that your e-mail log file has been reporting a large number of undeliverable or bounced e-mails. The addresses appear to be random. Upon examining the e-mail system, you notice that the outbound mail folder seems to be sending mail every second. A large number of files are being sent. After inspecting the workstations in the business, you determine that several of them have out-of-date antivirus software. How should you handle this situation?

For starters, you may have one or more viruses or worms in your system. This type of virus sounds like an SMTP virus. Outlook and Outlook Express are the most popular virus spreaders in use today. A virus, such as the Klez32 virus, can gain access to the address directory and propagate itself using SMTP.

You should investigate why the antivirus software is out-of-date, upgrade these systems as appropriate, and add server-based and mail-server virus protection capabilities to your network.

Repairing the Damage

One of your first considerations is to determine how to restore access to resources that have been compromised. Then, of course, you must reestablish control of the system. Most operating systems provide the ability to create a disaster recovery process using distribution media or state files of the system.

Once a problem has been identified, what steps will you take to restore service? In the case of a DoS attack, a system reboot may be all that is required. Your operating system manufacturer will typically provide detailed instructions or documentation on how to restore services in the event of a loss.

If a system has been severely compromised, as in the case of a worm, it may not be possible to repair the system. It may need to be regenerated from scratch. Fortunately, antivirus software packages can repair most of the viruses you encounter. But what if you encounter something new? You may need to start from scratch with a new system. In that case, you might be highly advised to do a complete disk drive format or repartition to ensure that nothing is lurking around on the disk waiting to infect your network again.

Real World Scenario: The Virus That Keeps On Giving

A virus recently hit a user in your organization through an e-mail attachment. He updated all of the programs in his computer and updated his antivirus software; however, he is still reporting unusual behavior in his computer system. He is also receiving complaints from people in his e-mail address book because he is sending them a virus. You have been asked to fix the problem.

He probably has contracted a worm that has infected the systems files in his computer. You should help him back up his user files to a removable media. Then you should completely reformat his drives and reinstall the operating system and applications. Once you have replaced these, you can install new antivirus software and scan the entire system. After the scan is complete, you can help him reinstall data files and scan the system again for viruses. This should eliminate all viruses from system, application, and data files.

Documenting the Response

During the entire process of responding to the incident, you should be documenting the steps that have been taken to identify, detect, and repair the system or network. This information is valuable; it needs to be captured in case an attack like this occurs again. The documentation should be accessible by the people most likely to deal with this type of problem. Many help-desk software systems provide detailed methods you can use to record procedures and steps. These types of software products allow for fast access.

You may also want to inform the software or systems manufacturer of the problem and how you corrected it. Doing so may help them inform or notify other customers of the threat and save time for someone else.

Adjusting the Procedures

Once the incident has been successfully managed, it is a worthwhile step to revisit the procedures and policies in place in the organization to determine what changes, if any, need to be made.

Answering simple questions can sometimes be helpful when you are resolving problems. Stated questions in a policy or procedure manual might include:

• How did the policies work or not work in this situation?

• What did we learn about the situation that was new?

• What should we do differently next time?

These simple questions can help you adjust procedures. This process is called a post mortem, and it is the equivalent of an autopsy.