What to Do If Your System Is Compromised

The paper "Steps for Recovering from a UNIX Root Compromise," available from CERT at http://www.cert.org/tech_tips/; RFC 2196, "Site Security Handbook"; and the SANS publication "Computer Security Incident Handling: Step by Step" discuss procedures to follow in the event of a successful security breach. These documents present more formal procedures that a business, government office, or university might follow. The procedures assume some amount of spare storage space for taking snapshots of the system, assume available staff to analyze and diagnose the security problem, and discuss situations in which the victim site might want to initiate formal legal action.

Regardless of how an anomaly is investigated, the intrusion analyst must take care when performing the investigation. If the attacker notices that there's an investigator currently looking around on the same system, it's much more likely that the attacker will slash and burn their way out of the system. If the attacker thinks he or she is being followed or monitored, the attacker might begin deleting anything and everything in the way, causing real damage to the systems in question. An attack that might have resulted in only a defacement of a website might suddenly turn into deletion of entire partitions if the attacker notices the investigation.

After it has been determined that there has been a successful attack or that an attack is currently underway, a number of responses frequently occur. These will, of course, be dependent on your security policy.

If storage space is available, take a snapshot of the entire system in its current state for later analysis. If that isn't an option for you, at least snapshot the system logs under /var/log and the system configuration files under the /etc directory.

Keep a log. Write down everything. Documenting what you do and what you find not only is good for reporting the incident to a response team, your ISP, or a lawyer, but also helps to keep a record of what you've examined and what remains to be done.

If an attack has occurred or is currently underway, one of the first priorities is usually to stop the attack and prevent further damage from occurring. Keeping in mind that an attacker who notices an investigator on the same system is more likely to cause collateral damage, unplugging the system from the network is a common recommendation. With the network cable unplugged, the attacker simply can't cause additional damage. There is, of course, the possibility that the attacker will be using a tool to monitor the network interface and automatically cover his tracks should that interface's status change. However, those types of attacks are as yet unseen in the wild.

Looking for subtle changes to the system is part of this phase. An attacker may have set up a cron job to restart his daemons if they are stopped. In addition, it's quite typical for an attacker to replace common Linux utilities such as ls and ps with his own versions in order to hide their processes and files. In this regard, a program such as Chkrootkit can be helpful for host-based intrusions. Chkrootkit is discussed in Chapter 10.

The tools you use to mitigate damage will be determined by the type of attack. For example, a denial-of-service attack against a router will necessarily use different steps to mitigate the attack. The steps you might take to determine whether the system is compromised are the same steps to take in analyzing the compromised system:

1.	Check the system logs and use netstat and lsof to see which processes are running and which ports are bound. Check the contents of the system configuration files. Verify the contents and access modes of all your files and directories by checking their digital signatures. Check for new setuid programs. Compare configuration and user files against clean backup copies. It's very likely that the attacker installed trojan horse programs in place of the very system tools you're using to analyze the system.
2.	Take stock of any volatile information, such as which processes are running and which ports are in use.
3.	Boot off a boot floppy or a backup copy of the system. Examine the system using the clean tools from the unaffected system. As an alternative, install the disk drives as secondary drives in a noncompromised system, and examine the disks as data.
4.	Determine how the attacker succeeded in gaining entry, and determine what was done to your system.
5.	If possible, completely reinstall the system from the original Linux distribution media.
6.	Correct the security vulnerability, whether by making a more careful selection of services to run, by reconfiguring servers more securely, by defining access lists at the xinetd or tcp_wrappers level and at the individual server level, by installing a packet-filtering firewall, or by installing application proxy servers.
7.	Install and configure any system-integrity packages.
8.	Enable all logging.
9.	Restore user and special configuration files known to be untainted.
10.	Create MD5 checksums for the system binaries and static configuration files.
11.	Reconnect the system to the network and install any new security upgrades from your Linux vendor.
12.	Create MD5 checksums for the newly installed binaries, and store the checksum database on a floppy or some other system.
13.	Monitor the system for recurring illegal access attempts.