Planning for Recovering of Services

---

Now that you’ve designed a plan for responding to a security incident, you will see how to plan for the recovery of services and/or data. There are many organizations that cannot afford, financially or publicly, to have services unavailable. Imagine if the services provided by Visa or MasterCard were simply unavailable. Purchases across the world would stop. Obviously, these companies have plans in place to recover services regardless of the situation. You’ll need to have these plans in place before the security incident occurs.

Once an incident is detected, you will need to take the necessary steps to bring your services back online. Prior to taking any steps, make sure you notify your organization’s security response manager. The team should take great care to document each and every step that is taken for recovery. This is one of the most important parts of the recovery process because the documentation will be used for review and will also prevent hasty decisions from hampering the recovery effort. This documentation may also be useful in the event that a legal investigation takes place.

As mentioned previously, the compromised system(s) should be disconnected from the network. All methods of access from the outside world should be included, such as Remote Access Service (RAS), dial-up (unplug the phone line), and the wireless network (it should be disabled). If possible, try to avoid rebooting the system or terminating sessions; this will result in active processes being terminated and can make it more difficult to locate evidence that may be in use at the time of discovery.

It is extremely helpful to make an image or snapshot backup of the compromised system. This snapshot can be used for review in a lab environment to investigate the compromise. There are several third-party utilities that can be used for this type of image; Symantec’s Ghost is probably one of the more popular choices. This snapshot can also be used by the legal forensics team to mount an effective case against the attacker(s).

Now that you’ve protected the data by taking a snapshot of the system at the time that the incident is discovered, you should analyze the intrusion. Check the event logs using Event Viewer and evaluate the configuration information. Figure 2.4 shows the Event Viewer application with security auditing enabled.

Figure 2.4: The Event Viewer

Look especially for modifications made to the operating system or its services. Check for out-of-place users and group memberships; many attacks involve creating a user on the compromised system and granting that user administrative privileges. Check for changes to the system Registry and look for any hidden shares (shares that don’t appear in Explorer). You can use the net share command at the command prompt to show all of the active shares on the server, as seen in Figure 2.5.

Figure 2.5: An example of the net share output

You should also check for services and processes to see if any are running that shouldn’t be. Double-check the name of the executing process; many attackers name the executable of their services to resemble operating system processes or other standard services that are usually running on a server. You should have a list of the services and processes that should be on the system and should be running, including the startup behavior of the services. This can be used to compare the list after an incident to see if there are any changes. Many applications are also run from within other services; svchost.exe is a common executable that is listed in the Windows Task Manager utility and running other applications that may or may not be applications that should be running. In Figure 2.6, you can see that there is more than one instance of SVCHOST.EXE running on the system. You should check to see what applications are running from within svchost.

Figure 2.6: Task Manager

As you can see, it is very difficult to know what is actually running; even certain security software will alert you that svchost.exe is accessing the network or the Internet. If you don’t know what is running through svchost.exe, it is impossible to know for sure whether or not you are allowing a Trojan horse program or some other kind of backdoor attack to get through by allowing it to run. You can use tasklist /svc from the command prompt to display the applications that are running within a specific thread and look for unknown processes in the resulting list, as seen in Figure 2.7.

Figure 2.7: Tasklist output

Verify that the data on the server is intact and has not been tampered with. Depending on the role of the compromised server, the data will vary. If the system is a web server, verify that the web pages and other data have not been changed or removed. You’ll also want to search for files that may have been left behind by the intruder. Intruders commonly plant network sniffers to record network activity, some of the programs that the intruder plants may be hiding their true functionality in a service or program that appears to be valid. These types of programs are referred to as Trojan horse programs, because they hide their real purpose. You’ll also want to check for files that may appear to be system files: Explore.exe instead of explorer.exe, for example.

Comb through the log files on the system to get an idea of how the system was compromised so that you can take the proper steps to prevent it in the future. In addition to checking the systems that you are sure were compromised, you will want to check the other systems on your network. Think of the compromised server as an infected computer that could be used to infect other machines that it comes in contact with.

Real World Scenario: Recovering Services by Making Hard Decisions

Figment Enterprises is a large-scale organization specializing in providing financial data in real time to its customers through a Web service. The company has sites in Los Angeles, Minneapolis, and Philadelphia and headquarters in Wilmington, Delaware. The computer security incident response team is located at corporate headquarters. Each site maintains service level agreements (SLAs) with its customers guaranteeing that any incident will be properly disclosed and the service will be made available within 15 minutes of the time it became unavailable.

There are LAN administrators that deal with the day-to-day management of network operations at each site. In the event of an emergency security situation, the response team can use remote tools such as Terminal Services or can be flown to the location where the incident has occurred.

During a routine security audit at the Philadelphia office, the security response team discovers that there has been a breach and that one or more attackers have compromised the Philadelphia web server. The team takes the necessary steps, notifying the response manager and removing the network cable from the back of the server. The response team switches the uncompromised backup service into production.

The response team has a meeting to determine how to proceed to get the server back into production. Because this server is critical to the company and needs to be returned to production quickly the decision is made to format the server and reinstall all necessary applications and services from scratch. This solution is the only way to be sure that all of the services that are running on the system are uncompromised. The server will be imaged prior to being formatted so it can be investigated in a lab environment.

Once you have verified that the systems are no longer compromised, you should return them to production, with an emphasis on auditing and logging so that they can be constantly monitored to be sure that there was nothing missed or that they don’t get compromised again.

Note

The only way to be 100% sure that there are no more remnants of the intruder is a complete format and restore from a time that was known to be valid and secure.

In the next section, you will learn about segmenting your network to separate the portions that are insecure. This practice is common when the security of the network is critical to ongoing activities in an organization.

---