An incident, in the context of this chapter, is an anomalous event that can impact the confidentiality, integrity, or availability of the infrastructure. The anomaly might be malicious, or it might be an indicator of a system fault. In either case, we need to know how to set up alerts that warn us about a potentially threatening condition. Both IDS and system-monitoring mechanisms are useful for detecting suspicious events, but they are not of much help if administrators are not actually aware of conditions that these systems observe.
One way to remain apprised of the state of your resources is to periodically check the status screens of the monitoring system or the IDS console. Relying solely on this approach, however, does not allow detection and reaction to problems as soon as they occur. This is an especially significant concern for organizations that need to respond to incidents around the clock but cannot afford to hire personnel to observe the alert screen of the monitoring system 24 hours a day. Configuring IDS and monitoring systems to send out alerts and knowing how to respond to alarm conditions is an integral part of effective security perimeter maintenance. In this section, we discuss how to configure notification options in a way that is consistent with your security policy and monitoring requirements. Additionally, we examine considerations for responding to detected system faults and malicious events.
Relying solely on automation of IDS or of the monitoring systems to detect all suspicious events in real time is dangerous. We illustrated limitations of this approach in Chapter 8 in the discussion of false positives and false negatives. Therefore, be sure to combine automated alerting with manual examination of data that such systems collect.
An old slogan of a New York telephone company reminded people that "We're all connected." Accessibility of system administrators, wherever they are, allows systems to get in touch with them in close to real time whenever a device requires attention. When evaluating or deploying an IDS or a monitoring system, consider the following aspects of its notification functionality:
Keep in mind that some of the cheaper paging devices, especially those that are purely numeric, do not guarantee a message will be delivered. To ensure that an alert eventually reaches the administrator when he is back in the communications range, you might want to subscribe to paging services that can queue messages to guarantee their delivery. Alternatively, or in conjunction with this, consider configuring the monitoring system to periodically reissue alerts until one of them is acknowledged.
One of the advantages of having a centralized alerting configuration is that only a single host needs to have the ability to generate alerts destined for the mailboxes or pagers of system administrators. This eliminates the need to set up dial-out modems on every observed system for dialing a numeric pager or the need to enable each server to send email when its business purpose does not require such functionality. In Big Brother, for example, a central server consolidates performance and availability information, and only the host that is designated as BBPAGER issues notification alerts. To eliminate a single point of failure, you might consider setting up multiple alert-generating systems or configuring them to operate in a failover mode.
General Response Guidelines
Make sure your company's policies and procedures clearly explain what an administrator should do when responding to an alert. Creating and distributing a document that answers the following questions will help you make sure problems are resolved in a consistent and thought-out manner that has the support of management and technical personnel:
It is not uncommon for the administrator who is responding to the alert to perform a preliminary examination and then to call in the heavy artillery for in-depth troubleshooting. One such scenario, perhaps most relevant to this book, is when the administrator suspects that the event is malicious in nature. Your security policy should account for the need to investigate such situations and define the roles and responsibilities for the staff involved in responding to malicious incidents.
Responding to Malicious Incidents
Any anomaly, whether reported by an IDS or a monitoring system or recognized by a human, might turn out to be a malicious incident. After the event is deemed to be malicious, the specialized procedures created for handling such situations should guide your staff's response. These procedures can be broken into several phases, as defined in Computer Security Incident Handling Step by Step, published by the SANS Institute.3 The following list presents a high-level overview of these phases:
As you can see, responding to alerts, whether they relate to a system fault or a malicious event, is no easy matter. As we discuss in the following section, you might be able to automate responses to some of the simpler events that require immediate action and that can be addressed without directly involving the administrator.
Automating Event Responses
Automating responses to events that are unlikely to be false alarms helps to expedite problem resolution. For instance, a monitoring system might detect that a critical process on the observed host died, issue an alert to the administrator, and automatically start a new instance of the process. When configuring such functionality, you might want to set limits on the number of times the system attempts to take corrective action in a given time period. If the process repeatedly dies soon after being restarted, chances are good that the automated recovery mechanism cannot help in this situation, in which case an administrator should become directly involved. Even if the fault was automatically resolved, the administrator should still follow up to assess the scope of the problem, determine its cause, verify that the corrective action was acceptable, and attempt to prevent the fault from reoccurring.
Another type of automated response can take place when an intrusion detection system or an intrusion prevention system detects a malicious event. Such products may allow you to automatically respond to the attackfor instance, by resetting the offending network stream or dynamically reconfiguring the firewall to block the attack. As we discussed in Chapter 8, "Network Intrusion Detection," and Chapter 11, "Intrusion Prevention Systems," such automated response carries the advantage of shunning the attacker as soon as malicious actions are observed, but it is often dangerous because it might deny service to a legitimate user. When deciding whether to enable such intrusion prevention functionality, weigh the risk of mistakenly blocking an authorized user against the risk of not blocking a particular attack right away.
In our discussion about maintaining a security perimeter, so far we have looked at monitoring the infrastructure for faults and malicious events and discussed how to efficiently respond to alarming conditions. An effective way of decreasing the rate at which anomalies occur in the first place is to ensure that perimeter components are updated in a controlled manner. We examine the process of managing changes to the environment in the following section.