Lesson 2: Intrusion Detection Systems

Lesson 2: Intrusion Detection Systems

Intrusion detection is the process of monitoring and evaluating computer events and network traffic for signs of intrusions. In Chapter 4, you learned that an IDS is a hardware device with software that is used to detect unauthorized activity on your network. IDS implementations can log and alert you to unauthorized activity on your network. IDS software can be implemented on individual hosts, servers, at the network perimeter, or throughout the entire network. This lesson takes a more detailed look at IDS's.

---

After this lesson, you will be able to

• Classify IDS roles as host-based or network-based

• Select appropriate IDS response types

• Identify steps in a staged IDS deployment

  Estimated lesson time: 60 minutes

---

An IDS can be implemented as a network intrusion detection system (NIDS), system integrity verifier (SIV), or log file monitor (LFM). A brief explanation of each is reiterated here:

• A NIDS monitors network traffic and traffic patterns to discover someone attempting a DoS attack, port scans, or attempts to guess the password to a secured resource.

• An SIV monitors a single system's file structure to determine if (and when) an attacker modifies, deletes, or changes a system file. An alternate name for SIV IDS is target-based IDS.

• An LFM parses system log entries (from one or more systems) to identify possible system attacks or compromises.

These functions are not the only way in which IDS implementations are classified. The most common and simplified categorization of IDS implementations is to separate them into host-based and network-based IDS. There are also subordinate classifications that separate the ways in which IDS implementations detect and respond to intrusions. The remainder of this lesson explores network-based and host-based IDS, detection methods, and response types.

The Internet Engineering Task Force (IETF) put together the Intrusion Detection Working Group (IDWG) to draft documents defining IDS requirements, language, and framework. You can learn more about this work group and the documents produced at the Intrusion Detection Exchange Format Web site at http://www.ietf.org/html.charters/idwg-charter.html.

Network-Based IDS

As the basic description of NIDS just given conveys, network-based IDS's detect attacks by capturing and analyzing network traffic. A single NIDS can protect multiple hosts by listening to traffic on one network segment. NIDS implementations often utilize sensors (hosts running IDS software) at various points on a network. These sensors monitor and analyze network traffic at their locations and report potential attacks to a central management console. The sensors are commonly bastion hosts limited to running only the IDS sensor software. These hosts are often designed to operate in a mode, sometimes called stealth mode, that is difficult for attackers to detect. IDS can do the following:

• Increase overall security.

  The more layers of protection you have on your network, the safer the network is from attack.

• Protect multiple systems.

  A few well-placed NIDS's can protect a large network because they monitor all traffic on their subnet.

• Allow monitoring traffic inside your firewall.

  Some attacks come from the inside. A firewall at the threshold of your network or even at the perimeter network is not going to protect internal client systems from each other. NIDS can help you to discover internal attacks.

• Alert you to incoming attacks.

  By monitoring network traffic, NIDS can alert you when an attack is taking place, such as an attempt to overflow a buffer.

• Detect slow attacks.

  NIDS can keep track of suspicious activities over a long period of time. For example, if an attacker conducts a scan over a period of a week or month, NIDS can keep track of this and report when a certain threshold is met or exceeded.

• Delayed analysis.

  Some NIDS's allow you to capture packets now for later forensic investigation, sometimes referred to as honeynet mode.

• Take corrective action.

  Some NIDS's can actually do something beyond logging and alerting you to attacks. For example, some NIDS's could be used to change the configuration of a client or firewall to eliminate a possible attack.

In addition, NIDS's have a low impact on network traffic because they don't act as gateways. They collect and analyze traffic as it passes by, but they don't restrict the flow of network traffic.

Although NIDS's are flexible, powerful, and increasingly advanced detection and response mechanisms, they are not a panacea for network security issues. They have limitations, which include the following:

• Processing speed.

  Just like every device on the network, NIDS's have limited processing power and collection capabilities. NIDS's can drop packets and misdiagnose network issues if they are overwhelmed with packets to analyze. Ensure that the NIDS you select can keep up with the network on which you are planning to place it.

• Issues with segmentation.

  NIDS's listen to the traffic on a given network segment. NIDS capability is reduced when a switch, virtual local area network (VLAN), or router is used to reduce network traffic. The solution to this problem typically involves adding sensors to each segment. In the case of a switch, plug the NIDS into the monitoring port, which allows it to receive all traffic passing through the switch.

• Issues with encryption.

  The payload of an encrypted packet is difficult to analyze. Most NIDS's don't decrypt packets, so attacks that are encrypted tend to bypass NIDS detection mechanisms. Later in the lesson, we cover potential solutions to this issue.

• Attack success.

  Most NIDS's cannot determine whether an attack was successful; they report only that an attack was initiated. This means that network administrators must follow up to check if an attacked system was compromised.

• False positives and missed detections.

  A NIDS can only discover what it is programmed to discover. If an event appears to be an attack, a NIDS reports it. If a real attack doesn't appear to be an attack, a NIDS won't detect it. Detection methods and related issues are discussed later in this lesson.

• NIDS attacks.

  Some creative software engineers have developed tools to fool NIDS's or even attack them. Such tools are discussed later in this lesson.

NIDS Examples

Many different organizations produce NIDS's. Some are free and others cost thousands of dollars. Amazingly, the saying "you get what you pay for" doesn't seem to apply to NIDS's. The free NIDS's are sometimes rated higher than the commercial ones (see the MCP Magazine article "Barbarians at the Gate" by Saoutine, Perfiliev, and Corti, available at http://www.mcpmag.com). Here are some examples:

• Snort is a free open-source NIDS.

• Enterasys Networks' Dragon is a commercial NIDS.

• Realsecure is a commercial NIDS available from Internet Security Systems.

• NetProwler is a commercial NIDS available from Symantec.

• NFR NID is a commercial NIDS from NFR Security.

• Argus is a free network-monitoring tool for UNIX-based systems, and includes intrusion detection capabilities.

• SWATCH (Simple WATCHer) is a free LFM for UNIX-based systems.

For more information on NIDS, visit the SANS IDS FAQ at http://www.sans.org/newlook/resources/IDFAQ/ID_FAQ.htm.

NIDS Attack and Evasion

As you learned in Chapter 4, attackers might attempt to attack, bypass, disable, or fool those systems. There are several publicly available tools for attacking or confusing NIDS's. Here are some examples and descriptions of NIDS attack tools:

• Stick launches a direct attack against the NIDS using a Snort signature file that is used to identify attacks. The attack causes a large number of false alarms, which reduces the system resources of the NIDS. So many false alarms could cause a real alarm to go unnoticed by security administrators or the real alarm might not even be logged by the NIDS due to lack of system resources.

• Fragroute is a NIDS evasion tool that hides attacks from the NIDS by obfuscating attack packets. The tool intercepts, modifies, rewrites, and reorders packets so that the NIDS cannot identify them.

• Tribe Flood Network 2000 is a DDoS attack tool that uses encryption, decoy packets, and IP address spoofing to avoid NDIS identification. (DDoS attacks are described in more detail in the next lesson.)

Host-Based IDS

A host-based IDS (HIDS) is installed on individual computers to protect those individual systems. HIDS's are much more reliable than NIDS's in detecting attacks on individual systems. HIDS typically utilize operating system audit trails and system logs. Operating system audit trails, generated by the core of the operating system (kernel), are quite reliable for tracking system events. System logs also track system events and are smaller and easier to interpret. However, system logs are sometimes modified by attackers, so focusing on both audit trails and system logs provides a double-checking feature.

Many HIDS's are part of personal firewall software. Personal firewalls, or host wrappers, can examine all network packets, connection attempts, and logon attempts to their host machines. In addition, HIDS's can check the integrity of system files (and other files) to ensure that they are not tampered with. Some HIDS's are also designed to report intrusion attempts to a central IDS console located somewhere else on the network. This configuration allows a security administrator to monitor multiple host systems from one location. There are several benefits to using a HIDS on a network to protect your systems, including these:

• They are better than NIDS's at monitoring and keeping track of local system events.

• They aren't typically hindered by encrypted attacks. HIDS's can read transmitted packets before they are encrypted and received packets after they are decrypted.

• They can help to detect software integrity breaches, such as Trojan horse software, file modifications, and so on.

• Because HIDS's only protect a single system, switches, VPNs, and routers do not affect their functionality.

Although HIDS's have several benefits, they also have limitations, such as these:

• Difficult to manage.

  HIDS's are more difficult to manage than NIDS's in large networks because they must be configured and controlled on individual systems.

• Susceptible to DoS attacks.

  Attacks against the HIDS-protected host might affect the HIDS itself. DoS attacks against the host might disrupt or disable the HIDS. A successful attack against a host protected by a HIDS could potentially disable and destroy evidence collected by the HIDS.

• Require host resources.

  HIDS's require resources from the protected host. HIDS's need extra hard disk space to store logs and tracking information. HIDS's must also utilize processor time and memory to analyze packets, user-issued commands, audit trails, and system logs to protect the client.

Application-Based IDS

In an effort to increase security, software and hardware vendors are increasingly likely to integrate IDS into their products. (See "IDS at the Crossroads," Information Security Magazine, June 2002, available at http://www.infosecuritymag.com.) One example of this is a breed of HIDS known as application-based IDS. An application-based IDS analyzes the events occurring within a specific software application using the application's transaction log files. An application-based IDS is able to detect suspicious behavior that might go unnoticed by other forms of IDS because application-based IDS can analyze interactions between the user, the data, and the application.

There are several benefits to using an application-based IDS, which include the following:

• Monitor user/application interaction.

  Application-based IDS's can monitor the interaction between the user and the application, which could allow for the tracing of unauthorized activities.

• Unaffected by encryption.

  Application-based IDS's read and analyze application transactions and commands, so they are completely unaffected by network encryption and decryption.

As with any IDS, there are also limitations to application-based IDS's, such as these:

• Vulnerable to attack.

  Similar to a HIDS, an application-based IDS is close to the potential target of attack, the protected application. If the application is under attack, the logs that the application-based IDS analyzes (or the application-based IDS software itself) might be targeted in that attack.

• Difficulty detecting malicious software.

  Unlike HIDS's, application-based IDS's typically won't identify Trojan horses or other malicious software because they focus on a specific application's security, not the overall security of the host or network, as do HIDS and NIDS, respectively.

HIDS Examples

Several software vendors produce HIDS's and personal firewalls that include a HIDS. The following list is a sample of some of those HIDS's and personal firewalls:

• BlackICE Defender, a personal firewall and IDS product from Internet Security Systems.

• Zone Alarm, a personal firewall and IDS product from Zone Labs.

• Symantec Host Intrusion Detection System from Symantec.

• Host Intrusion Detection system from NFR Security.

• TCPWrappers for Unix systems.

• Host-based intrusion detection tools from Foundstone, such as fport, filewatch, attacker, and carbonite.

• Tripwire, a file and directory integrity-checking tool from Tripwire Security.

There are many other tools for many different operating systems that can help you discover intrusions. Sometimes these tools are grouped as forensic analysis tools and sometimes they are deemed intrusion detection tools. Any tool that can analyze file or directory integrity could be considered an intrusion detection tool, such as Message Digest 5 (MD5), Pretty Good Privacy (PGP), backup software, imaging software, and file comparison utilities (WINDIFF and DIFF). Further, tools that display processes or open connections could also be used for intrusion detection. This includes lsof, ps, tlist, sclist, and TCPView.

Detection Methods

There are two basic methods for detecting and analyzing attacks that IDS's utilize. The first and most commonly implemented method is called misuse detection. This type of detection requires the IDS to identify a predefined attack pattern. The second method is anomaly detection, which involves recognizing something suspicious or atypical. Each method has benefits and limitations, which are discussed in the following two sections. More recent IDS implementations increasingly use both methods.

Misuse Detection

A misuse detection in IDS (also called signature-based detection) works a lot like a virus scanner. A virus scanner locates viruses by identifying the virus programming code within a file, in memory, attached to e-mail, or somewhere else on a system's removable or fixed media. Misuse detection involves identifying an attack signature, which is some indicator that a specific attack is occurring. Misuse detectors analyze system activity, looking for events that match a predefined pattern of attack. Misuse detectors must have their attack signatures updated as new attack types are discovered. Again, this is similar to virus scanners, which must have their virus signature files updated.

The benefits of misuse detectors include the following:

• They can quickly identify defined attacks.

• They help system administrators track attacks.

• They don't generate many false alarms because they are programmed to recognize situations that are typically attacks. Of course, as you saw earlier, some attackers create fake attacks, which generate false alarms. Of course, these false alarms are still evidence of an attack (at least against the IDS).

Misuse detectors also have limitations, such as the following:

• They require updated attack signature files because they can only detect predefined attacks based on those signatures. Some misuse detectors employ state-based detection, which allows them to recognize variations of specific attacks.

• They can be targeted for attack with virus signature files (as with Stick, described previously).

Anomaly Detection

Anomaly detectors identify unusual activities or situations, called anomalies. Anomaly detectors classify abnormalities as potential attacks. To determine what is normal and what is not, an anomaly detector must gather information about the systems and networks on which it operates. Once enough information is available, the anomaly detector can identify abnormalities based on historical data.

The benefits of anomaly detectors are as follows:

• They don't need to rely on predefined attack signature files to identify attacks.

• They can help to identify attack patterns that can be turned into attack signatures for misuse detectors.

The limitations of anomaly detectors include the following:

• They require more experienced security administrators because the detector can only point out abnormalities, which might or might not be attacks. The security administrator must make the final determination of whether the anomaly is an attack or not.

• They are more likely to produce false alarms than misuse detectors because not all irregularities are actual attacks.

• They require more administrative involvement than misuse detectors.

Response Types

Once an IDS implementation has obtained and analyzed event information that could indicate a possible attack, it generates a response. That response might be sending an alert to an administrator, writing information to a log file, or reconfiguring the firewall or other devices. The response depends on the capabilities and configuration of the IDS. IDS responses can be active, passive, or a combination of the two.

Active Response

An IDS active response (also called active detection) is an automatic action that a system takes when it recognizes an attack. The response might be innocuous, such as increasing information collection; moderate, such as reconfiguring the network; or severe, such as launching a counterattack against the intruder.

Of all the active response types, increasing information collection is the safest because it is unlikely to cause the organization any additional problems. In such a response, the IDS might increase logging activities or the number of packets captured for analysis. The additional information can be used to help determine the identity of the attacker, origination point of the attack, number of targeted systems, and so on. Further, the organization can use the information to pursue legal action against the attacker and assist with forensic analysis (described in Lesson 3).

Reconfiguring the network is a more complex response than increasing logging. The objective is to stop the attack as soon as it is identified. For example, when the IDS detects an attack, it might reconfigure firewall filters, isolate a host on a particular VLAN, or reroute network packets. The benefit is that the IDS might be able to stop an attack in progress. The drawback is that the network might be reconfigured because of a false alarm or an attack on the IDS. If an attacker can determine what your IDS does when it is attacked, the attacker could use that reconfiguration against you in some way. For example, if the IDS simply closes ports when they are attacked, an attacker could attack all ports to shut down a particular device, causing a DoS condition.

Automatic counterattacks are the most dangerous active response because they can bring an organization more trouble than the original suspected attack. The counterattack might be an attempt to probe or disable the intruder's computer or site. However, there are several problems that can arise from such attempts. For example, a counterattack might enable the intruder to avoid legal prosecution for the initial intrusion that led to the counterattack, or the intruder could take legal action against your organization for the counterattack. Further, attackers often spoof the credentials of other people, so the counterattack might further injure an innocent party, which could also result in legal action against your organization.

Passive Response

An IDS passive response (or passive detection) is much less complex than active response. Passive responses leave the response to the intrusion in the hands of the system or security administrator. A passive response involves alerting the administrator that an attack might be taking place. This means the IDS might activate an alarm or send a network alert, e-mail, or page indicating that a threshold was exceeded or an attack signature was identified. The administrator can then determine whether the attack is legitimate and how best to respond.

IDS Deployment

Before you can effectively deploy an IDS, you must understand your network infrastructure and organizational security policies. Also, initial IDS deployments are best done in a staged approach because they are typically high-maintenance at first. Until you learn how to adapt IDS alert thresholds and configuration parameters to your organization, implementing an IDS throughout the organization (large-scale deployment) is inadvisable. The staged approach also gives personnel time to adapt to the new technology, which is especially important for the people who are expected to investigate alerts and monitor IDS logs. For example, consider the following four-stage approach:

1. Deploy a limited NIDS.

   A NIDS can protect multiple hosts from a single location. You can improve the security of your entire network and begin to customize an IDS for your network by deploying a NIDS. If your NIDS implementation comes with a management console, deploy the console before you begin adding sensors. Get comfortable with the reporting, alarms, and thresholds of your NIDS implementation.

2. Deploy NIDS sensors.

   Incrementally deploy sensors throughout your network. Each time you deploy a new sensor, take time to understand the differences in traffic, reporting, logging, and alerts that you receive from that sensor. Figure 11-4 illustrates ideas for sensor placement on an example network (discussed in greater detail later).

   

   Figure 11-4. NIDS sensor placement

3. Deploy a limited HIDS.

   A HIDS can generate many alerts and could be quite difficult to manage if the initial deployment is done on a large scale. Deploy a HIDS only to critical hosts or servers at first. Again, take time to understand the reporting, logging, and alerts that you receive from your hosts. If your HIDS implementation allows for a management console, deploy and configure it before you add additional hosts.

4. Fully deploy a HIDS.

   Once you are comfortable with the HIDS on your critical servers, you can optionally roll out the HIDS to all client systems. Such a rollout makes every host independently capable of recognizing potential attacks. However, many organizations opt not to deploy the HIDS on all systems because doing so could be quite expensive. Wide-scale HIDS deployment usually involves additional software licenses, higher deployment costs, and increased technical support time.

Figure 11-4 illustrates locations (A, B, C, D, and E) where NIDS sensors could be deployed on a sample network. The advantages of deploying these NIDS sensors in each location are discussed here:

• Location A.

  This location is outside the organizational network and perimeter firewall, which means it can document any inbound attack against the organization's network. This is a good place to gather information on possible incoming attacks and provide ongoing justification for new security measures. A sensor in this area could also be used to track outgoing attacks originating inside the organization. However, this area is also highly likely to generate false alarms and could require very high maintenance. Organizations that decide to place NIDS sensors here often configure the IDS to be least sensitive or to simply log potential attacks instead of activating alerts. Many organizations choose not to configure a NIDS at this location because their firewall logs are sufficient for gathering information from this location.

• Location B.

  This location is ideal for a NIDS to protect the perimeter network. Attacks that successfully traverse or bypass the external firewall can be picked up here. A NIDS sensor can help to protect the Web, FTP, and other servers that typically reside in the perimeter network. A sensor here can also detect outbound attacks. These sensors are typically configured for low to moderate sensitivity, because otherwise they typically generate too many false alarms.

• Location C.

  NIDS sensors at this location protect the organization's internal network. They can catch attacks that successfully traverse or bypass the internal firewall. In addition, sensors at this location can detect internal and outbound attacks, and they are typically configured for moderate to high sensitivity.

• Location D.

  A sensor at this location can protect critical servers from compromise. The advantage of placing a sensor here is that it can catch internal and external attacks. Sensors are typically configured for maximum sensitivity at this location.

• Location E.

  A sensor at this location gathers a large amount of internal traffic, which helps to detect potential internal attacks. Sensors at this location are typically configured for moderate to high sensitivity.

NIST Special Publication 800-31 titled "Intrusion Detection Systems (IDS)" provides more detail on many of the topics discussed in this lesson. (NIST articles are available at at http://csrc.nist.gov/publications).

Exercise: IDS Staged Deployment Steps

Place the following staged IDS deployment steps in the appropriate order:

1. Deploy a HIDS to critical hosts

2. Fully deploy a HIDS

3. Fully deploy a NIDS

4. Partially deploy a NIDS

Lesson Review

The following questions are intended to reinforce key information presented in this lesson. If you are unable to answer a question, review the lesson and then try the question again. Answers to the questions can be found in Appendix A, "Questions and Answers."

1. What are the main differences between NIDS, SIVs, and LFMs?

2. What are the benefits of NIDS's?

3. What are some problems with NIDS's?

4. What are some ways that attackers try to avoid detection?

5. What benefits do HIDS's and application-based IDS's have over NIDS's?

Lesson Summary

• Intrusion detection systems (IDS's) can collect and analyze information in different ways. Some analyze information from the network, others from system files, and still others from log files. Many IDS's analyze information from multiple sources.

• IDS's can be network-based, host-based, or application-based. NIDS's are able to protect a larger number of systems and are easier to implement than HIDS's. However, NIDS's are limited by their processing power and ability to decode packets quickly. NIDS's also have trouble with encryption, VLANs, and encrypted tunnels. HIDS's are able to work around encryption and provide better individual host protection. However, HIDS's might be compromised during an attack on the target and might lose valuable information. Application-based IDS's are best for detecting specific attacks on applications and are not limited by data encryption. However, like HIDS's, they can be compromised or disabled in an attack.

• Typical IDS responses are passive, allowing the administrator to take action when an incident occurs. Active IDS responses have different levels of severity. The most benign level is to increase logging. An intermediate level is to reconfigure the network in some way as a response to an attack. This has the drawback of potentially creating a security hole or disrupting normal operations. The most severe active response is to launch a counterattack. This could cause the organization additional difficulties, especially if the counterattack is launched against an innocent party.

• IDS deployment is best done in stages. This allows network staff to customize and become familiar with IDS implementations. The NIDS should be deployed first. Once the NIDS is fully configured and deployed, the HIDS can be deployed to critical hosts. After critical hosts are successfully configured and running with a HIDS, a full HIDS deployment can be contemplated.