Network-Based Intrusion Detection | Extreme Exploits: Advanced Defenses Against Hardcore Hacks (Hacking Exposed)

Network-based intrusion detection systems monitor one or more network segments for indications of an attack. Normally, these devices are deployed either on a switch's span (or mirror) port, or on a conventional hub. A switch's span port will have all traffic observed on other ports directed into it, allowing any device sitting on this port to observe all network activity passing through the switch. Some vendors also have a custom network tap device available, which is used to intercept all network traffic on an incoming segment.

One of the most fundamental changes occurring in the area of network intrusion detection and prevention is the gradual merging of detection and prevention technologies into firewall systems. Sometimes referred to as "deep packet inspection" by firewall vendors, firewalls are increasingly using methods similar to those of intrusion detection systems in order to filter and prevent attacks. This is a logical transition as the functions performed by both slowly begin to overlap and customers are hesitant to deploy multiple inline devices to accomplish the same goal.

Intrusion detection systems have historically been plagued with a variety of accuracy problems. The tendency for intrusion detection systems to falsely identify an attack when none exists is one of them. This is commonly known as a false positive. The tendency for an intrusion detection system to miss an attack, when one has in fact occurred, is better known as a false negative. Both scenarios are problematic but the first presents real problems when attempting to correlate and react to threats that were believed to have taken place, when in fact they haven't. These accuracy problems exacerbate themselves when a device is put inline, and has the ability to actively block attacks. Whereas a passive device generating a false positive would generate an unneeded alert, an active intrusion prevention device generating a false positive will result in the blocking of legitimate network traffic. These accuracy problems are normally the result of overly broad or poorly written signatures.

Passive Intrusion Detection

Passive intrusion detection systems (as opposed to intrusion prevention systems) passively monitor network activity. That is, they are not inline devices, routing packets in one interface and out another, like a switch or router would. They normally have a single interface with which to read network packets. As a result, these devices are effective at detecting attacks; however, they have a very limited ability to stop or prevent an attack once it has been observed.

As a result of their inability to prevent or stop attacks by directly prohibiting the passing of traffic, several other techniques have been used in an attempt to provide some mitigation capability to terminate existing sessions:

TCP Reset Capability When an attack is observed using the TCP protocol, the intrusion detection system will forge a TCP reset (RST) packet in order to terminate the TCP connection. Since the intrusion detection system keeps state by monitoring the TCP sequence and acknowledgment numbers , it is normally successful in doing this by forging packets from both the client to the server, and from the server to the client. Unfortunately, while this will terminate the active connection, preventing further use of this session, it may be too little too late. Since the attack has already passed through to the target server, it would likely have already succeeded at this point.
UDP Termination While UDP is not a connection-oriented protocol, client and server systems normally maintain the notion of a connection through the traditional socket-based interface. By sending an ICMP port unreachable message, this connection is terminated on both servers and subsequent attempts to send data using the same connection will fail. This method has even less effect than it does for TCP, due to the connectionless state of UDP.
Device Reconfiguration Even though the intrusion detection system itself cannot actively block an attack, it can interface with other network devices in order to proactively prevent further attacks. Many intrusion detection systems can support the addition of new rules to well-known firewall systems, or the addition of access control lists to popular routers. Inherent in this, however, is the possibility of inadvertently blocking access to or from a legitimate host while implementing such a rule. This has resulted in the failure of many organizations to adopt this type of scenario.

As a result of these shortcomings, the return on investment of intrusion detection systems has come into question over the past several years . This has resulted in a swift industry move towards putting these devices inline, enabling the ability to both detect and prevent such attacks.

Intrusion detection systems also possess inherent technical shortcomings in their ability to reassemble network traffic. One of the most well-known weaknesses of passive intrusion detection systems is the ability for attackers to circumvent their inspection engines in order to avoid detection. By manipulating the differences between various endpoint TCP/IP stacks, attackers can confuse intrusion detection engines into having a different view of the traffic passing through to an endpoint than the endpoint itself processes. Two primary types of attacks were documented in depth by Thomas Ptacek and Tim Newsham in their paper "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection."

Insertion Attacks This occurs when an intrusion detection system processes a packet that an endpoint host discards. If, for example, the endpoint discards a packet with an invalid checksum, and the intrusion detection system processes the packet, the intrusion detection system now has an inaccurate view of the data associated with a particular session. Using this weakness to insert invalid packets into a session over which an attack is launched can cause the intrusion detection system to miss the attack. The endpoint, meanwhile, drops the offending packet, and the attack succeeds.
Evasion Attacks This occurs when an endpoint system accepts a packet that an intrusion detection system discards. This technique results in the intrusion detection system discarding packets due to an overly strict enforcement of protocol compliance or a lack of understanding of a protocol. Examples of this have historically involved confusing an intrusion detection engine with particular TCP protocol behavior in order to avoid detection.

In theory, insertion and evasion attacks can occur at any protocol layer. In "Insertion, Evasion, and Denial of Service: Eluding Network Intrusion Detection," Ptacek and Newsham give numerous examples of this at both the network (IP) and transport (TCP) layer. By examining open source operating systems such as BSD and Linux, it is easy to identify all situations in which those TCP/IP stacks will discard a packet. Then, by systematically injecting packets meeting those criteria into sessions containing an attack, it is possible to determine whether the intrusion detection engine incorrectly processes these packets, resulting in a false negative. Some rudimentary examples of when this can occur include

IP Checksum If the intrusion detection system does not perform IP header checksum validation, it will accept an invalid packet, even though the endpoint will discard it.
IP TTL Attackers can manipulate the time-to-live (TTL) IP header value in order to cause an intrusion detection system to accept a packet, while the endpoint will discard it due to the TTL being expired . This is particularly difficult to handle when there are multiple hops between the location of the intrusion detection sensor and the endpoint host. The attacker can craft a packet with a TTL high enough to pass through the intrusion detection sensor, but low enough to be discarded before it has reached the endpoint system.
IP Fragmentation By exploiting the nuances of an endpoint's IP fragment reassembly algorithm, an attacker can manipulate a fragment stream to cause an intrusion detection sensor to accept packets, while an endpoint may discard them.

Dozens of insertion and evasion scenarios exist and it is difficult for a given intrusion detection system to completely avoid them. The reason why they exist is that a given TCP/IP implementation from one vendor differs in subtle ways from that of another. For example, Microsoft Windows TCP/IP has subtle differences in how the TCP and IP protocols behave as opposed to Linux. Sometimes implementations from the same vendor will even differ across operating system versions.

To avoid these nuances, some intrusion detection system vendors have taken to target-based analysis. By noting the architecture and operating system of an endpoint, they can now successfully simulate the TCP/IP characteristics of that endpoint. This means that the analysis engine will now accept all valid packets as the endpoint would, and discard all invalid packets as the endpoint would. While this in large part solves much of the insertion and evasion problems, it still does not solve the problem of the device not being inline.

Unfortunately, for the layperson it is difficult to determine a given product's susceptibility to these attacks. Unless some effort is exerted to set up a test network and perform these tests yourself, you must rely on external resources for this validation. Fortunately, the tools to perform some variations of these tests are publicly available and have been used with some success by vendors to make their products more resilient.

Intrusion Prevention

Intrusion prevention systems solve many of the shortcomings of intrusion detection systems, specifically , the ability to proactively block attacks before they have successfully entered a network. While they are successful at doing this, however, they also have an accuracy requirement that was not needed when doing so passively. The risk of inadvertently blocking legitimate traffic presents a dilemma for a technology that has evolved from one that had no such requirement. One of the most common complaints of passive intrusion detection systems was false positives or the mistaken identification of an attack when none existed. With the move now towards the active blocking of attacks based on these same signatures, vendors have had to readdress this false positive problem in order to prevent inaccurate identification.

Intrusion prevention devices sit inline on a network segment, much like a router or switch does, and as such they have the ability to not only process and analyze all network traffic, but also to block attacks before they are passed along. In addition to having this ability, they are less prone to insertion and evasion attacks, as they can strictly enforce protocol correctness and control exactly what an endpoint receives. Intrusion prevention systems can detect attempts to manipulate IP fragments and TCP streams and normalize this traffic before passing it along. By doing this, they can scrub and shape the traffic passing through them and therefore ensure that their view of a network stream exactly matches that of an endpoint system. Since passive systems do not sit inline, they do not have the capacity to do this.

With intrusion detection systems now moving inline, performance and latency become a much larger factor. Whereas passive devices could simply drop traffic if network utilization surpassed their capabilities (and therefore also miss attacks), inline intrusion prevention systems would increase latency and decrease throughput if this occurred.

Types of Network-Based Detection Engines

In the following section we'll discuss many of the common types of network intrusion detection and prevention engines. For both passive detection and inline prevention the technologies remain very similar, the latter adding the ability to block attacks inline.

Signature-Based Intrusion Detection

Signature-based (sometimes referred to as rule-based) intrusion detection systems were the first to appear on the market in the late 1990s. These systems use a database of signatures or predefined patterns to identify specific network traffic that is indicative of an attack. They work by statefully tracking network sessions, while performing signature (or pattern) matching on data present within the session. A match indicates that an attack has been seen. Most systems support some level of regular expressions in order to provide for both broad and narrow scope signatures. Signature-based intrusion detection systems normally do not have an understanding of the underlying protocol that they are examining and simply perform byte-by-byte pattern matching.

Each intrusion detection system vendor has its own method to define what is known as a signature. The early open source Snort sensor used an extremely primitive pattern matching syntax, which over the years has been extended in order to provide much more granular capabilities. Other vendors have developed highly complex custom languages in order to define their rules. Both extremes offer advantages and disadvantages. A primitive rule syntax will have a dramatic performance benefit, while limiting the accuracy and granularity by which a signature can be written. This can lead to an abundance of false positives, witnessed by the early Snort signature set. A more complex rule syntax will benefit from increased granularity and accuracy while requiring more CPU cycles in order to execute. Somewhere in between both of these extremes is a middle ground that most would feel comfortable with.

There is an industry-wide discrepancy in the accuracy and reliability of signatures produced by various vendors. The quality of a signature can vary dramatically based on a number of factors. These include

The granularity of the signature syntax As mentioned previously, this will vary among vendors, and some vendors will simply not have the ability to detect particular threats as accurately as others.
Whether the signature detects a specific exploitation of a vulnerability or the core vulnerability itself While it may be easy to quickly write a signature detecting a particular exploit, it normally requires more research and protocol knowledge in order to detect the core vulnerability, in turn also detecting all variants of an exploit. Also known as virtual patching, the ability to block the exploitation of the core vulnerability can provide protection until an official vendor patch becomes available and is installed. Vendors are clearly moving towards detecting and blocking the core vulnerability; however, the art of signature writing in many cases remains just thatan art.
The author of the signature and the protocol knowledge he possesses For example, an author who has in-depth knowledge of a particular protocol may produce a superior signature to one who has less knowledge.

While signature-based intrusion detection systems are great at detecting known attacks, they are normally unable to detect attacks against new or unknown vulnerabilities. Frequently called "zero-day attacks," there has been a large movement of intrusion detection vendors claiming to protect against them. Zero-day attacks can best be defined as attacks that exploit a vulnerability that neither the vendor nor the public is aware of, and as a result there is no patch or workaround available. Threats exploiting zero-day vulnerabilities have been predicted for some time. Fortunately, almost all threats that have surfaced as of this writing (with the exception of a handful) have had their associated vulnerability publicly known prior to the threat surfacing , allowing the development of a signature to protect against future exploits.

Protocol Anomaly-Based Intrusion Detection

Protocol anomaly-based intrusion detection systems work by looking for abnormal or anomalous activity without the knowledge of a specific vulnerability. Protocol anomaly-based systems have the benefit of understanding the underlying application layer protocol that they are examining. They are capable (to varying extents) of parsing individual protocol options and fields, and then evaluating the value and contents of those fields. There are two primary methods that are used by protocol anomaly systems in order to identify an attack.

Protocol Compliance Protocol compliance ensures that a protocol conforms to a predefined specification such as an associated RFC. While this may sound effective, and in many cases it is, it is not unusual to see protocols deviating from their documented standard. In the real world, protocols are implemented by people, and as such, the implementer often takes liberties when interpreting such a standard. Much like the TCP/IP implementations discussed earlier, application layer protocols may differ subtly from one vendor to the next , but work just well enough to be interoperable. Enforcing strict protocol compliance in those situations can result in a false positive. Another reason why enforcing protocol correctness is not always effective is that many attacks fit very well into a protocol's standard, and would not be detected anyway. Also, for many proprietary protocols no published standards exist, making protocol validation difficult without completely reverse-engineering the protocol. Even then, without vendor input it would be difficult to infer the acceptable protocol behavior.

Preemptive Analysis Preemptive analysis involves identifying abnormal characteristics of a protocol request in advance of their appearing in an attack. This involves having an understanding of the underlying protocol, and identifying situations that are either already suspicious or likely to be used in a malicious manner in the future. In some cases these can be specific protocol options that conform to the RFC or specification, but can be used for insidious purposes (for example, a DNS packet with an invalid length field). In other cases they can be situations where specific protocol fields contain values that should never be seen in the course of normal operation. An example of this is a text-based protocol containing binary characters when this does not occur in normal operation and is likely indicative of an attack.

This type of preemptive analysis can be useful in a situation where a zero-day threat may surface and the protocol in question had previously been analyzed in order to identify all potential exploitation vectors. Normally, while these types of protocol anomaly checks will not be as precise as a signature, they are sufficient to temporarily block an attack.

Statistical Anomaly Detection

Statistical anomaly systems vary from protocol anomaly systems as they use a broad statistical approach in order to identify abnormal activity. By baselining or profiling the frequency of individual requests , or broad activity in general, they can identify a deviation from that norm. While this approach is not effective at stopping individual attacks, it can detect widespread events such as worm infestations that result in an excess of network traffic. An initial baseline must be gathered to build a profile of what is considered "normal." After this, statistical deviations from this norm can be identified and alerted on as anomalous.

One of the primary shortcomings with any statistical-based approach is that if attacks are already occurring during the initial learning period, they will be factored in the baseline or learned profile.

Since statistical anomaly detection provides a broad detection mechanism, it is normally used to supplement more thorough inspection mechanisms such as signature- and anomaly-based systems.

Behavior-Based Intrusion Detection

Behavior-based intrusion detection systems work by building a profile of normal network interactions, and then detecting changes from that baseline. When looking at these network interactions, the following types of variables can be tracked:

The source and destination systems that are communicating. This may be in the form of the IP addresses or even Ethernet addresses.
The operating system types associated with those systems, through either active or passive operating system fingerprinting.
The time of day at which the communications are normally occurring.
The transport layer protocols being used during this communication, for example, TCP and UDP.
The application layer protocols that were observed, and the specific versions of those protocols.
The protocol request types being used, such as the specific command types being sent by the client to the server.
Additional application layer options such as command arguments, for instance, file names or account names that may be observed in a request.

Detecting changes based on these variables provides insight into new assets and communications surfacing on the network, as well as assets becoming dormant . They point out abnormal communications that may be indicative of an attack, and also serve as record-keeping mechanisms for forensic purposes in the future.

Much like the shortcomings of the statistical anomaly model, however, it is possible for malicious activity to be learned into a base profile. When using either method, it is important for a product to provide the ability to review any learned behavior, in order to identify any potentially suspicious behavior before accepting a baseline.

Denial-of-Service Attack Mitigation

Early in February of 2000, the Internet saw a number of widely publicized distributed denial-of-service (DDoS) attacks against prominent Internet search and e-commerce sites, including eBay, Amazon, e*Trade, and CNN. Since that time, a number of solutions have appeared in order to protect against denial-of-service attacks. These attacks can immobilize an entire organization by leveraging a sufficiently large number of "Zombie" systems to launch a coordinated distributed attack.

These attacks can range from saturating an organization's network with unsolicited packets to inundating its web infrastructure with spurious requests. A variety of protocols can be used to launch such an attack, and the defenses for each vary accordingly . Today, denial-of-service mitigation devices work in one of two ways:

By profiling the base level of traffic that an organization normally sees passing into their perimeter on a given service, and then detecting a substantial increase in those levels.
By looking for an abundance of " half-open " or unestablished TCP sessions towards a particular server, indicative of TCP SYN requests that have been spoofed from nonexistent systems.

Denial-of-service mitigation devices come in a variety of flavors. There are passive carrier-scale solutions and inline carrier/datacenter filtering devices, as well as lower-priced switches that do inline filtering.

High-capacity carrier scale solutions normally work by collecting flow data from an organization's routers and switches. Flow data consists of records representing individual streams that are passing through a particular device. By statistically baselining this flow traffic, a pattern of regular behavior can be learned. Using this approach, the behavior of hundreds or even thousands of network segments can be learned effectively. Deviations from this baseline can then be detected, and the appropriate actions can be taken. In addition to the collection of flow data, some devices also learn or collect routing data (such as BGP) in order to discover and track the topology of the network they are monitoring. This can later be used to subvert or blackhole an attack while it is in progress so as not to impact others on the network. A blackhole is created by such devices by modifying the appropriate router's routing table in order to either drop the traffic destined to the target prefix outright , or redirect it to a quarantine network for analysis.

In addition to commercial solutions, there are a variety of freely available tools that will allow you to analyze the flow data within your networks. Two such tools, cflowd and flow-tools , can be obtained at the following locations:

http://www.caida.org/tools/measurement/cflowd/
http://www.splintered.net/sw/flow-tools/

Much like the difference between intrusion detection systems and intrusion prevention systems, DDoS mitigation devices can be either passive or inline. As described previously, passive devices can also be reactive by reconfiguring network infrastructure. Inline devices, however, work by directly shunning the associated traffic as it passes through them. They may take a statistical approach, much like passive systems do, in learning the baseline traffic levels of an organization, and then detecting deviations from that baseline. Inline devices are much more limited in their ability to process data on high-speed networks, and as a result are found more on individual segments beyond the carrier, within an organization.