Network Intrusion Detection Basics

Network intrusion detection systems are designed to sniff network traffic and analyze it to identify threats in the forms of reconnaissance activities and attacks. By detecting malicious activity, network intrusion detection enables you to identify and react to threats against your environment, as well as threats that your hosts might be directing at hosts on other networks. (Many network IDSs today also offer intrusion prevention capabilities, which means they can stop detected attacks. See Chapter 11, "Intrusion Prevention Systems," for much more information on network-based intrusion prevention.)

Although the focus of this chapter is network IDS, there is also a related IDS technology known as host-based intrusion detection. Host-based IDS software focuses on detecting attacks against a particular host, such as a workstation or server, and is run from the host itself. There are several types of host-based IDS software products, including log analyzers and file integrity checkers. Log analyzers monitor operating system and application logs, looking for entries that might be related to attacks or security violations. File integrity checkers alert you if particular files are altered, which might indicate a successful attack. You can learn more about host-based intrusion detection products in Chapter 10, "Host Defense Components."

The Need for Intrusion Detection

Many people have heard of intrusion detection, but they might not realize why they need to deploy it in their environment. Without intrusion detection, you may be unaware of many attacks that occur. Because you don't have information about attacks, when a successful one occurs, you won't have the information you need to stop it from happening to that host again, or to other hosts. Most ominously, you may never know about an attack that doesn't damage your host, but simply extracts information, such as a password file. Without intrusion detection, you will be unaware of these events until it's much too late. This is why intrusion detection is beneficial in many environments.

Many attacks involve multiple steps or phases. For example, an attacker might start by launching a scan that sends a DNS query containing a version.bind request to each IP address in a range. Some DNS servers respond to such a query with their BIND version number. The goal of this scan is to identify which hosts are DNS servers and what versions of BIND they are using. Based on the results of the first scan, the attacker then sends a specially crafted DNS query to some of the DNS servers to exploit a buffer overflow vulnerability in a particular version of BIND. This second set of queries could occur hours or days after the first setor it could occur within seconds. It depends on the tools being used and the attacker's methodology.

If any of the buffer overflow exploits are successful, the attacker might be able to perform unauthorized actions on one or more of the servers, potentially gaining administrator-level privileges. An attacker who gains privileged access to a server could do many things, including using that server to launch attacks against other hosts on the network and against external hosts. It is important to understand that this is not a hypothetical attack scenario we are describing. Various attack tools use this exact technique to scan, attack, and compromise servers; many worms also do this by spreading throughout the Internet on their own. Sometimes it only takes seconds from the initial scan until the vulnerable host is compromised.

If you are not using intrusion detection, how will you know when a successful attack has occurred? Some attacks are immediately visible, such as a website defacement; others are not. Perhaps you will look through a directory and see some strange files or directories. Perhaps your server's performance will decline and you will find many unknown processes running. Perhaps your server will crash and you won't be able to reboot it. Or perhaps no one will notice that anything has happened.

A properly configured, robust IDS can play more than one role in identifying typical attacks. An IDS can detect reconnaissance activity that may indicate future targets of particular interest. It also generates alerts for the subsequent attempts to breach host security. Alerts are usually generated through one of two methods. The first method, anomaly detection, relies on statistical analysis to identify traffic that falls outside the range normally seen in this environment, or it relies on protocol analysis to identify traffic that violates protocol standards or typical behavior. The second method, signature detection, identifies known attack signatures observed in traffic. Although most network IDS products throughout the years have been signature-based, anomaly-based products are growing in popularity as a complement to signature-based products. The following sections describe the anomaly and signature detection methods.

Anomaly Detection

Although there are various ways to detect anomalies in network traffic, many IDS products based on anomaly detection methodologies work by establishing baselines of normal network activity over a period of time, then detecting significant deviations from the baseline. For example, a product could monitor a network for two weeks to identify which network services are provided by each host, which hosts use each service, and what volume of activity occurs during different times of the day and days of the week. A month later, if the IDS sensor sees a high volume of traffic involving a previously unused service on a host, this could indicate a distributed denial of service (DDoS) attack against the host or a compromised host providing a new service. This class of product is sometimes known as a DDoS attack mitigation system, because it not only detects the DDoS attacks, but also acts to stop them through intrusion prevention techniques (as described in Chapter 11). Examples of DDoS attack mitigation systems include Captus IPS, Mazu Enforcer, and Top Layer Attack Mitigator.

One obvious drawback of this type of anomaly detection is that the baseline needs to be updated constantly to reflect authorized changes to the environment, such as the deployment of a new server or the addition of another service to an existing server. Some products permit the baseline to be updated manually, at least in terms of identifying which hosts are authorized to provide and use certain services. If the baseline can be kept current, or if the environment is quite static and the baseline changes rarely, anomaly detection can be extremely effective at identifying certain types of attacks and reconnaissance activities, such as port and host scans, network-based denial of service attacks, and malicious code (particularly worms). Unfortunately, this type of anomaly detection cannot identify most other types of attacks, so it is best used to complement other IDS technologies.

Many signature-based IDS products perform some type of protocol anomaly detection. This means that the IDS compares traffic to the expected characteristics for widely used protocols, such as HTTP, DNS, and SMTP. When a serious discrepancy is discoveredfor example, a header field that contains hundreds of binary values instead of the expected small number of text charactersthe IDS generates an alert that an anomaly has been discovered. This can be very effective at identifying certain previously unknown instances of attacks that a purely signature-based IDS could not identify.

One significant limitation of anomaly-based IDS is that generally it cannot determine the intent of an attackjust that something anomalous is occurring. Analysts need to study the data captured by the IDS to determine what has happened, validate the alerts, and react appropriately. However, because network and protocol anomaly detection based methods can detect attacks that signature-based IDS cannot, a robust IDS solution for an enterprise should incorporate both anomaly-based and signature-based methods if resources permit.

Signature Detection

A network IDS signature is a pattern you are looking for in traffic. When a signature for an attack matches observed traffic, an alert is generated, or the event is otherwise recorded. A simple signature example is one that detects the Land attack; the source and destination IP addresses in a packet are the same (http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-1999-0016). Some older operating systems could not properly handle such packets, which violate standards, so attackers would send crafted packets with the same source and destination addresses.

Although some network traffic signatures, such as ones for the Land attack, are quite simple, others are considerably more complicated. Many signatures are protocol or application specific; for example, many signatures pertain only to DNS traffic. Because DNS zone transfers traditionally use TCP port 53, a signature pertaining to a zone transfer vulnerability includes the TCP protocol and also needs to look for destination port 53. Each signature specifies other characteristics that help to identify the vulnerability that the attacker is trying to exploit. Some IDSs focus on long sequences of code from published exploits, whereas other IDSs actually perform full protocol analysis, examining and validating the header and payload values of each packet. Although full analysis is more resource intensive, it tends to produce a much more robust signature solution.

Most intrusion detection vendors release new sets of signatures regularly. If a major new vulnerability is discovered or an exploit is widely seen, intrusion detection vendors quickly research the exploit or vulnerability, create new signatures for it (or determine that existing signatures already match it), and release the signatures to their users as soon as possible. This process is similar to the process that antivirus software vendors follow when addressing threats from new viruses, worms, Trojans, and other types of malicious code. When a major new threat emerges suddenly, such as the Slammer and Netsky worm, IDS vendors typically have a signature available for their customers in a matter of hours.

How Signatures Work

Let's look at an example of a signature for the Nimda worm. Because the worm uses a common text-based protocol (HTTP) and uses a relatively simple exploitation technique, it's an ideal subject for understanding the basics of signatures. When the Nimda worm tries to infect a web server, it sends a set of HTTP requests, including this one:

 GET /scripts/..%c0%af../winnt/system32/cmd.exe?/c+dir 

The purpose of this particular request is to exploit a Unicode-related vulnerability in certain unpatched Microsoft IIS servers to gain unauthorized privileges. %c0%af is the Unicode equivalent of a slash, so this command is actually trying to traverse the root directory to exploit the vulnerability. Although no damage is done to the server, success at this point tells the worm that the server is vulnerable. The worm can now launch a damaging attack to this server.

You want your network intrusion detection sensors to notify you when they see this traffic. Many different possible signatures exist. A simple text-matching signature would look for /scripts/..%c0%af../ in a URL (more precisely, in the payload of a TCP packet that a client sends to a server's port 80). Because the Nimda worm issues several GET requests with slightly different values, you would need a separate signature for each one of them if you wanted to identify each attack attempt.

Some IDS sensors have a much more robust way of identifying these types of requests as attacks. Rather than doing simple text matching, they actually decode the request, substituting a slash for the %c0%af sequence, and then analyze the validity of the decoded URL. As a result of the analysis, the IDS sensor determines that the attacker is trying to go past the root directory, and it generates an alert. Although this method is more resource intensive, it provides a much better detection capability than the simple text-matching method because it actually examines traffic for the attack technique, not for specific known instances of that technique being used.

False Positives and False Negatives

Signature development is always a balancing act. A specific signature might be extremely accurate in identifying a particular attack, yet it might take many resources to do so. If an attacker slightly modifies the attack, the signature might not be able to identify it at all. On the other hand, a general signature might be much faster and require far fewer resources, and it might be better at finding new attacks and variants on existing ones. The downside of a general signature is that it also might cause many false positiveswhen a sensor classifies benign activity as an attack. Another factor is the original release date of the signature; over time, false positives are identified by the IDS vendor and corrected, leading to the release of higher-quality signatures. Brand-new signatures often generate high numbers of false positives.

Every intrusion detection system generates false positives. When you first look at an IDS console and see dozens or hundreds of alerts, you might think that your systems are under massive attack. But those alerts are much more likely to be false positives. By selecting a solid IDS product and tuning your sensors, you can reduce false positives, but you can't completely eliminate them. No matter how precisely a particular signature is written, there's still a chance that some benign traffic will accidentally match that signature. False positives are a headache for an intrusion analyst because you have to spend time and resources determining that they are, in fact, false.

The Intrusion Detection System Is Possessed!

I will never forget the first time I had to review and analyze intrusion detection alerts. I opened the IDS console to review the data, and my mouth dropped as I saw hundreds of alerts that all said "Devil." I scrolled through screen after screen of Devil alerts, and I started to wonder if the IDS was possessed! The Devil turned out to be an alert for the Devil Trojan. After several hours of research, I determined that the alerts were false positives caused by a poorly written signature.

A false negative occurs when a signature fails to generate an alert when its associated attack occurs. People tend to focus on false positives much more than false negatives, but each false negative is a legitimate attack or incident that IDS failed to notice. False negatives get less attention because you usually have no way of knowing that they have occurred. Sadly, you're most likely to identify a false negative when an attack is successful and wasn't noticed by the IDS, or when a different type of network IDS sensor or host-based IDS product notices it. In addition, if a signature generates many false positives, the analyst usually shuts off the signature altogether or ignores all the corresponding alerts. This means that the analyst doesn't review legitimate instances of the alert, and this also becomes a type of false negative. It's important to understand what causes false positives and negatives so that you can choose an IDS that minimizes them.

Developing Signatures That Minimize False Positives and Negatives

To better understand false positives and negatives, let's look at a simple example. Suppose that in your intrusion detection system, you write a signature that looks for cmd.exe anywhere in a URL. (cmd.exe is often used in root traversal exploits against IIS servers, such as the Nimda worm.) This is a general signature that matches many different attack attempts. Unfortunately, it also matches cmd.exe-analysis.html, a web page that contains an analysis of cmd.exe attacks. It matches against nascmd.exe as well, which is unrelated to these exploits. These are examples of false positives caused by an overly general signature.

Because of the false positives, you decide to rewrite your signature to use some contextual information as well. Now your signature is a URL that contains /winnt/system32/cmd.exe. Because this is a more specific signature, it triggers fewer false positives than the more general signature.

Unfortunately, reducing false positives often comes at the expense of increasing false negatives. If an attacker uses a URL that includes /winnt/system32/../system32/cmd.exe, the more specific signature misses it, causing a false negative to occur. However, the more general first signature would have sent an alert. And the more robust method described earlier, which decodes the entire URL and evaluates it for root traversals, would have sent an alert for both versions. More complex signatures are usually better than simple text-matching signatures, which are unable to identify most variations on attacks. In fact, some attackers purposely craft their attacks to avoid detection by simple signatures; this is known as IDS evasion.

Detecting IDS Evasion Techniques

As if developing an IDS signature that minimizes false positives and false negatives isn't difficult enough, you must also consider the effects of IDS evasion techniques on signature development. Attackers have many methods of altering their code to avoid IDS detection. Many are simple, such as the example discussed previously that obfuscated the path by including .. sequences. Another technique used in URLs is replacing a character with its hex or Unicode equivalent. Therefore, cmd.exe could become cmd.%65xe because %65 is the hex representation of e. Only IDS products that perform hex decoding before performing signature comparisons would determine that this string matches cmd.exe.

Some evasion techniques actually relate to the IDS sensor's core architecture, not specific signatures. One such technique is to fragment a packet so that it has a small initial fragment; the attacker hopes that the IDS examines only the first fragment, which looks benign, and ignores the remaining fragments, which contain malicious content. An IDS that does full-packet reassembly is not fooled by this evasion method.

Avoiding Unwanted Alerts

Besides false positives, you might also have alerts that address legitimate problems that you simply don't care about. Unwanted alerts seem to occur in just about every environment. For example, if your environment is an ISP or a wireless company, you are providing network services to customers. You probably have a lot of traffic passing on your networks between your customers and other outside hosts. If you are monitoring these networks with IDS sensors and you tune them to identify all possible scans and attacks, you are likely to be overwhelmed with a huge number of alerts that you honestly don't care about. Do you really want to know if one user sends another a FOOBAR.TXT.EXE file? This could trigger an alert because of the suspicious extension combinationand the alert would almost certainly be accuratebut the activity is outside the scope of your control.

Note

Tuning your sensors to produce only the information relevant in your environment is critical if you're going to get the best results from your IDS solution.

Alerting, Logging, and Reporting

When an intrusion detection system sees traffic that matches one of its signatures, it logs the pertinent aspects of the traffic or generates an alert, depending on the severity of the activity and the configuration of the IDS sensor. Alerts can be delivered to intrusion analysts in a variety of ways, including messages on the analyst console, emails, and SNMP traps, depending on the product being used.

Reporting is another key element of intrusion detection. If you have only one sensor, reporting should be simple. If you have dozens of sensors, you probably don't want separate reports from each of them. Instead, you want all data to be collected in one place because it's more convenient to view and easier to back up. It's also far easier to correlate events that all sensors see. Reporting formats vary widely among IDS products; most also permit sensor alerts and logs to be exported to other databases for further queries and reporting.

Review IDS Sensor Status Regularly

I was helping an intrusion analyst tune the dozens of network IDS sensors deployed in his environment. He told me he had noticed recently that the number of alerts he was seeing on his analyst console had dropped significantly. After a few days of this, he noticed that several sensors weren't generating alerts. Puzzled, he finally checked the sensor status reports and discovered that none of them were seeing packets. An investigation revealed that a co-worker had reconfigured the switches to which the sensors were connected.

Check the status of your sensors regularly. Most IDS sensors provide some sort of status report that indicates how the sensor is functioning and what volume of traffic it is seeing. If you notice unexpected drops in traffic levels, you need to investigate the situation as quickly as possible; your sensor might no longer be seeing traffic.

Intrusion Detection Software

Many different network IDSs are available, each with its own distinct features. Some of the best-known and most popular products include Cisco Secure IDS, Enterasys Dragon Network Sensor, ISS RealSecure Network Sensor, NFR Sentivist IDS, Snort, and Sourcefire Intrusion Management System. They all provide alerting, logging, and reporting capabilities. Signature sets are available through a variety of means, depending on the product. Users of some products have written their own signatures and made them publicly available. Other products must be purchased to get signatures; some vendors hide the signature details, whereas others allow users to view and even modify their signatures.

Network IDS sensors have other helpful features. For example, some systems can perform network monitoring and traffic analysis, collecting statistics on connections, protocols in use, and the like. This capability can be invaluable in identifying policy violations, such as unauthorized services in use, and traffic oddities in general. We will discuss policy violations later in this chapter. Another feature covered in more depth later is that of intrusion prevention, in which an IDS sensor works with other network devices or acts on its own to halt certain traffic from occurring. You can learn more about such features in Chapter 11.

Some IDS products provide tiered solutions, which typically involve deploying multiple IDS sensors on various network segments. Each sensor transmits its IDS data to a centralized server that stores all the data. The individual sensors or the centralized box, which is able to correlate events from all the sensors' data, can make alert decisions. Many IDS solutions also have separate analyst consoles that connect to the centralized system or the individual sensors; these enable analysts to view recorded data and alerts, as well as reconfigure the sensors and update signatures. Some software vendors, such as ArcSight, e-Security, GuardedNet, Intellitactics, and netForensics, offer console products that can process and correlate data from various brands of IDS sensors, firewalls, and other hosts; Chapter 20, "Network Log Analysis," discusses such products.

Intrusion-Related Services

Besides intrusion detection products, some intrusion detectionrelated services might be able to assist you with the analysis of your intrusion detection data. Some organizations choose to outsource the analysis of their IDS sensor data to specialized commercial monitoring services. Other organizations elect to do their own analysis, but also submit their data to free distributed IDS services for additional help.

Distributed IDS Services

As IDS products have become more popular, the use of free distributed IDS services has also grown significantly. The basic concept is that administrators from organizations all over the world submit logs to a distributed IDS service from their own IDS sensors, firewalls, and other devices. These services analyze the data from all the sites and perform correlations to identify likely attacks. Two of the best-known distributed IDS services are Symantec's DeepSight Analyzer, located at http://analyzer.symantec.com, and DShield, located at http://www.dshield.org.

Using a distributed IDS service has a few benefits. Because the service receives logs from many environments, it has a wealth of data to use for its intrusion analysis. A distributed IDS service finds patterns among all the attacks that individual sites cannot see. Another great feature of some distributed IDS services is that if they conclude that a host at a particular IP address attacked a substantial number of sites, they will email the technical contact for that domain so that the activity can be investigated. Distributed IDS services can reduce some of the burden placed on intrusion analysts by analyzing logs, correlating events, and following up on highly suspicious activityessentially providing another tool that can help analysts.

Outsourced Intrusion Detection System Monitoring

Many companies, such as Counterpane, ISS, and Symantec, offer managed security services that include IDS monitoring capabilities. This generally entails 24-hour remote monitoring of your IDS sensors, analysis of IDS data, and rapid notification of your staff in case of serious attack or IDS failure. One advantage of using these services is staffing related; they can provide trained intrusion analysts to monitor your sensors at all times. Another advantage is that because these services collect data from so many networks, they can correlate attacks among them. The disadvantage is their lack of knowledge about your environment and their inability to correlate events unless they can access firewall and host log files in near real time. (In addition to outsourcing IDS monitoring, many organizations also outsource the monitoring of other security event sources, such as firewalls and antivirus servers; in such cases, correlation is generally not an issue.)

At best, outsourced IDS monitoring can identify serious attacks and help your organization respond to them quickly and appropriately. If your organization doesn't have the time or expertise to analyze its own IDS sensor data properly and promptly, outsourcing that work is important for maintaining a strong perimeter defense. At worst, such services add little or nothing to the existing IDS solution, while requiring substantial financial commitments. Because outsourced IDS monitoring is very resource-intensive, it can easily cost millions of dollars a year for larger organizations.