Chapter 9: Intrusion Detection Systems

Overview

The goal of the network firewall is straightforward. A network firewall protects one part of the network from another by allowing or denying traffic based upon a number of criteria. With security, however, there is always a lingering doubt. How can you know that your firewall is doing its job? How do you know that you have configured the firewall properly in the first place, and how can you know that your firewall is not letting through attacks that you had not foreseen when you first configured it? The device that has traditionally served this purpose is known as an intrusion detection system (IDS).

If a firewall is the lock on your door, the IDS is the burglar alarm. The firewall is the network element that actually provides the protection. Should that protection fail, however, the IDS sounds the alarm.

What an IDS attempts to do is make an evaluation of each of the millions of packets that it might see in a normal working day. For each packet that is examined, the IDS logic determines if the packet is a "good" packet — in other words, something that might normally be seen on the network — or a "bad" packet. In this case, "bad" can mean virtually anything but normally will be construed as meaning something that should not, for whatever reason, be seen on the network. This might even mean a normally "good" packet that is seen at a "bad" time.

The actions to take when an IDS sees a good or bad packet can vary. Because it is assumed that the majority of traffic seen on a network will be good, only "bad" packets are normally attended to by the IDS. The IDS may elect to totally ignore the bad packet, log the bad packet to a log for later review by an administrator, or sound the alarms and let every one it has the ability to alert know that a particularly "bad" packet has been seen on the network.

The determination of what is a "bad" packet and what is a "good" packet has been the subject of endless discussion and a number of patents and products. To understand the choices that we can make when sorting the good from the bad, let us create an IDS of our own.

At the most basic level, an IDS is really nothing more than a packet sniffer. That is, it is a program that monitors the transmission medium and records what it detects for further analysis. So, we will start our examination of the IDS with a packet sniffer on the network. Where to place this packet sniffer for maximum effect is a subject of later discussion. For now, we are just going to plug our fledgling IDS into a hub.

For starters, we simply capture all packets on the wire and log them. After a couple of days, we realize that there are megabytes and megabytes of text data for us to sort through. In an effort to make our jobs somewhat saner, we attempt to filter the data to some extent. We can do this in several ways. The first thing we try to do is enter in all the types of "good" packets. We do not get very far before we realize that there sure are a lot of different types of good packets out there. We quickly change course. Instead, we try to find some sort of distinguishing characteristic of a number of bad packets so we search the Internet and research journals for information on various network attacks. For each attack we find, we create a matching filter in our new IDS. For example, if we notice that IP packet fragments have over-lapping fragment identifiers, we include in our "bad" filter instructions for the IDS to record whenever it sees IP packets that match that behavior. If we know that we should never see an IP packet from the IP address 0.0.0.0, we add that information to our filter.

While this may take some time, the number of bad packets out there is fairly small compared to the number of good (or at the very least "not bad") packets we are likely to see on a network. Satisfied with our work, we reinstall the IDS and turn it on. Right off the bat, we notice that something is wrong. We discover a number of alarms going off — the IDS is detecting bad packets on our network right now. Leaping into action, we seek the source of this attack. After some exhaustive research and digging, we find out that some of our "bad" packets, as far as our IDS is concerned, look a lot like some good packets we normally have on our network. To respond to this, we spend several days or weeks tuning the IDS. Each time bad packets are detected, we investigate to see if they are really bad packets or just normal network traffic that has the appearance of bad packets to our IDS. Over time, our enthusiasm for the brand-new IDS begins to fade as we spend time tracking down bad packets. We may get so used to these false alarms that we may even shut off the alarm system. The odds of this happening are directly related to the number of times our pager has gone off in the middle of the night when another "bad" packet is detected.

After a break-in period, we begin to have faith in our IDS again. We note that the number of false alarms has dramatically decreased and we even catch a real honest-to-goodness bad packet on the network that we track down and eradicate. This is what makes our jobs fun and we feel that we are making a difference. As time goes by, we continue to add new rules to the IDS, saying, "look for this in a packet, it's bad!" or "look for that in a packet, it's bad too!"

Our confidence is suddenly shaken, however, as to our dismay we learn that our network had been hacked — bad. There are Trojans on several servers and host PCs, and we learn of packet sniffers that have also been installed by the Trojan software. We begin the process of restoring everything up to the last-known good backup; but because we do not really know when this attack occurred, we mostly end up installing everything from the original media. That is a long and depressing weekend, to be sure. We keep asking ourselves, where did we go wrong? Our firewall was up and running and the IDS did not find a single bad packet that we could attribute to this break-in.

That Monday, reading the information security news, we find that there is a new attack that is finding its way around the Internet — and the attack looks just like what we had been hit with. Everyone is highly recommended to update their systems and install the new rules to detect this attack onto their IDS. Now the rules are available for downloading, but just a bit too late for our network.

We have learned the hard way that our IDS, which is based on the signatures of attacks in packets, suffers from a major flaw. If there is no attack signature in the IDS, then the attack will not be recorded. By its nature, our signature-based IDS is reactive. We only know about the attacks after someone else had been attacked, noticed it, and informed the rest of the Internet community of what they found. This time, we were part of the unlucky group that gets to discover a new attack.

A bit wiser, we seek to improve our IDS. After all, it has helped us out on a few occasions, especially after we took the time to tune it properly. We decide to change our focus a bit. What if, instead of thinking of everything as good and bad packets, we instead start thinking of user behavior instead of packets? The idea is that normal users are pretty predictable. They generally use the network in the same way day after day. Our servers are the same. We have learned when creating our first attempt at an IDS that, on our network at least, there are some things that just happen all the time. We decide to categorize user traffic as "normal" and "abnormal" traffic. Tasks that users and network hosts perform all the time are categorized as "normal." Traffic that is not part of the "normal" group is considered "abnormal" and will generate an alarm or log a report.

The piece de resistance, however, is that instead of going through the arduous process of logging normal and abnormal traffic and entering it into the IDS, we will let the IDS itself learn what is normal and what is not normal. By monitoring traffic, we learn that we can statistically predict what type of traffic we should see between any two hosts on the network. The longer we monitor, the more accurate it becomes. To cut down on the amount of tuning we need to do, we define normal to the IDS as a small range of options. That is, we give the IDS some leeway in its definition of normal. This is so that every time users do something just a bit different, they will not summon the information security team to their desk. What is considered normal, then, is actually a small window of behavior on either side of normal as statistically defined by the IDS.

We also realize that this approach will protect us in the future against new attacks. Presumably, a new attack will set off alarms indicating that abnormal traffic had been detected, which is exactly what we would expect based on the fact that this attack would have traffic that would be considered new and not part of the statistical database of established network traffic patterns.

So the new IDS is configured and installed and we are confident in our ability to detect attacks on our network and now respond to them. We are until we sit down for lunch with a lecturer at a security conference. This expert points out that our definitions of normal and abnormal are based on the observed behavior of network traffic. We are troubled by two of her points. The first is that how do we know that someone has not actually attacked our network before we installed the IDS? They could have subsequently taught the statistical IDS that their hacking was actually normal traffic on the network. Furthermore, as a sour taste that was not part of the lunch begins to rise from the back of our throats, what if someone was patient enough and careful enough that he slowly retrained the IDS? What if he changed the behavior slowly enough and ended up changing what the IDS thought was normal? His attack could go completely undetected by our new system.

In short order, we are reminded of the general law of security. No solution is perfect. Given time, money, and motivation, any security system can be compromised. Wiser, we begin looking for another solution. We eventually strike on the idea that combines the best elements of both our ideas. We notice that most attacks are based on illegal network protocol usage. For example, we do not normally see a URL GET request (how we request a Web page) with a long string of../../../../ in it. If we do, that is normally someone trying to attempt a directory traversal. We go back to our original idea that if we simply define all the normal ways in which we would see a protocol in action, we can conclude that abnormal behavior is some sort of attack.

Eagerly, we set to coding this new IDS application. Because there is so much variation in protocols, we decide to refer to the standards in writing our protocol rules. We teach the IDS what normal protocol operations are, according to the RFC documents that define them, and configure them to alert us of any packets that violate this behavior. Shortly after installing the new IDS, we are beset with alerts of all types. Surely our network has not been under attack all this time, has it? Given our experience, we know now that we should expect some sort of tuning period to adjust the IDS sensors to our specific network environment, but the amount of alerts we are seeing far surpasses our expectations.

As we will learn shortly, observing the alerts and cross referencing traffic with a protocol analyzer, a great deal of vendor protocols do not actually work the way that the RFCs specifically define them. For example, we find that our Microsoft Exchange e-mail server uses SMTP commands in a very non-standard way. This time, to tune our IDS, we need to observe and program the IDS with all the non-standard protocol behavior that occurs transparently on our network all the time. The task seems daunting but we eventually accomplish it.

We sit down to lunch with our new security friend, beaming of our latest success, only to be dashed by her overly pessimistic (in our opinion) view that some attacks do use normal protocol behavior — for example, logging in with a stolen password at odd times or from odd locations. Back to the drawing board.

This short fable points out that IDSs, as they currently exist, are incomplete at best. No single model of IDS operation is able to conclusively detect attacks in progress all the time. They can also require a considerable amount of resources in the initial setup and tuning, because, no matter the product or the manner in which they attempt to detect attacks, an IDS will always require a learning period to learn the specific needs of your network. At the same time, they are essential tools for modern networks. As discussed above, IDSs are currently constructed to operate in one of three major ways. Each is described below in detail.