Signature Types

Signatures fall into one of the following two basic categories depending on their functionality:

• Atomic signatures

• Stateful signatures

This section examines these signature types in further detail. Furthermore, the triggering mechanisms explained later in this chapter can be used with both of these base signature types. The major distinction between these two base signature types is whether or not the inspection process requires the IPS device to maintain state about previous actions that have been observed.

Atomic Signatures

Atomic signatures represent the simplest signature type. For an atomic signature, a single packet, activity, or event is examined to determine if the signature should trigger a signature action. Because these signatures trigger on a single event, they do not require your intrusion system to maintain state. The entire inspection can be accomplished in an atomic operation that does not require any knowledge of past or future activities.

State

State refers to situations in which you need to analyze multiple pieces of information that are not available at the same time. It also refers to tracking established TCP connections (connections that have gone through the initial three-way handshake). Valid TCP traffic also refers to traffic that has the correct sequence numbers for an established connection. For Network IPSs, state signatures usually refer to signatures that require analyzing traffic from multiple packets.

Atomic Signature Considerations

One drawback with atomic signatures is that you have to know all the atomic events that you want to look for. For each of these events, you then have to create the appropriate signature. As the number of atomic signatures increases, just managing the different signatures can become overwhelming.

Another drawback is that these signatures can be applied only to situations in which the context of the event is not important. For example, assume that you have a simple string match signature that triggers an alert action whenever the traffic that it is analyzing contains /etc/passwd. If you apply this simple string signature to monitor TCP traffic, an attacker can generate alerts by sending a flood of TCP packets with the /etc/passwd string in payload. The alerts are generated even if the connection is not part of a valid TCP connection (because it is an atomic signature). Furthermore, analyzing the alerts can minimize the time that your security staff spends identifying more serious attacks that represent valid attacks against your network. Generating a large number of bogus alerts can also impact the performance of your monitoring applications and devices.

Nevertheless, atomic signatures have their advantages. First, these signatures consume minimal resources (such as memory) on the IPS/IDS device. These signatures are also easy to understand because they search only for a specific event. Finally, traffic analysis for these atomic signatures can usually be performed very quickly and efficiently.

Host-Based Examples

Host-based IPS examines many operations on the system, including function calls, files accessed, and so on. One common method for detecting anomalous user behavior is to establish a baseline of the operations that a user normally performs on the system. Then by monitoring deviations from the baseline, you can detect potentially malicious activity. For example, if a function call is never invoked normally (except in connection with malicious activity), then triggering a signature action whenever it is called is a simple example of a host-based atomic signature. Another example of this is an application that you consider a problem. For example, you might want to trigger a signature action whenever a command shell is invoked on the local system.

Note

Command shells are used to access the command-line interface on most operating systems. Accessing the command-line interface is a common mechanism to launch attacks against the system. On any operating system, such as Windows, which relies heavily on a graphical user interface, utilizing the command shell to configure the system can be indicative of uncommon behavior.

Network-Based Examples

A good example of a network-based atomic signature is the LAND attack. By inspecting a single packet, your Network-based (or Host-based) IPS can identify this attack. Because everything is contained in a single packet, no state information is needed to identify this attack.

Land Attack

The LAND attack is a denial-of-service (DoS) attack in which an attacker sends a TCP packet (with the SYN bit set) to a system in which the source and destination IP address(along with the source and destination port) are the same. When it was first discovered, many IP stacks crashed the system when they received a LAND attack.

Stateful Signatures

Unlike atomic signatures, stateful signatures trigger on a sequence of specific events that requires the IPS device to maintain state. The length of time that the signatures must maintain state is known as the event horizon. Configuring the length of the event horizon is a tradeoff between consuming system resources and being able to detect an attack that occurs over a long period of time.

Event Horizon

Stateful signatures usually require several pieces of data to match an attack signature. The maximum amount of time over which an attack signature can successfully be detected (from the initial data piece to the final data piece needed to complete the attack signature) is known as the event horizon. The intrusion system must maintain state information for the duration of the event horizon. The length of event horizon varies from one signature to another. The important point to consider is that an IPS cannot maintain state information indefinitely without eventually running out of resources. Therefore, an IPS uses a configured event horizon to determine how long it looks for a specific attack signature once an initial signature component is detected.

Stateful Signature Considerations

The main limitation to stateful signatures is that maintaining state consumes memory resources on your IPS/IDS device. Usually, however, this is not a significant problem if the IPS product is designed to efficiently use its resources. If your IPS does not efficiently manage resources when maintaining state, then the large consumption of resources (such as memory and CPU) can lead to a slow response time, dropped packets, missed signatures, and so on, which adversely impacts the effectiveness of your IPS.

Requiring a specific event to be detected in a known context increases the likelihood that the activity represents legitimate attack traffic. This minimizes the false positives generated by the stateful signatures.

Host-Based Examples

For a host-based example, we are going to use a commonly used Windows command shell called cmd.exe. As opposed to our atomic host-based example earlier in this chapter, in this situation, we do not want to trigger a signature action whenever cmd.exe is invoked (because our users use this program frequently). Our examination, however, reveals that many attacks invoke cmd.exe remotely. To remotely execute cmd.exe, the attacker must make a network connection to the host. This information can be used to refine our atomic signature by adding state. The stateful signature triggers a signature action when cmd.exe is invoked, but only if the application invoking cmd.exe first accepted a network connection.

The Host-based IPS must remember which applications have accepted network connections. This state information can then be examined whenever cmd.exe is invoked.

Network-Based Examples

Often, Network-based IPS signatures are stateful signatures because the information needed can usually be distributed across multiple packets. Even a simple string match signature is usually stateful because the string can occur across multiple packets (because the IPS must examine the data from all the packets until the successful match is made). For example, if you want to search for the string /etc/password in an HTTP URL, you might have to check multiple packets because the string can be distributed across more than one packet (although it can occur in a single packet as well).

Other examples of stateful signatures are the signatures used to monitor TCP traffic. To minimize the ability of an attacker to generate a large number of bogus alarms, most TCP attack signatures are valid only if the signature trigger is observed on a valid TCP connection. For example, suppose your signature triggers on the string /etc/password in a Telnet connection. Telnet uses TCP port 23, so the first thing that the IPS needs to track is established connections to TCP port 23. Then it also needs to track the sequence numbers for the established Telnet connections. Finally, whenever the string /etc/password is observed on an established TCP connection with the correct sequence numbers, then the signature triggers. Without maintaining this state, an attacker can generate a flood of invalid alarms by sending a flood of TCP packets to port 23 containing the string /etc/password (without ever actually establishing any valid TCP connections to port 23).