Host-Based Intrusion Prevention

Host-based intrusion detection and prevention systems function as software or agents installed on endpoint devices. They are normally used to protect critical application servers; however, they can just as easily be used to protect workstations. Host-based intrusion prevention technologies are gradually making their way into consumer and corporate desktop solutions to provide protection on the endpoint. A number of different approaches have been taken in order to provide protection on the endpoint. Current solutions provide protection at multiple layers within the operating system in order to subvert attacks originating both locally and from the network. These mechanisms may include

• Network firewall capabilities in order to block incoming connections by default.

• Network intrusion prevention capabilities to block attacks on network connections that may be permitted. This technology may in principle be very similar to what is used in a network-based intrusion prevention system (limiting the signatures or rules to that specific architecture). Some vendors use a common engine in both.

• Wrapping operating system calls in order to add additional security checks around them, and also to apply a per-application policy to them. This method is used to provide limitations on which resources an application can access, for example, files, directories, and registry keys. In doing so, an additional "shield" is placed around critical resources.

System Call Learning

System calls are the functions exposed to user applications by the operating system kernel. They provide core functions such as file access and network access among many other things. Even the most basic applications cannot function without using system calls to perform a function. They include the basic building blocks of an application, such as:

fork()	Fork a new copy of the existing process
read()	Read from a file descriptor (file or network connection)
write()	Write to a file descriptor (file or network connection)
open ()	Open a file
close()	Close an existing file descriptor
execve ()	Execute another program
socket()	Allocate a new network socket
accept()	Accept an incoming network connection
connect()	Create a new outbound network connection

The above are only a few examples from UNIX-based systems (on modern systems some 200 system calls may exist). An identical notion of system calls exists on Windows-based systems, otherwise known as the WIN32 API.

The system call learning approach operates by building a graph of system call sequences that application calls during normal execution. When an attacker attempts to exploit a vulnerable application, they can do very little without invoking a system call in order to execute a command or open a network connection. The premise is that these system calls will deviate from the previously learned graph, or legitimate sequence of system calls that was learned. When this occurs, the application can either be shut down, or at minimum an alert can be generated. An oversimplified example is shown here:

Policy-Based Intrusion Prevention

Policy-based intrusion prevention systems work by using a predefined policy in order to enforce application and host behavior. This policy can either be provided by the vendor, specifying the appropriate behavior for common applications, or it can be defined by the administrator, based on an enterprise-wide security policy. In addition, this policy may be a learned policy, such as that which was learned using the mechanism discussed in the previous section. A policy may define in detail which files an application can access, acceptable system calls (and their order), network resources, and what other parts of the operating system the application can interact with, including applications that may be called from this one.

Buffer Overflow Prevention

One additional function that a host-based intrusion prevention system can provide is protection from buffer overflow attacks. One method that intrusion prevention systems use to do this is to hook the OS system calls with a wrapper function. This wrapper function ultimately invokes the real system call, but before doing so it validates the calling function's stack to ensure that the system call has not been called from code running on the stack. While having minimal impact on performance, this method is effective at detecting and subverting many traditional buffer overflow attacks.

Stackguard, a freely available compiler extension, was one of the first mechanisms to demonstrate a compiler-based approach to buffer overflow prevention. It shares many similarities with ProPolice, another similar approach developed by IBM Research. ProPolice is incorporated by default into the OpenBSD operating system distribution.

ProPolice is a GNU Compiler extension that adds additional code into both the beginning and end of each function call at compile time. This additional code is run each time a function is invoked, placing a randomly generated canary value onto the stack next to the function's return address. Since a traditional stack-based buffer overflow would need to overwrite this return address in order to execute the attacker's code, it would also need to overwrite the canary value in the process. By checking the value of the canary variable after the function has executed (and before returning), it's possible to determine whether this value has changed. If it has changed, then the stack has been corrupted and can no longer be trusted, and the application should be terminated and an alert generated. In order to prevent attackers from guessing the canary value, the mechanism used to generate it must use a well- seeded random number generator.

To get this protection, users need to compile applications with the ProPolice-equipped GNU C compiler that comes with OpenBSD or use the already-protected applications that ship with OpenBSD. Interestingly enough, the use of the canary methodology has been so successful that Microsoft has now incorporated a similar approach into Windows XP Service Pack 2.

Another mechanism that has made its way into operating systems such as OpenBSD is the ability to prevent memory segments from having both WRITE and EXECUTE permissions at the same time. The POSIX standard defines three permissions for each memory page: PROT_READ, PROT_WRITE, and PROT_EXEC. This prevention mechanism, known as W^X, is a short form for "PROT_WRITE XOR PROT_EXEC." Since buffer overflows rely on the need to both write code to memory, and then jump to that code to begin execution, preventing writable memory segments from being executed (and vice versa) prevents this.

System Integrity and Change Detection

Prior to today's host-based intrusion prevention technologies, one of the most widely used mechanisms to detect an attack was detecting changes to an operating system's state. Tripwire, a commercial product with an open source version available under the GPL, is one of the most well-known tools for doing so.

The concept behind this is simple: Build a profile of a system while it is in a known trusted state and look for anything to change. By building a comprehensive database of MD5 or SHA-1 checksums on all system files, you are able to build a full database of a system's current state. Then, by routinely looking for changes between the original checksums and the current system state, changes are easily visible. In addition to file changes, new files or directories that have been added to the system are also evident.

While it has been shown that MD5 and SHA-1 collisions can be induced in some situations, it is not yet possible for an attacker to create collisions for a specific known hash value. This means that for the time being mechanisms using these functions continue to remain secure.