Host-Based Intrusion Prevention Systems

We'll now investigate the technology supporting HIPS products in more detail. Essentially, they are an iterative improvement on personal firewalls. We expect most vendors that are offering personal firewalls or host-based intrusion detection systems (HIDSs) today to be offering HIPSs by 2005. One of the major benefits to HIPS technology is the ability to identify and stop known and unknown attacks, both at the network layer where personal firewalls operate and in the operating system. It is this functionality that lets enterprises have a wider window to deploy patches to systems, because already deployed HIPS software is able to prevent common attack techniques, including worm activity.

Currently, all commercial HIPS software uses a technique called system call interception (which is very similar to what antivirus vendors have been doing for many years). The HIPS software uses something called an OS shim to insert its own processes between applications, accessing resources on the host and the actual OS resources. This way, the HIPS software has the ability to deny or permit those requests based on whether the request is identified as malicious or benign. For instance, if Internet Explorer was to initiate interrupt call 13H to make a direct write to the boot sector of the hard drive, which is a signature of a boot sector virus, the HIPS would intercept the call. Another approach might be to implement HIPS via a device driver. The SANS Institute was testing Windows software based on this approach as early as 2002, but it is still not ready for commercial use as of this writing. Finally, Computer Associates' eTrust Access Control resembles a HIPS in that it offers server-based access control at a higher degree of granularity than most operating systems support.

HIPS tools use a combination of signature analysis and anomaly analysis to identify attacksthis is performed by monitoring traffic from network interfaces, the integrity of files, and application behavior. Let's take a detailed look at each of these functions as well as how each monitoring mechanism can stop common attack techniques.

Real-world Defense Scenarios

HIPS products such as Cisco Security Agent, Security Architect's Ozone, and Platform Logic's AppFire have been deployed in enough organizations to start getting some "real-world" experience in defending against attacks from worms and other exploits, including Blaster, Nachi/Welchia, Sobig, IIS WebDAV, and so on. The organizations that have deployed HIPS software have reported favorably about their vendor's ability to stop unknown attacks against systems. The best defenses in this area are from vendors that offer intrusion prevention that is not solely based on signature or rule-based analysis. Because rules have to be updated to detect and catch new exploits, organizations are in a race to deploy signature updates to the HIPS agents to defend against new attacks. Using application analysis techniques, the best-in-class vendors are able to stop attacks that have common exploit methods (such as buffer overflows) without requiring updates to the software.

Dynamic Rule Creation for Custom Applications

Another development in the HIPS market is being designed to support customers who are using applications that have not been thoroughly analyzed by the vendor for application analysisdetection techniques as well as those organizations using custom applications. HIPS vendors are readying tools that monitor how an application operates in a learning mode, identifying what files are opened, what Registry keys are accessed, what system calls are made, and so on. An organization using this technology would "train" the HIPS software in learning mode to recognize the traditional behavior of the production software and use the results of this training later in production to identify and stop anomalous events.

This functionality is helpful for both vendors and customers. Vendors can use this method of analyzing applications to reduce the amount of resources needed to add an application to their list of supported applications for monitoring, and customers can use this tool to monitor custom or unsupported applications. An organization using this technology should always use caution before wide-scale deployment, preferably starting with applications such as instant messaging and email before moving on to protecting ERP applications from misuse.

Monitoring File Integrity

Whereas traditional file integrity analysis tools use cryptographic hashes on files to determine if changes have been made (at a later date), HIPS software uses its operating system shim functionality to monitor any files that are opened as read/write or write-only on the operating system. When a program or process attempts to call a function that would change the contents of a file, such as write(), fwrite(), or fsync(), or use any other file-modification system calls, the operating system checks whether the file handle corresponds to a list of files that should be monitored for change. If the file is supposed to be monitored for change, the HIPS software then checks to determine if the user or application requesting the change is authorized to do so.

The lack of authorization to change the contents of a file causes the HIPS software to drop the request to write to the file and then to generate an alert. When authorization is granted to make changes to the file, the HIPS software honors the request by passing the necessary information to the requested operating system calls.

A significant advantage of HIPS software is the ability to define authorized users in real time for monitoring the integrity of files. For instance, you could utilize HIPS software on a web server to prevent unauthorized people from making changes to web pages (such as the user account running the web server, which is IUSR_HOSTNAME on Windows IIS servers), but permit your web developers to make changes when necessary.

Monitoring Application Behavior

Application behavior monitoring is a feature of HIPS software where a manufacturer selects a supported application and records the intended functionality of the application in normal use. For example, if a vendor has provided application behavior monitoring for Microsoft Word, it would record how Microsoft Word interacts with the operating system and other applications, identifying all the product functionality. After collecting all the data about how the application should work, the vendor creates a database that details the functionality of the application to feed to the HIPS software. Once installed, the HIPS software identifies and monitors the use of the supported application. If Microsoft Word were to open a file from the file system and print the document, the HIPS software would recognize this as intended functionality. If Microsoft Word started parsing through each contact in the Outlook Contact Book to send repeated email to each recipient, the HIPS software would recognize that as anomalous activity and shut down the application, generating an alert for the analyst.

Another example of application behavior monitoring involves a web server product such as the Apache web server. If the HIPS software sees the request GET /index.html, it would recognize this as intended functionality and let the web server respond to the request. If the HIPS software sees a request for ./././././././././././././././. repeated 100 times, it would recognize the request as unintended functionality for the application and stop the request from being delivered to the application.

In practice, application behavior monitoring is difficult to get right because applications are constantly changing functionality with updates and new releases. Most vendors are developing hybrid solutions that utilize a combination of application behavior monitoring and anomaly analysis, using a specified list of anomalous events that should not be allowed on the system.

It is important to remember that application behavior analysis only works for supported applications. If your vendor supports Microsoft Exchange and the Microsoft IIS web server, and you run the IIS SMTP engine, the HIPS software offers no protection for the SMTP engine.

HIPS Advantages

Now that you have a better understanding of how HIPS software functions and what it can do, let's take a look at the advantages of using HIPS.

HIPS software includes nearly all the capabilities of HIDS software. Identifying unauthorized change to files, monitoring network activity, and the ability to see the results of network-encrypted traffic are all advantages to using HIPS software as well. The added benefit for HIPS, of course, is the ability to stop attacks from being successful. This is a welcome advantage for many organizations that struggle with patch management challenges and the short window of time between when a vulnerability is announced and when it is actively being exploited. HIPS is one more tool that might help the problem of the so-called zero-day exploit, an attack that occurs before the vulnerability is published.

Organizations are further challenged with an expanding network perimeter. Years ago we only had to worry about attacks from our Internet connections; now attacks come from wireless networks, modems, VPN connections, malware introduced by traveling users to our networks, and more. HIPS software provides a better method of defending our perimeter when distributed throughout the enterprise than traditional tools allow.

HIPS Challenges

HIPS deployments have implementation and maintenance challenges that include testing updates, deploying updates, troubleshooting updates…all the joys of complex operational software. False positives are a major challenge in the IPS market as well, although they are slightly less significant with HIPS because a false positive is this risk, however, because the false positive you experience may be how your web server responds to HTTP requests, thus limiting your ability to serve pages to people on the Internet.

The ability to detect unknown attacks is a big advantage for IPS technology, but it is often tied to specific application functionality such as IIS, Apache, or Exchange. The ability to monitor for anomalous behavior from applications is limited to those applications selected by your vendor, with almost no support for protecting custom applications. Hardening operating systems and secure coding practices are still good ideas for protecting custom application software.

More HIPS Challenges

Despite the ability for HIPS software to identify and stop attacks, it is not a replacement for regular system patching or antivirus defenses. IPS software is still in an early stage of maturity, and it isn't yet clear what weaknesses attackers will discover and exploit in this technology. It is best to use HIPS software as another piece of defense for your organization's security.

With all the advantages and detection techniques offered by HIPS software comes the additional burden of processing requirements on servers and workstations. This will contribute to the total cost of ownership (TCO) of HIPS software, possibly reducing the life cycle of your current server and workstation investments. Expect HIPS software to utilize about 20MB of RAM and between 2% and 3% of available CPU power, depending on configuration and analysis options.

Finally, the need for a management console to oversee HIPS software throughout the organization is obvious, just as many organizations use it to manage antivirus software updates and signature data files. Vendors are struggling with the extensibility of managing large numbers of nodes from a management console. Though it is growing, version 4.5 of Cisco's management console is expected to support 100,000 agents. If you are planning a HIPS deployment larger than your console supports, expect to make multiple investments in management consoles and the labor to replicate the management burden across multiple HIPS groups.

HIPS Recommendations

This section contains recommendations to keep in mind when evaluating or planning a HIPS deployment. This is a major software rollout, so you must plan, test, and manage.

Document Requirements and Testing Procedures

Carefully evaluate vendor products in a lab and production environment to ensure they deliver the desired functionality without generating false-positive detects. If a vendor's product requires significant troubleshooting and tweaking to get it working properly, record the time spent on this effort and add it to the TCO calculation for each application you wish to use on hosts protected with the HIPS software.

Develop a Centrally Managed Policy for Controlling Updates

Strong configuration management practice can significantly reduce the risk of problems with the HIPS rollout. The lower the total number of repeatable-build operating systems deployed at your facility, the better suited you are for a HIPS solution. If every single operating system is different, you should be considering a NIPS solution, not a HIPS solution. Identify who should be responsible for managing updates to HIPS software, and how often the software should be updated. Include information about how the organization should react to updates based on new Internet threats, such as a new worm or other exploitative threat. Having this policy in place before a new worm threatens your organization will impact how well the organization will be able to leverage the HIPS technology.

Don't Blindly Install Software Updates

Despite the claims from manufacturers that they extensively test the updates to their products before deployment, they still can make mistakes and ship updates that render workstations and servers useless or severely impaired. Establish a test environment for the supported workstation and server images for your organization and thoroughly test product functionality before approving the distribution of software.

Don't Rely Solely on HIPS to Protect Systems

You should use HIPS software to augment defense-in-depth techniques. Exclusively relying on HIPS software to protect systems is not a wise choice. Instead, use the extra time from the defenses provided by HIPS to carefully test and plan the delivery of patches to ensure workstations are not vulnerable to the common exploits used by attackers.

Expect Your HIPS to Come Under Attack

The popularity of HIPS software has started to get the attention of the attacker community, looking for ways to circumvent this technology. Some groups are focusing their attention on attacking the management station and disabling HIPS software on clients throughout the organization centrally. Other groups are looking at how HIPS software examines system calls, and how the process might be circumvented. Because malware has been released into the wild that disables antivirus software and/or personal firewalls, you should expect attacks against the HIPS itself. To date, there have not been any public exploits or security advisories for HIPS software, but attackers will continue to research, looking for weaknesses in these tools and ways they can be exploited.