Host-Based Intrusion Detection

As you saw in the previous section, firewalls can be used to defend whole networks, as well as individual servers. Similarly, IDSs exist in two varieties: network-based and host-based. We introduced the notion of intrusion detection in Chapter 8, "Network Intrusion Detection," where we examined optimal uses of IDS components for monitoring a network for malicious events. We deployed network IDS sensors in a way that allowed us to sniff traffic going between hosts and network devices. Network IDS enabled us to apply certain criteria to captured traffic to single out events that were likely to be indicative of an attack.

Much like their network counterparts, host-based IDS products also exist primarily to detect and examine malicious activity at various stages of an attack, but they are optimized for monitoring individual hosts. Additionally, many host-based IDS products can be configured to integrate their findings, providing a unified view of multiple systems throughout the network. Some host-based IDS products also provide intrusion prevention capabilities, stopping attacks before they can cause damage; such products are described in Chapter 11, "Intrusion Prevention Systems." In this section, we examine key aspects of the host-based intrusion detection process and look at several categories of host-based IDS products.

The Role of Host-Based IDS

At its core, host-based intrusion detection involves monitoring the system's network activity, file system, log files, and user actions. (Most host-based IDS products typically monitor only one or a few of these categories, not all.) This allows a host-based IDS to identify activities that network-based IDS may not be able to see or understand, because the actions take place on the host itself. For example, a host-based IDS could alert if accounts such as guest or nobody suddenly possessed administrative privileges. Specifically, host-based IDS software has the following key advantages over its network counterparts:

• Host-based IDS software can monitor user-specific activity on the system. The software can observe the user's local activity because it has access to such host-specific information as process and service listings, local log files, and system calls. Network IDS sensors, on the other hand, have a hard time associating packets to specific users, especially when they need to determine whether commands in the traffic stream violate a specific user's access privileges.

• Host-based IDS programs can monitor data exchanges of encrypted network streams by tapping in at the connection's endpoint. Running on the VPN's endpoint allows host-based IDS to examine packets in their clear-text form, before the host encrypts outbound packets, or after it decrypts inbound packets. A network IDS sensor, on the other hand, cannot examine the payload of an IPSec packet or the contents of a packet that is part of an SSL session. The need to perform content analysis of network traffic at the hosts continues to increase as companies continue to deploy VPN solutions.

• Host-based IDS programs can detect attacks that utilize network IDS evasion techniques. As we discussed in Chapter 8, such techniques exploit inconsistencies in the way a network IDS interprets the packet's intentions from the effect that the packet might have on the targeted host.⁷ For example, unusual packet fragmentation might confuse a network IDS sensor, but would have no effect on host-based IDS products.

Host-based intrusion detection is also useful for correlating attacks that are picked up by network sensors. If a network IDS sensor detected an attack that was directed at one of your hosts, how would you know whether the attack was successful? The host's IDS software can help you determine the effect of the attack on the targeted system. Of course, if the host is compromised, its logs might be altered or deleted. But if you are automatically relaying all host IDS data to a central, dedicated log server, you can use that data instead of the original IDS logs if they are unavailable or untrusted. From an incident-handling perspective, host-based IDS logs are also vitally important in reconstructing an attack or determining the severity of an incident.

Ideally, you would deploy host-based IDS software on every host in your organization. However, that might not be a viable option for your company, considering the amount of resources involved in deployment, maintenance, and monitoring. When deciding whether to deploy host-based IDS software on a system, consider the host's risk level. Critical servers that store sensitive data and are accessible from the Internet will surely benefit from the extra layer of protection that host-based IDS provides. You might consider deploying such products on workstations as well, depending on the nature of threats to which they are exposed.

Host-based IDS products can watch over the channels that need to remain open for the host to perform its business function. If a host needs to function as a public web server, its firewall cannot block inbound connection attempts to TCP port 80, but its host-based IDS can be tuned to monitor HTTP traffic or other host attributes for malicious activity. Additionally, some host-based IDS products can interact with host-based firewall software to shun the attacker if an intrusion attempt is detected.

Host-Based IDS Categories

Now that you understand the roles that host-based IDS plays when operating as part of a network's security perimeter, let's examine different types of host-based IDS solutions. As you know, multiple sources for data can be used to perform intrusion detection at the host level. The primary reason for wanting to look at the host's file system, log files, and network connections is because the malicious activity on a host can exhibit itself in multiple ways. Some commercial products, such as ISS Proventia Intrusion Detection and Enterasys Dragon Intrusion Defense System, can monitor several data sources on the host. Other products are optimized to perform host-based intrusion detection based on a specific data source:

• The host's file system: AIDE, OSIRIS, Samhain, Tripwire

• The host's network connections: BlackICE, PortSentry

• The host's log files: LANguard, Logcheck, OsHids, Swatch

Note

Chapter 11 describes host-based intrusion prevention products that monitor another type of data source: application behavior. Products such as Finjan SurfinGuard and Cisco Security Agent monitor the behavior of locally running programs to detect and block malicious actions. Some host-based antivirus and firewall products also offer limited application behavior controls, such as not allowing web browsers to run programs stored in temporary directories.

Checking the File System's Integrity

A category of host-based IDS tools known as file integrity checkers work by detecting unauthorized changes to the host's file system. They operate by taking a "snapshot" of the file system in a trusted state, when all the files are considered to be valid. During subsequent scans, these tools compare the system's files to the initial baseline and report noteworthy deviations. To tune the integrity checking mechanism so that it only monitors relevant aspects of files, you can specify what file attributes are allowed to change, or what files can be ignored altogether. For example, applications frequently create temporary files in C:\WINNT\Temp or /tmp directories; alerting the administrator every time a new file appears or disappears from these directories would generate too many false positives. On the other hand, contents of core system libraries rarely change, and it is normal for the host's log files to grow in size while retaining initial ownership and access permissions.

Note

The AIDE software we discuss in this chapter is a free integrity verification tool. This is different from the commercial AIDE software that carries the same name, but is an intrusion detection tool for correlating log information from different firewall and IDS products. Litton PRC developed the commercial AIDE program; it has no relation to the freeware AIDE utility discussed here.

Integrity checking tools are able to detect changes to a file's contents by calculating the file's checksum or cryptographic hash during the scan and comparing it to the file's "signature" that was obtained when creating the baseline. An alternative method, which is impractical in many situations due to space requirements, is to make an actual copy of each file to be monitored. Increasing the difficulty of tampering with the database of baseline signatures can be accomplished in several ways:

• Obfuscate the contents of the baseline database by using a proprietary binary format instead of plain text when saving the database to disk. Although this mechanism makes it more difficult to tamper with the database, it hardly prevents the attacker from discovering the obfuscation scheme or from using the integrity checker to update the baseline.

• Place the baseline database onto read-only media, such as a write-protected floppy disk or a CD-ROM. This method requires that the disk or the CD be accessible to the integrity checker when it performs the verification scan. This method is reliable and is most useful for hosts whose baseline does not need to be frequently updated. Keep in mind, though, that even if the attacker is unable to modify the baseline database, he might be able to change the integrity checker or modify its configuration to use an unauthorized baseline. Placing the checker onto the read-only media helps defend against some attacks of this nature, but having access to the host might allow the attacker to modify the system's kernel or file system drivers to conceal his presence on the host anyway.

• Digitally sign the baseline database. In this scenario, updating the program's baseline typically requires the administrator to present the appropriate cryptographic keys and supply the necessary passwords. This technique achieves a good balance between the first two approaches. It is frequently used in environments that need to be able to remotely update the baseline periodically, such as when installing system patches or otherwise updating the host's configuration.

Tripwire is the best-known file integrity checking utility. In many ways, Tripwire is a benchmark against which other tools in this category are measured. The original version of Tripwire was developed in 1992 at Purdue University in West Lafayette, Indiana, and it is still available free under the name Tripwire Academic Source Release (http://www.tripwire.com/products/tripwire_asr/). This version of Tripwire runs only on UNIX platforms. Despite its age, this version of Tripwire is still effective at detecting unauthorized changes to the host's files, although it is no longer being actively maintained. Full commercial versions of Tripwire for servers and network devices (http://www.tripwire.com/products/servers/) are not free, but they boast a number of improvements over the initial version:

• The commercial software Tripwire for Servers runs on both Windows and UNIX hosts. The Windows version of the tool can monitor the system's Registry in addition to the file system.

• The commercial software Tripwire for Network Devices can monitor the integrity of configuration files on routers and switches.

• Multiple hosts and devices monitored by the commercial versions of Tripwire can be controlled centrally through a unified configuration and reporting interface through Tripwire Manager.

Note

An open source version of Tripwire for Linux is available for free at http://www.tripwire.org. It is included with many Linux distributions, including Red Hat Linux. The open source version of Tripwire was derived from the commercial product, but it is not being actively maintained.

Tripwire Manager is a console available with the commercial version of Tripwire that offers the ability to centrally manage multiple Tripwire "agents" that are deployed on remote hosts and devices across different operating systems. Tripwire Manager is the key difference between the commercial version of Tripwire and free tools that exist in the same category of host-based IDS products. When you need to support many Tripwire instances, this feature alone can help justify purchasing Tripwire instead of using the freeware alternatives.

AIDE (http://sourceforge.net/projects/aide), which stands for Advanced Intrusion Detection Environment, is a free integrity checker with similar features to the academic release of Tripwire. Some of the key differences between AIDE and various Tripwire versions are listed next:

• AIDE is free, just like the academic version of Tripwire and the Linux version of Tripwire Open Source.

• AIDE is maintained through a steadier development cycle than the academic version of Tripwire, which is no longer maintained. At the same time, the commercial version of Tripwire is being developed much more actively.

• AIDE runs on a wide range of UNIX platforms but, unlike the commercial version of Tripwire, it does not run on Windows.

• AIDE does not cryptographically sign its baseline database, making it more difficult to ensure the integrity of its findings. (The academic version of Tripwire does not do this either.)

Overall, organizations that cannot justify paying for the commercial version of Tripwire will probably benefit from installing AIDE instead of the academic version of Tripwire. Another free alternative is Samhain (http:/samhain.sourceforge.net/), which offers file integrity checking capabilities for various versions of UNIX. Samhain provides the ability to cryptographically sign not only the baseline database, but also its configuration file. Samhain can be configured to monitor files for a single host or for a group of hosts through a secure log server. It also offers stealth capabilities, which assist in concealing the presence of Samhain from attackers.

Those who need to perform file system integrity verification for critical Windows systems will not find many robust alternatives to the commercial version of Tripwire, but they might consider taking advantage of similar features built in to some other host-based products, such as personal firewalls that monitor changes to executables. On older Windows platforms, the sysdiff utility from the Windows Resource Kit can be used to detect added or deleted files and changes to configuration files and Registry entries. Sysdiff does not use cryptographic techniques enjoyed by tools such as AIDE and Tripwire, and it is not available starting with Windows XP. A more powerful alternative to the sysdiff tool is a relatively inexpensive monitoring utility for Windows called Winalysis (http://www.winalysis.com/). Winalysis has a graphical interface, supports the collection of SHA-1 digital signatures, and can be scheduled to observe remote hosts from a central location.

File integrity checking software typically relies on the administrator supplying a policy file that defines which attributes of which files need to be monitored for changes. One of the ways to go about defining this policy is to list a limited number of critical files on the system that you know are supposed to change infrequently. A more comprehensive approach, applicable primarily to hardened server systems, calls for monitoring all files on the host with the exception of those that change frequently. Defining such a policy is a time-consuming process because each server is configured slightly differently and can modify different files during normal operation.

When defining a comprehensive file monitoring policy, it is recommended that you start by specifying that the program should monitor all files and directories recursively from the root of the file system, with the exception of files that are expected to change constantly (such as swap files and logs). You can then run the integrity checker, see which attributes of which files changed during the system's normal operation, and modify the policy appropriately. After several iterations, you should be able to achieve a stable state that will save you from having to constantly update your policy and the baseline database, while monitoring as many aspects of the host's file system as possible.

Network Connection Monitors

Now that you know how to detect unauthorized changes to the host's file system, let's switch our attention to monitoring another critical aspect of the host's operation: its network connectivity. Specifically, we want to use available data about network connections that attempts to initiate or terminate on the host to detect malicious behavior. The impetus behind connection monitoring is similar to the one in network IDS products that run in promiscuous mode to examine network streams for multiple hosts and devices. A host-based IDS, however, can also associate network sockets with specific processes and users on the system, and it can be tuned to the exact characteristics of the host. Additionally, host-based network-monitoring software is unlikely to be overwhelmed by the voluminous network traffic that continues to push the limits of network IDS performance.

One popular host-based IDS product for monitoring the system's network connections is BlackICE (http://blackice.iss.net/), produced by Internet Security Systems (ISS). There are two versions of the software: BlackICE PC Protection runs on Windows-based operating systems and is optimized for protecting a workstation, whereas BlackICE Server Protection offers similar capabilities for servers.

Whenever BlackICE observes a suspicious network connection that targets its host, it creates a log for this event. A host-based firewall would typically create an individual record for each blocked packet. The IDS mechanism in BlackICE is able to group events associated with multiple offending packets into a single log entry that identifies the attack. For example, BlackICE can correlate several suspicious packets as being a single port scan. Instead of logging each packet that comprised the scan, BlackICE creates a single entry in the log. However, BlackICE can be configured to capture full packets that it identifies as belonging to an attack sequence and log them for future analysis. In addition to performing IDS services, BlackICE comes with a built-in host-based firewall that can block unauthorized inbound and outbound connections.

Host-based IDS products that monitor network connections frequently have the option of responding to detected attacks by blocking the attacker's host from accessing ports on the protected system. This capability is known as intrusion prevention, and we described it in the context of network IDS sensors in Chapter 8. Host-based IDS products, such as BlackICE, can perform intrusion prevention by automatically reconfiguring its host-based firewall component to shun the attacking IP address for some time period. This capability is useful for blocking an attack at an early stage, such as during a port scan, before it escalates into something more forceful, such as a buffer overflow attempt.

PortSentry (http://sourceforge.net/projects/sentrytools/) is another host-based IDS product that can detect port scans and other unauthorized connection attempts to the system. PortSentry is free and can run on most UNIX operating systems. When PortSentry detects a network-based attack, it can block the attacking host by automatically reconfiguring the compatible firewall on the local host or by placing an appropriate entry into the hosts.deny file used by TCP Wrappers. For example, the following are Syslog records that document PortSentry actions when it detects a port scan coming from 192.168.44.1:

[View full width]
 Jan 19 10:35:57 localhost portsentry[1252]: attackalert: TCP SYN/Normal scan from host:  192.168.44.1/192.168.44.1 to TCP port: 13 Jan 19 10:35:57 localhost portsentry[1252]: attackalert: Host 192.168.44.1 has been  blocked via wrappers with string: "ALL: 192.168.44.1" Jan 19 10:35:57 localhost portsentry[1252]: attackalert: TCP SYN/Normal scan from host:  192.168.44.1/192.168.44.1 to TCP port: 21 Jan 19 10:35:57 localhost portsentry[1252]: attackalert: Host: 192.168.44.1/192.168.44.1  is already blocked Ignoring 

In this example, PortSentry detected an unauthorized connection to TCP port 13 on the local host. It responded by reconfiguring TCP Wrappers in an attempt to block subsequent connections from the offender.

As useful as active response capabilities can be in a host-based network connection monitor, they are also dangerous because the supposed offender might get blocked based on a false positive. As a result, this functionality is much more appropriate for workstations than it is for servers. Workstations are usually not accessed via inbound connections as often as servers are, and they are less likely to receive a slew of benevolent inbound connections that look like a port scan.

Log File Monitors

So far, we have examined host-based intrusion detection techniques that involve examining the system's network connections and its file system. Another core data source for useful security information is the host's log files, which may include system, audit, authentication, and application events. Log file monitors observe the contents of logs and alert administrators when suspicious events are detected. One such host-based IDS product is called Swatch (its name stands for "simple watcher") and is available at http://swatch.sourceforge.net/. Swatch is free and runs on most UNIX operating systems. We could use Swatch, for example, to stay abreast of attacks that PortSentry detects. To set this up, we would configure Swatch to email the administrator when it locates a line with the string attackalert in a Syslog record.

Another free UNIX-based tool for monitoring log files is Logcheck (http://sourceforge.net/projects/sentrytools/). Unlike Swatch, Logcheck does not monitor logs in real time; it runs periodically and emails alerts in batches. This helps the administrator to limit the number of email messages that he receives, but it might also delay the administrator's response to an attack.

Log file monitoring utilities are available for Windows platforms as well. The following represent a couple of the products worth your look:

• TNT ELM Log Manager (http://www.tntsoftware.com/)

• LANguard Security Event Log Monitor (http://www.gfi.com/lanselm)

Log file monitors have the benefit of being able to observe events generated by multiple security components on the host. Moreover, in scenarios in which logs from several systems are submitted to a single host, log file monitors can perform intrusion detection based on data from multiple perimeter security hosts and devices. This is a powerful technique that transcends the boundaries of defense components of a single host, and we discuss it in greater detail in Chapter 19, "Maintaining a Security Perimeter," and in Chapter 20, "Network Log Analysis."