3.6 Host-Based Intrusion Detection

In an article printed on the SANS Web site, Allison Hrivnak states:

Intrusion detection systems monitor system and network resources to detect unusual activity or changes. There are two types of intrusion detection systems: host and network based. A network based IDS is placed on the network near the system or systems being monitored and analyzes network traffic for attack patterns and suspicious behavior. A host based IDS resides on the system being monitored and tracks changes made to important files and directories. While both are part of a good defense- in-depth strategy to prevent attackers from being able to enter networks and alter or compromise critical information, only a host based intrusion detection system with a well written policy will provide a strong foundation to good system security. [4]

There are many methods of accomplishing the task of intrusion detection. The remainder of this chapter presents a few of the most widely known tools and techniques for accomplishing host-based intrusion detection.

3.6.1 TCP Wrappers

In 1990, Eindhoven University of Technology, in Eindhoven, Netherlands, was under heavy attack by a Dutch hacker who had obtained root-level access to several computer systems there. The hacker caused a huge amount of damage through the execution of the rm -rf / command in Unix. This command is extremely destructive, working much the same way as the format command in DOS works. Wietse Venema was employed by the Mathematics and Computer Science Department at the university. He developed a program called TCP Wrappers that could control host access in response to these attacks. The program also tracked and logged intruders.

TCP Wrappers acts like a guard at a checkpoint, verifying a client's access rights before allowing entry. TCP Wrappers leverages the client/server relationship by inserting itself between the client and the server to act as the server until the client has properly authenticated to the host. TCP Wrappers uses this access control feature to authenticate other hosts . TCP Wrappers is freely available [5] on the Internet.

How Wrappers Work

Before the creation of TCP Wrappers, Unix systems typically used a daemon called inetd as a conduit between hosts and processes. It would wait for a network connection, and then it would run the appropriate server program. After the host and service were connected, it would become dormant and wait for the next request. TCP Wrappers requires all of the network server programs to be moved to another directory, and the TCP Wrapper programs are installed in their place. TCP Wrappers operates by intercepting and filtering incoming requests for the network services, including the following:

  SYSTAT, FINGER, FTP, TELNET, RLOGIN, RSH, EXEC, TFTP and TALK

TCP Wrappers relies on the client/server relationship necessary for most TCP/IP protocols. When invoked, TCP Wrappers substitutes the network service being called upon with the TCP daemon ( tcpd ). After it does all of this, it moves the substituted service to another file until the client can be authenticated. TCP Wrappers acts as a barrier around the network services that are called. Wrappers do not have any interaction with the client/ user or the host/server. This is important for two reasons: (1) the wrappers are application independent, which means the same program can protect many kinds of network services, and (2) no interaction with users means that the wrappers are invisible from the outside.

Once tcpd has been activated, it searches for matches against the host/ client's IP address and the service requested . Next, it uses its access control function by searching the host.allow and host.deny files. The host.allow file determines who is allowed to execute what functions or processes. Conversely, the host.deny file filters out unwanted users or hosts to specified servers and/or processes. Each file contains, or should contain, a set of rules that specifically allow or deny users, hostnames, networks, or services. Tcpd will search the host.allow file first. If no rule prevents the host's connection, the request will be turned over to the requested process. If a host is allowed in the .allow file and denied in the .deny file, the host will be permitted access because tcpd stops at the first match. Access can be controlled per host, per service, or in combinations thereof.

Problems with TCP Wrappers

In some instances, UDP and RPC/udp daemons will go into a "wait" mode after they have completed a connection. They wait just in case another request is initiated. These daemons are classified with the "wait" option in the inetd configuration file. This classification is not recognized by TCP Wrappers, so it causes problems. Also, TCP Wrappers will not work with RPC over TCP because they are registered as RPC/tcp in the inetd configuration file.

According to information found on the CERT [6] Web site, in January 1999, an intruder modified TCP Wrappers version 7.6 at its primary FTP site. This modified version contained a Trojan horse that provided intruders with the ability to connect from port 421, allowing root-level access. The trojanized version also sent an e-mail to an external address upon compilation. The e-mail identified the site and the account that compiled the program.

The TCP Wrappers software has many bugs , most of which are discussed in the makefile distributed with the program. Even so, TCP Wrappers should be an essential tool in every security administrator's box. It protects against many common Unix compromises, and it allows system administrators to offer otherwise vulnerable services to authenticated clients with static IP addresses without compromising the service.

3.6.2 NukeNabber

NukeNabber is a program that set itself up to listen on TCP and UDP ports, which are commonly attacked over the Internet. A total of 50 ports can be monitored at once. ICMP dest_unreach attacks are logged, and it is designed to provide trace information on the attacker, including a method of finding the attacker's nickname on IRC (mIRC, VIRC, and PIRCH clients are also supported). NukeNabber can be used to block some of the more common hacking attempts, such as Back Orifice and NetBus. Although it cannot offer total protection, it can be used to deter most attacks. Nukenabber is not easily available anymore, and it appears to have been taken out of production after version 2.9.b was released.

3.6.3 BackOfficer Friendly

BackOfficer Friendly is a spoofing server application that runs on your Windows or Unix system and notifies you whenever someone attempts to obtain remote control of your system by using the Back Orifice tool. Basically, this program pretends to be a Back Orifice server. BackOfficer Friendly gives an attacker false answers that appear to have come from the Back Orifice application. All the while, BackOfficer Friendly is logging the attacker's IP address and all of the operations the hacker attempted to perform. It contains routines that allow it to selectively emulate a variety of other services. When someone runs an automated probe, such as a Ballista scan, ISS scan, or SATAN scan against your desktop, BackOfficer Friendly produces a string of alerts, making it obvious that a scan attempt has occurred.

3.6.4 Back Orifice

Back Orifice (BO) is advertised as a remote administration tool that allows system administrators to remotely control a computer. In reality, it is a backdoor that was designed by a group called the Cult of the Dead Cow Communications. It is usually distributed in the form of a Trojan horse attack. During an installation, it gives no indication of what actually occurs. Once installed, the server is intentionally difficult to detect on a machine. This program gives almost complete control over the victim's computer to a remote attacker. It runs invisibly , and it has to be run once to be installed, but it is almost never detected by the victim.

Because it is a Trojan horse, BO can be packaged with legitimate software, attached to any program or file, or run by itself. It is configurable in a variety of ways, and it allows a high degree of access and control by a remote operator. This remote operator uses a simple pushbutton client program to access the "server" on a victim's machine. Once the hacker has gained access to the victim's system, he can perform practically any function the user could perform. He can see passwords, run a DOS session, use the computer as a relay point for communications (so as to make himself untraceable), read mail, track keystrokes ”and lots more. BO has been around since late 1998. Formerly, intrusion tactics were mainly the game played by highly skilled hackers. With BO, any child with general computer knowledge can hack. When attempts are made to clean a machine of BO, detection and removal are fairly simple.

3.6.5 AtGuard

AtGuard is a well-respected firewall in the network security and home PC security fields. It was originally developed by a company called WRQ and was one of the first products available to home users that allowed precise control over both inbound and outbound network connections. Using a rule-based system, complete with an "interactive learning mode" that allows the user to determine responses to connection attempts as they occur, AtGuard permits extremely precise control over any and all network connection attempts. This helps the novice user become more familiar with which types of programs make which types of connections and allows the user to fine-tune the firewall to allow only the programs he chooses to access the Internet. AtGuard provides an almost overwhelming amount of control for a user's network connection. After an initial configuration period, the user can simply turn off the interactive learning mode for rule-building, and the firewall will block all connection attempts that it does not already have rules for. Thus, safe Internet access is practically guaranteed . The AtGuard program was recently acquired by Symantec, Inc., and it has been incorporated into their product offerings. This new package includes other applications to enhance Web and e-mail security, as well as a firewall application.

3.6.6 Syslog

RFC 3164 [7] describes the syslog protocol, which has been used for the transmission of event notification messages across networks for many years . It was originally developed at the University of California at Berkeley. Its value to operations and management has led it to be ported to various operating systems as well as being embedded into many networked devices.

Basically, the syslog protocol provides a transport mechanism to allow a machine to send event notifications over IP networks to event collectors (also known as syslog servers). There is little to no uniformity to the content of syslog messages. No assumption about formatting or message contents should be inferred. The protocol is designed only to transport these event messages. The common factor, in all cases, is that one device originates the message. No acknowledgment of the receipt is made. In order to keep this process as simple as possible, no coordination is required between the transmitters and the collector. Transmission of syslog messages may be started on a device without even having a collector configured or physically present. This approach has enabled the acceptance and deployment of syslog across a wide spectrum of users.

3.6.7 Tripwire

In 1992, Dr. Eugene Spafford and Gene Kim developed Tripwire at Purdue University. It quickly became one of the most widely used tools among security professionals, even though it was freeware. In 1997, Tripwire, Inc. was formed . The first fully supported product was released in 1999. Tripwire [8] is now commercially available for multiple platforms, including Windows NT and 2000, Solaris, AIX, HP-UX, Free BSD, and Linux. Tripwire's basic purpose is to check the integrity of important files and directories against prestored values kept as a baseline in a database. When any changes between the current file and the baseline are detected, Tripwire will raise an alert. There is now an open source version of Tripwire [9] available (discussed as follows ).

Practically speaking, Tripwire is a file system integrity checking program, originally built for Unix operating systems. In order to use it, you need to build a baseline configuration file. This file designates which directories and files you want to verify and specifies the attributes you want to have verified for each file. The first time Tripwire is run with the initialize option, it will create a database of cryptographic checksums that correspond to the files and directories specified in the configuration file.

In order to protect the Tripwire program from catastrophe, both the configuration file and the initialized database should be backed up to a medium that can be physically write-protected, such as a CD-ROM. This backup version, which exists as read-only data, becomes the authoritative baseline reference you can reliably use to test the integrity of directories and files on your system. In addition to one or more cryptographic checksums representing the contents of each directory and file, the Tripwire's database also contains information that allows you to verify the following attributes:

Access permissions
Creation date and time associated with the item's inode
Date and time of last access
Effective execution settings
File mode settings
Group ID of the group of users to which access may be granted
Inode number in the file system
Last modification made
Number of links
Size of the item
User ID of the owner

For any system, you should verify the integrity of all critical operating system directories and files, plus any other directories and files that you consider sensitive or that have no reason to change under normal conditions. When choosing which attributes of files and directories to verify, consider how they are used on your system.