In the context of perimeter security, the primary goal of system and network monitoring is to provide administrators with the awareness of the state of a protected environment. We discussed monitoring in the context of intrusion detection in Chapter 8, "Network Intrusion Detection," and Chapter 10, "Host Defense Components," where we tuned IDS sensors to automatically detect events that are frequently indicative of misuse. System and network monitoring also involve anomaly detection but concentrate on availability and performance parameters of the resources. Security professionals should be interested in seemingly mundane issues, such as server uptime, network utilization, and disk space availability for numerous reasons:
A well-defined monitoring process allows administrators to detect problems at an early stagehopefully before the problems escalate into critical incidents. One of the most powerful ways to increase effectiveness of this process is to perform monitoring centrally, which generally involves configuring a serveror a cluster of unified serversto observe multiple resources throughout the network. A centralized monitoring infrastructure often benefits from the use of client modules that report to the central server information gathered from remotely monitored systems. You should already be familiar with such a configuration because it is frequently used to manage multiple firewalls, VPN nodes, or IDS sensors centrally.
In this section, we look at a typical way of implementing a centralized monitoring infrastructure. To illustrate monitoring concepts, we use a popular monitoring application called Big Brother; however, the same monitoring principles apply to other applications of this nature.
Big Brother Fundamentals
Big Brother is a flexible software tool for performing network- and system-monitoring functions (http://www.bb4.org). Numerous products can fulfill monitoring needs; a small sample of these includes BMC PATROL (http://www.bmc.com), HP OpenView (http://openview.hp.com), IBM Tivoli (http://www.ibm.com/software/tivoli), and Nagios (http://www.nagios.org). Unlike several other contenders, Big Brother focuses solely on monitoring aspects of system and network maintenance, without offering features to remotely manage devices. This makes it a lightweight application that is particularly well suited for demonstrating monitoring concepts. As a result, we use Big Brother in the examples throughout this section for many of the reasons we would consider using it on our networks:
Big Brother is able to track the status of remote nodes by attempting to connect to monitored systems over the network. In this role, Big Brother can detect the availability of a system via ping packets, as well as by attempting to elicit an expected response from specific TCP services. Big Brother also supports the use of agent modules that run on remote nodes as Big Brother clients and gather local information, such as CPU performance, disk space utilization, and process status.
The commercial version of Big Brother, called Big Brother Professional Edition, supports several enhancements to the free version. In particular, the Professional Edition offers a simplified installation routine and point-to-point encryption. You can purchase the commercial version from Quest Software (http://www.quest.com/bigbrother).
Big Brother's architecture consists of four core components that are integrated into a unified application. These components, described in the following list, can run on distributed systems to achieve the desired extent of scalability or can coexist on a single server:
If Big Brother determines that a monitored parameter has exceeded an acceptable threshold, it alerts administrators by displaying a warning on the BBDISPLAY's web page. Figure 19.1 shows how Big Brother consolidates status information regarding multiple systems onto a single summary page. Icons on Big Brother's summary matrix represent the current status of monitored parameters for each system. A green square represents the normal state of the system, which is the case for most services in this figure. A flashing star icon for the HTTP service on "lemon" indicates that Big Brother experienced problems connecting to the web server on "lemon" over the network. Administrators can obtain additional information regarding the problem by clicking the icon. Additionally, Big Brother can send the alert via email, pager, Short Message Service (SMS), or a custom script.
Figure 19.1. Big Brother presents a unified view of monitored resources by using icons to represent the status of observed attributes.
Now that you know a little about fundamental capabilities of monitoring software, the next section covers how to determine which resources should be observed in the first place and how to set up such monitoring.
Establishing Monitoring Procedures
Be sure to define your goals and priorities before rolling out a monitoring solution. For example, if you are maintaining an e-commerce site that makes a significant portion of its revenue via the Internet, you might be interested in focusing on the following data points:
Considering that most of us operate under the constraints of limited staff, it is impractical to treat all changes in the state of the environment with the same urgency. Moreover, monitoring all aspects of performance and availability on each system is likely to produce so many alerts that those that are truly important will be drowned out by the noise of inconsequential events. Knowing when a firewall process dies is likely to be more important than receiving notifications every time a user's workstation is rebooted.
We have been using risk analysis to define the extent of hardening and segregation for security perimeter components throughout the book. Similarly, we can define monitoring requirements based on the resource's importance and the likelihood that its availability might be disrupted. For example, if email communications are a critical business function for your company, you might consider monitoring numerous aspects of the mail server's availability and performance, such as the accessibility of the host over the network, the status of the mailer's processes, and the state of file system utilization. If that same mail server has been acting flaky, you might be interested in adjusting monitoring criteria to help track down the cause of the problem or to enable yourself to react to problems more quickly. In that case, it might help to adjust the thresholds for alerts or to monitor additional attributes, such as auxiliary system processes, as well as CPU and memory utilization.
Monitoring and alerting go hand in hand. It is not uncommon to monitor more devices and attributes than we want to be alerted about. We then configure alerting thresholds so that we are notified of events that warrant immediate attention. Events that are less critical might not require a real-time response. Additionally, historical information that is collected through the monitoring process will be available for when we need to troubleshoot problems or to determine the relationship between events that we were alerted about.
Monitoring applications, such as Big Brother, can keep track of multiple attributes for resources throughout the network and offer you the flexibility to drill down into specific parameters of individual systems that need to be observed. Let's see how to establish monitoring procedures to track aspects of the environment that are most likely to require a watchful eye:
Hosts and Devices
One of the first steps in setting up a monitoring system is to define which hosts and devices should be watched over. As we discussed a few paragraphs earlier, you need to strike the balance between your ability to handle information and the number of systems that warrant your attention at this stage. So that you see what's involved in configuring core aspects of a monitoring system, let's take a look at how to accomplish this with Big Brother. Big Brother uses a text file called bb-hosts to list the systems it should observe. The bb-hosts file follows the format of the regular hosts file, with the addition of directives that are specific to Big Brother.
For every system that you want to monitor, you can specify which attributes should be observed. A good starting point is ensuring that the host or device is accessible over the network. This can be defined for Big Brother by using a bare-bones bb-hosts file like this:
# # THE BIG BROTHER HOSTS FILE # 192.168.214.132 apple # BBPAGER BBNET BBDISPLAY 192.168.214.17 lemon # 192.168.214.97 peach #
In this bb-hosts file, we defined apple as the host that houses Big Brother components that centrally monitor other systems. (Note that the apple host will be monitored as well.) Because we did not define directives for lemon and peach, Big Brother only checks whether it can access them over the network via an ICMP echo request (ping).
Specifying IP addresses for hosts, as we did in the bb-hosts file, makes the monitoring system more self-contained and decreases the possibility that a domain name resolution problem will skew monitoring results. However, maintaining IP addressto-hostname mappings specifically for the monitoring system creates additional administrative overhead. Furthermore, defining IP addresses within the monitoring system might be impractical when observing hosts that dynamically obtain IP addresses from a DHCP server. To address this, you can configure Big Brother to use the host's native domain name resolution mechanisms, such as DNS or its hosts file. You can accomplish this by simply specifying 0.0.0.0 for IP addresses in the bb-hosts file.
Some systems are configured not to respond to ICMP echo requests, or a firewall between the monitored system and the monitoring host will not let such traffic through. You can use the noping directive to prevent Big Brother from performing the ping test. In such cases, you will probably want to check the accessibility of specific services on the monitored host instead.
Accessibility of Network Services
One of the most useful features of monitoring programs is the ability to determine whether a remote service is accessible over the network. After all, a service might have crashed or have gotten disabled even if its host maintains network connectivity. You can use one of two primary methods to determine the accessibility of a TCP-based service:
Port probes that are based on plain TCP connections obtain information without requiring us to program the monitoring system with knowledge of the protocol specific to the observed service. However, this technique does not help us detect problems with remote applications that accept TCP connections but do not respond properly. By programming the monitoring system to know the application-level protocol of the monitored service, we allow it to detect problems with greater accuracy and granularity. Another advantage of the application-level approach is that it can work with TCP- and UDP-based services.
Monitoring systems usually come with built-in support for determining accessibility of popular network services, such as SMTP, Telnet, FTP, POP, SSH, and IMAP. For example, the following line in Big Brother's bb-hosts file can be used to determine whether SMTP and SSH services run on the remote system plum:
192.168.214.101 plum # smtp ssh
Big Brother usually attempts to evoke a response from the application when testing its availability, instead of simply verifying that it can establish a TCP connection. In most cases, Big Brother accomplishes this by sending the string quit to the remote service. Big Brother can handle some services in a slightly more intelligent manner by attempting to follow their protocols to make the test less obtrusive. For example, it sends ABC123 LOGOUT to IMAP services and sends an identifying string with the prefix Big-Brother-Monitor when connecting to the SSH service.
Sometimes, it makes sense to check whether a particular service is not running on a remote system. For example, if you transfer files only through Secure Shell (SSH), you might want to be alerted if a particular host suddenly starts accepting FTP connections. You can set up Big Brother to do this by using an exclamation mark (!) prefix when specifying directives in bb-hosts, like so:
192.168.214.101 plum # smtp ssh !ftp
When testing accessibility of remote services, do not be surprised to see error messages in logs on the systems being observed. Monitoring tools often do not bother exercising all options of the remote service; they close the connection as soon as they receive some data in return or without exchanging data at all. For example, the Sendmail daemon may record a message such as the following one whenever Big Brother verifies accessibility of the SMTP service:
Big Brother has provisions for verifying service accessibility through plain TCP-based connections instead of exchanging data through a simplistic application check. To configure Big Brother to perform such probes, you can append :s (which stands for "silent") to the applicable directive in bb-hosts, like this:
192.168.214.101 plum # smtp:s ssh
Unfortunately, this approach does not eliminate spurious log messages in many cases because Big Brother often confuses the monitored service by establishing a full TCP connection and not exchanging meaningful data. For example, using the :s suffix to test the SMTP service may not prevent Sendmail from issuing the alarming message shown previously. However, using :s might be effective for eliminating errors in the logs for certain software, so you might want to give this option a shot if the need arises.
Monitoring web servers often warrants special considerations because web-based services are ubiquitous and are frequently critical to company operations. As a result, monitoring applications frequently offer you the ability to retrieve specific web pages. To set this up with Big Brother, you can use the bb-hosts file to specify which URLs to monitor:
192.168.214.17 lemon # https://lemon/cart/order.jsp 192.168.214.97 peach # http://peach/
The preceding configuration lines set up Big Brother to verify the availability of the /cart/order.jsp page on server lemon via HTTPS and to check the root page on server peach via HTTP. Clicking the HTTP icon on Big Brother's summary page enables you to see details of the established HTTP connection, as shown in Figure 19.2.
Figure 19.2. Big Brother can capture details of the HTTP response when connecting to the desired URL (in this case, to http://peach/).
By monitoring applications, you can determine the accessibility of network services without installing additional software on monitored systems. However, obtaining detailed information about local attributes of the monitored resources often requires the further cooperation from the monitored system.
Local System Attributes
Several attributes that are local to the monitored host might warrant your attention not only because they are critical to the host's function, but also because they can serve as an early warning system. Detecting the following conditions might help you discover problems before a host's network service stops being accessible over the network:
One of the most effective ways to observe these attributes is through the use of a monitoring agent that reports its findings to a monitoring server. Figure 19.3 shows a configuration screen of such an agent for Big Brother, with the client module running under Windows. Typical of such configurations is the need to define the central server to which notifications will be sent (in this case, 192.168.214.132).
Figure 19.3. A GUI-based configuration utility defines monitoring settings for Big Brother clients that are running under Windows.
In this example, we set up the Big Brother client to monitor the state of two local services: Event Log and IPSec Policy Agent. Also, the client is set to issue a warning-level alert when the C: partition becomes 90% full and a panic-level alert when the partition's utilization reaches 95%. Even when the system is functioning normally, you can obtain useful information about the state of its resources. For instance, Figure 19.4 shows a status page displayed on Big Brother's central server when the remote host's file system utilization is below alarming thresholds. Status green means everything is OK with this particular attribute of the monitored host.
Figure 19.4. Monitoring systems can offer useful diagnostics information even when remote nodes operate below alarming thresholds.
Another way to obtain detailed information about a monitored system is via SNMP. The use of the application-specific monitoring agents we've discussed so far requires that client software be deployed on systems that need to be observed. One of the biggest advantages of using SNMP is that it is already built in to numerous devices and applications. An SNMP-compatible monitoring server can periodically poll remote SNMP agents for performance and availability attributes, such as configuration parameters, network statistics, and process details. For instance, here are some of the many attributes that Cisco routers natively expose via SNMP:
Universal support for SNMP is one of the biggest advantages of this protocol, as well as a potential weakness. The SNMP vulnerability announced in February 2002, which we mentioned in Chapter 6, "The Role of a Router," simultaneously affected SNMP implementations of numerous vendors. Don't assume that an implementation of an established protocol is bug free. Even if the software has been around for years, you never know what dormant vulnerabilities lie within.
SNMP agents can also issue alerts to the monitoring system when a predefined condition occurs on the local device. These traps are generated asynchronously, regardless of the regular polling schedule the monitoring system implements. Traps can notify the monitoring system of an event that threatens performance or availability of the device. You can use the following command to enable all supported SNMP traps on a Cisco router (you will also need to use the snmp-server host command to specify where the traps should be sent):
router(config)#snmp-server enable traps snmp
One of the notifications that is enabled by using this command is the authentication trap, which alerts administrators when a remote host fails to properly authenticate when attempting to access the SNMP agent. This notification can help detect SNMP network scans and brute force attacks targeting the device over SNMP. If you are not interested in enabling all SNMP traps that the router supports, consider activating at least the authentication trap using the following command:
router(config)#snmp-server enable traps snmp authentication
Big Brother offers limited support for SNMP polling and trap handling through the use of add-on modules that can be downloaded from http://www.deadcat.net.au.
Another method of obtaining detailed information about remote system attributes is available, in addition to using SNMP or custom monitoring agents. This third method involves remotely executing commands that are native to the monitored host. For example, Mercury SiteScope (http://www.mercury.com) can obtain data from remote Windows machines through the PerfMon API, which Microsoft designed to gather performance-related information. To obtain detailed information from UNIX hosts, SiteScope executes commands such as ps and uptime on the remote system via Telnet or SSH. When it comes to remotely executing commands or obtaining potentially sensitive information from the observed systems, you must take care to prevent attackers from exploiting the monitoring channel. We examine such considerations in the next section.
Security Considerations for Remote Monitoring
Attackers can exploit monitoring mechanisms, just like any other service operating on the network, to obtain sensitive information or to gain unauthorized access to remote systems. When selecting and configuring a monitoring solution, consider the following factors relating to its defenses:
Unauthorized access from monitoring agents to the central server is a particularly significant concern because the server is often located in a more secure area than the agents. Consider the configuration in Figure 19.5, where the server on the internal network collects performance and availability information from hosts on the screened subnet. If this scenario were implemented using Big Brother, client modules that run on public servers would be sending notifications to the monitoring server's TCP port 1984 and could exploit the CAN-2000-0450 vulnerability described earlier. (IANA officially assigned port 1984 for Big Brother's client-to-server communications.)
Figure 19.5. The central monitoring server could be exposed to attacks coming from a host on the screened subnet if agents can initiate connections to the server.
To minimize the risk of such attacks, run monitoring services as user accounts with constrained access. Big Brother, for instance, should be set up as a dedicated user without administrative privileges. If possible, also configure the hosts so that connections are initiated from the more secure subnet to limit the possibility of privilege escalation across network security zones. In the case of SNMP, for example, this would involve using the polling mechanisms when observing hosts across subnets as well as limiting trap communications to a single subnet. Monitoring applications might allow you to set up dedicated servers in each subnet for collecting status reports and then link them together; this way, the amount of traffic crossing subnet boundaries is minimized.
Perhaps one of the safest ways to monitor devices is by remotely probing their ports from the central server, as we discussed in the "Accessibility of Network Services" section. This method does not provide detailed information about the local state of the observed machines, but it also does not require that additional software be enabled on remote systems. Moreover, monitoring services that need to be accessible over the network, such as HTTP, DNS, and SMTP, often can be tested over channels that are already open. When a packet-filtering device blocks access to remote services from the network where the monitoring server is located, be sure to open access only to and from IP addresses that are required for monitoring to take place.
Monitoring products out of the box rarely provide many options when it comes to restricting which systems can obtain status information from a remote agent or who can submit status reports to the central server. Most products support restrictions based on IP addresses. To set this up in Big Brother, for example, you need to edit its security file to list addresses of hosts that should be able to communicate with the monitoring server.
Monitoring products can provide supplementary modules that enhance the security of communications between the monitoring components. For example, you can purchase an add-on to HP OpenView Operations, called Advanced Security, to support encryption and cryptographic authentication.
IP-based restrictions form a useful layer for improving the security of the monitoring infrastructure and are most effective when combined with strong authentication in client-to-server communications. Cryptographic authentication is supported in SNMPv3; you should take advantage of it when setting up your monitoring infrastructure. SNMPv3 can also encrypt its messages, which helps to protect their confidentiality. As SNMP agents are beginning to adopt the security mechanisms of SNMPv3, it is becoming easier to find a monitoring system that can support them. Here is a list of some of the products able to securely communicate with SNMPv3 agent modules:
By this point in the chapter, we have discussed the fundamental principles of setting up system and network monitoring in a secure and reliable manner. After such infrastructure is in place, administrators can connect to a central server to view the status of the network's hosts and devices. The next step in maintaining a security perimeter is ensuring that administrators are alerted when critical conditions occur and empowering administrators to quickly respond to such events.