System and Network Monitoring

In the context of perimeter security, the primary goal of system and network monitoring is to provide administrators with the awareness of the state of a protected environment. We discussed monitoring in the context of intrusion detection in Chapter 8, "Network Intrusion Detection," and Chapter 10, "Host Defense Components," where we tuned IDS sensors to automatically detect events that are frequently indicative of misuse. System and network monitoring also involve anomaly detection but concentrate on availability and performance parameters of the resources. Security professionals should be interested in seemingly mundane issues, such as server uptime, network utilization, and disk space availability for numerous reasons:

  • Monitoring helps detect interruptions in availability of the services that the security perimeter is meant to protect. A host's partition might have filled up, a critical process might have died, or a denial of service attack might have taken a system offline. Ensuring availability, as you might recall, is one of the key objectives of information security.

  • Monitoring helps detect changes in the performance and availability of the system, which might be indicative of misuse. Administrators who routinely monitor their systems are able to sense even the slightest change in the behavior of the host or device and are wonderful intrusion detection sensors.

  • Monitoring helps detect bottlenecks and shifts in performance of the infrastructure, which might indicate that the design or the implementation of the security perimeter is not properly tuned. For example, a web-based application might function too slowly over HTTPS if its servers cannot cope with the load that SSL imposes on them. Monitoring can help determine which aspects of the security perimeter need to be fine-tuned and in what way.

  • Monitoring helps detect unexpected changes in the environment that might adversely impact the effectiveness of the security perimeter. For example, a new server might have been brought online without proper hardening, or hasty changes to the firewall rule set might have unintentionally blocked access to a critical service.

A well-defined monitoring process allows administrators to detect problems at an early stagehopefully before the problems escalate into critical incidents. One of the most powerful ways to increase effectiveness of this process is to perform monitoring centrally, which generally involves configuring a serveror a cluster of unified serversto observe multiple resources throughout the network. A centralized monitoring infrastructure often benefits from the use of client modules that report to the central server information gathered from remotely monitored systems. You should already be familiar with such a configuration because it is frequently used to manage multiple firewalls, VPN nodes, or IDS sensors centrally.

In this section, we look at a typical way of implementing a centralized monitoring infrastructure. To illustrate monitoring concepts, we use a popular monitoring application called Big Brother; however, the same monitoring principles apply to other applications of this nature.

Big Brother Fundamentals

Big Brother is a flexible software tool for performing network- and system-monitoring functions ( Numerous products can fulfill monitoring needs; a small sample of these includes BMC PATROL (, HP OpenView (, IBM Tivoli (, and Nagios ( Unlike several other contenders, Big Brother focuses solely on monitoring aspects of system and network maintenance, without offering features to remotely manage devices. This makes it a lightweight application that is particularly well suited for demonstrating monitoring concepts. As a result, we use Big Brother in the examples throughout this section for many of the reasons we would consider using it on our networks:

  • It is free for noncommercial use.

  • It is relatively inexpensive for commercial use.

  • It runs on Windows and UNIX-based operating systems.

  • It is relatively easy to set up and maintain.

  • It supports a wide range of monitoring and alerting requirements.

  • It is expandable through the use of custom scripts.

  • It has been around since late 1996 and has established itself as a trustworthy monitoring tool.

Big Brother is able to track the status of remote nodes by attempting to connect to monitored systems over the network. In this role, Big Brother can detect the availability of a system via ping packets, as well as by attempting to elicit an expected response from specific TCP services. Big Brother also supports the use of agent modules that run on remote nodes as Big Brother clients and gather local information, such as CPU performance, disk space utilization, and process status.


The commercial version of Big Brother, called Big Brother Professional Edition, supports several enhancements to the free version. In particular, the Professional Edition offers a simplified installation routine and point-to-point encryption. You can purchase the commercial version from Quest Software (

Big Brother's architecture consists of four core components that are integrated into a unified application. These components, described in the following list, can run on distributed systems to achieve the desired extent of scalability or can coexist on a single server:

  • The BBNET host probes remote systems to determine the availability of monitored network services.

  • The remote client agent runs on a monitored system and gathers performance and resource availability data local to its host.

  • The BBDISPLAY host generates and displays web pages that administrators can look at to determine the status of monitored devices, services, and processes.

  • The BBPAGER host issues notification alerts and processes event-escalation measures.

If Big Brother determines that a monitored parameter has exceeded an acceptable threshold, it alerts administrators by displaying a warning on the BBDISPLAY's web page. Figure 19.1 shows how Big Brother consolidates status information regarding multiple systems onto a single summary page. Icons on Big Brother's summary matrix represent the current status of monitored parameters for each system. A green square represents the normal state of the system, which is the case for most services in this figure. A flashing star icon for the HTTP service on "lemon" indicates that Big Brother experienced problems connecting to the web server on "lemon" over the network. Administrators can obtain additional information regarding the problem by clicking the icon. Additionally, Big Brother can send the alert via email, pager, Short Message Service (SMS), or a custom script.

Figure 19.1. Big Brother presents a unified view of monitored resources by using icons to represent the status of observed attributes.

Now that you know a little about fundamental capabilities of monitoring software, the next section covers how to determine which resources should be observed in the first place and how to set up such monitoring.

The Capabilities of HP OpenView

Of course, Big Brother isn't the only game in town when it comes to monitoring networks and systems. A wide variety of offerings are available to suit different requirements and budgets. HP OpenView is a product at the higher end of the price and functionality spectrum. The foundation of the OpenView family of products is the OpenView Network Node Manager (NNM). This application provides the framework for performing basic monitoring and discovery functions by using Simple Network Management Protocol (SNMP), ping, and other network probes, as well as for collecting historical data and handling event notification.

An extension of NNM, which is more full featured and expensive, is a product called OpenView Operations. This application is aimed at enterprises that require more extensive managing and monitoring features, along with native support for additional devices and applications. OpenView Operations also supports programmable actions that can respond to system events, such as by automatically restarting a crashed process. A core component in OpenView Operations, called Service Navigator, allows administrators to "teach" OpenView about relationships between particular services distributed throughout the network. For example, if a router goes down, Service Navigator (if appropriately configured) can flag all the services this event might affect, such as email or a mission-critical database. This functionality can help to assess the event's impact, determine its root cause, and assist in the troubleshooting process.

HP OpenView isn't for everyone due to its high price and significant implementation complexities. However, if a particular feature is important to your business and slimmer or less expensive tools do not support it, you should look into enterprise-centric solutions such as HP OpenView.

Establishing Monitoring Procedures

Be sure to define your goals and priorities before rolling out a monitoring solution. For example, if you are maintaining an e-commerce site that makes a significant portion of its revenue via the Internet, you might be interested in focusing on the following data points:

  • Bandwidth utilization

  • Server load

  • Where your hits are coming from

  • What percentage of site visitors purchase a product

Considering that most of us operate under the constraints of limited staff, it is impractical to treat all changes in the state of the environment with the same urgency. Moreover, monitoring all aspects of performance and availability on each system is likely to produce so many alerts that those that are truly important will be drowned out by the noise of inconsequential events. Knowing when a firewall process dies is likely to be more important than receiving notifications every time a user's workstation is rebooted.

We have been using risk analysis to define the extent of hardening and segregation for security perimeter components throughout the book. Similarly, we can define monitoring requirements based on the resource's importance and the likelihood that its availability might be disrupted. For example, if email communications are a critical business function for your company, you might consider monitoring numerous aspects of the mail server's availability and performance, such as the accessibility of the host over the network, the status of the mailer's processes, and the state of file system utilization. If that same mail server has been acting flaky, you might be interested in adjusting monitoring criteria to help track down the cause of the problem or to enable yourself to react to problems more quickly. In that case, it might help to adjust the thresholds for alerts or to monitor additional attributes, such as auxiliary system processes, as well as CPU and memory utilization.


Monitoring and alerting go hand in hand. It is not uncommon to monitor more devices and attributes than we want to be alerted about. We then configure alerting thresholds so that we are notified of events that warrant immediate attention. Events that are less critical might not require a real-time response. Additionally, historical information that is collected through the monitoring process will be available for when we need to troubleshoot problems or to determine the relationship between events that we were alerted about.

Monitoring applications, such as Big Brother, can keep track of multiple attributes for resources throughout the network and offer you the flexibility to drill down into specific parameters of individual systems that need to be observed. Let's see how to establish monitoring procedures to track aspects of the environment that are most likely to require a watchful eye:

  • Hosts and devices

  • Accessibility of network services

  • Local system attributes

Hosts and Devices

One of the first steps in setting up a monitoring system is to define which hosts and devices should be watched over. As we discussed a few paragraphs earlier, you need to strike the balance between your ability to handle information and the number of systems that warrant your attention at this stage. So that you see what's involved in configuring core aspects of a monitoring system, let's take a look at how to accomplish this with Big Brother. Big Brother uses a text file called bb-hosts to list the systems it should observe. The bb-hosts file follows the format of the regular hosts file, with the addition of directives that are specific to Big Brother.

For every system that you want to monitor, you can specify which attributes should be observed. A good starting point is ensuring that the host or device is accessible over the network. This can be defined for Big Brother by using a bare-bones bb-hosts file like this:

 # # THE BIG BROTHER HOSTS FILE # apple   # BBPAGER BBNET BBDISPLAY  lemon   #  peach   # 

In this bb-hosts file, we defined apple as the host that houses Big Brother components that centrally monitor other systems. (Note that the apple host will be monitored as well.) Because we did not define directives for lemon and peach, Big Brother only checks whether it can access them over the network via an ICMP echo request (ping).

Specifying IP addresses for hosts, as we did in the bb-hosts file, makes the monitoring system more self-contained and decreases the possibility that a domain name resolution problem will skew monitoring results. However, maintaining IP addressto-hostname mappings specifically for the monitoring system creates additional administrative overhead. Furthermore, defining IP addresses within the monitoring system might be impractical when observing hosts that dynamically obtain IP addresses from a DHCP server. To address this, you can configure Big Brother to use the host's native domain name resolution mechanisms, such as DNS or its hosts file. You can accomplish this by simply specifying for IP addresses in the bb-hosts file.

Some systems are configured not to respond to ICMP echo requests, or a firewall between the monitored system and the monitoring host will not let such traffic through. You can use the noping directive to prevent Big Brother from performing the ping test. In such cases, you will probably want to check the accessibility of specific services on the monitored host instead.

Accessibility of Network Services

One of the most useful features of monitoring programs is the ability to determine whether a remote service is accessible over the network. After all, a service might have crashed or have gotten disabled even if its host maintains network connectivity. You can use one of two primary methods to determine the accessibility of a TCP-based service:

  • Check whether the monitored port is open by initiating a plain TCP connection with the service without exchanging data. You can do this with a half-connect, in which the monitoring server does not complete a three-way TCP handshake. Most monitoring systems, however, complete the handshake and close the connection as soon as it is established.

  • Attempt to initiate a conversation with the remote service using the application-level protocol that is appropriate for the monitored service. For example, we could check the accessibility of a POP service by actually attempting to access a mailbox.

Port probes that are based on plain TCP connections obtain information without requiring us to program the monitoring system with knowledge of the protocol specific to the observed service. However, this technique does not help us detect problems with remote applications that accept TCP connections but do not respond properly. By programming the monitoring system to know the application-level protocol of the monitored service, we allow it to detect problems with greater accuracy and granularity. Another advantage of the application-level approach is that it can work with TCP- and UDP-based services.

Monitoring systems usually come with built-in support for determining accessibility of popular network services, such as SMTP, Telnet, FTP, POP, SSH, and IMAP. For example, the following line in Big Brother's bb-hosts file can be used to determine whether SMTP and SSH services run on the remote system plum: plum   # smtp ssh 

Big Brother usually attempts to evoke a response from the application when testing its availability, instead of simply verifying that it can establish a TCP connection. In most cases, Big Brother accomplishes this by sending the string quit to the remote service. Big Brother can handle some services in a slightly more intelligent manner by attempting to follow their protocols to make the test less obtrusive. For example, it sends ABC123 LOGOUT to IMAP services and sends an identifying string with the prefix Big-Brother-Monitor when connecting to the SSH service.


Sometimes, it makes sense to check whether a particular service is not running on a remote system. For example, if you transfer files only through Secure Shell (SSH), you might want to be alerted if a particular host suddenly starts accepting FTP connections. You can set up Big Brother to do this by using an exclamation mark (!) prefix when specifying directives in bb-hosts, like so: plum   # smtp ssh !ftp 

When testing accessibility of remote services, do not be surprised to see error messages in logs on the systems being observed. Monitoring tools often do not bother exercising all options of the remote service; they close the connection as soon as they receive some data in return or without exchanging data at all. For example, the Sendmail daemon may record a message such as the following one whenever Big Brother verifies accessibility of the SMTP service:

[View full width]

Feb 24 16:50:02 apple sendmail[23130]: NOQUEUE: apple [] did not issue MAIL /EXPN/VRFY/ETRN during connection to MTA

Big Brother has provisions for verifying service accessibility through plain TCP-based connections instead of exchanging data through a simplistic application check. To configure Big Brother to perform such probes, you can append :s (which stands for "silent") to the applicable directive in bb-hosts, like this: plum   # smtp:s ssh 

Unfortunately, this approach does not eliminate spurious log messages in many cases because Big Brother often confuses the monitored service by establishing a full TCP connection and not exchanging meaningful data. For example, using the :s suffix to test the SMTP service may not prevent Sendmail from issuing the alarming message shown previously. However, using :s might be effective for eliminating errors in the logs for certain software, so you might want to give this option a shot if the need arises.

Logging Incomplete OpenSSH Connections

The OpenSSH daemon may create the following log entry when a client initiates a connection to its port and closes the socket without going through the full SSH connection negotiation process:

[View full width]

Feb 24 16:51:03 apple sshd[19925]: Did not receive ident string from

This message might indicate that a monitoring system is configured to observe the SSH service via plain TCP connections. The message is different when the system attempts to exchange data with the remote SSH service. Here's an example:

[View full width]

Feb 24 16:52:46 apple sshd[1295]: Bad protocol version identification 'Big-Brother-Monitor-1.9e' from

Of course, similar messages might appear in the logs when the SSH server is subjected to a port scan, so be careful not to blindly disregard these messages when examining the logs.

Monitoring web servers often warrants special considerations because web-based services are ubiquitous and are frequently critical to company operations. As a result, monitoring applications frequently offer you the ability to retrieve specific web pages. To set this up with Big Brother, you can use the bb-hosts file to specify which URLs to monitor:  lemon   # https://lemon/cart/order.jsp  peach   # http://peach/ 

The preceding configuration lines set up Big Brother to verify the availability of the /cart/order.jsp page on server lemon via HTTPS and to check the root page on server peach via HTTP. Clicking the HTTP icon on Big Brother's summary page enables you to see details of the established HTTP connection, as shown in Figure 19.2.

Figure 19.2. Big Brother can capture details of the HTTP response when connecting to the desired URL (in this case, to http://peach/).

By monitoring applications, you can determine the accessibility of network services without installing additional software on monitored systems. However, obtaining detailed information about local attributes of the monitored resources often requires the further cooperation from the monitored system.

Local System Attributes

Several attributes that are local to the monitored host might warrant your attention not only because they are critical to the host's function, but also because they can serve as an early warning system. Detecting the following conditions might help you discover problems before a host's network service stops being accessible over the network:

  • File system is almost full.

  • CPU utilization has exceeded an acceptable threshold.

  • Critical processes are failing to run.

One of the most effective ways to observe these attributes is through the use of a monitoring agent that reports its findings to a monitoring server. Figure 19.3 shows a configuration screen of such an agent for Big Brother, with the client module running under Windows. Typical of such configurations is the need to define the central server to which notifications will be sent (in this case,

Figure 19.3. A GUI-based configuration utility defines monitoring settings for Big Brother clients that are running under Windows.

In this example, we set up the Big Brother client to monitor the state of two local services: Event Log and IPSec Policy Agent. Also, the client is set to issue a warning-level alert when the C: partition becomes 90% full and a panic-level alert when the partition's utilization reaches 95%. Even when the system is functioning normally, you can obtain useful information about the state of its resources. For instance, Figure 19.4 shows a status page displayed on Big Brother's central server when the remote host's file system utilization is below alarming thresholds. Status green means everything is OK with this particular attribute of the monitored host.

Figure 19.4. Monitoring systems can offer useful diagnostics information even when remote nodes operate below alarming thresholds.

Another way to obtain detailed information about a monitored system is via SNMP. The use of the application-specific monitoring agents we've discussed so far requires that client software be deployed on systems that need to be observed. One of the biggest advantages of using SNMP is that it is already built in to numerous devices and applications. An SNMP-compatible monitoring server can periodically poll remote SNMP agents for performance and availability attributes, such as configuration parameters, network statistics, and process details. For instance, here are some of the many attributes that Cisco routers natively expose via SNMP:

  • MAC and IP address accounting

  • CPU utilization

  • Memory availability

  • Startup and running configuration

  • Network interface status


Universal support for SNMP is one of the biggest advantages of this protocol, as well as a potential weakness. The SNMP vulnerability announced in February 2002, which we mentioned in Chapter 6, "The Role of a Router," simultaneously affected SNMP implementations of numerous vendors. Don't assume that an implementation of an established protocol is bug free. Even if the software has been around for years, you never know what dormant vulnerabilities lie within.

SNMP agents can also issue alerts to the monitoring system when a predefined condition occurs on the local device. These traps are generated asynchronously, regardless of the regular polling schedule the monitoring system implements. Traps can notify the monitoring system of an event that threatens performance or availability of the device. You can use the following command to enable all supported SNMP traps on a Cisco router (you will also need to use the snmp-server host command to specify where the traps should be sent):

 router(config)#snmp-server enable traps snmp 

One of the notifications that is enabled by using this command is the authentication trap, which alerts administrators when a remote host fails to properly authenticate when attempting to access the SNMP agent. This notification can help detect SNMP network scans and brute force attacks targeting the device over SNMP. If you are not interested in enabling all SNMP traps that the router supports, consider activating at least the authentication trap using the following command:

 router(config)#snmp-server enable traps snmp authentication 


Big Brother offers limited support for SNMP polling and trap handling through the use of add-on modules that can be downloaded from

Versions of SNMP

SNMP has been around since 1988 and has been exposed to scrutiny by standards bodies, network engineers, and system administrators. As a result, numerous revisions have been made to the original SNMP protocol, but some of them failed to gain the community's acceptance and were phased away. One of the more popular phased-out SNMP provisions was the party-based authentication specification of SNMPv2 (now called SNMPv2p or SNMPv2 classic), which attempted to provide a more secure alternative to SNMPv1's community string authentication. Cisco routers used to support party-based authentication through the use of the snmp-server party command but stopped supporting it after release 11.2 of IOS.1 Instead, Cisco followed the example of many other vendors in limiting itself to handling only the SNMPv1-style community string authentication defined by SNMPv2c.

Additionally, Cisco adopted a full implementation of SNMPv3, which includes a rich set of cryptographic security mechanisms. One of the few major vendors that still supports the enhanced security framework of the SNMPv2 generation is Sun, which uses user-based authentication mechanisms defined by SNMPv2u in its Sun Management Center product. (Sun sometimes refers to this protocol as SNMPv2usec.)2

Another method of obtaining detailed information about remote system attributes is available, in addition to using SNMP or custom monitoring agents. This third method involves remotely executing commands that are native to the monitored host. For example, Mercury SiteScope ( can obtain data from remote Windows machines through the PerfMon API, which Microsoft designed to gather performance-related information. To obtain detailed information from UNIX hosts, SiteScope executes commands such as ps and uptime on the remote system via Telnet or SSH. When it comes to remotely executing commands or obtaining potentially sensitive information from the observed systems, you must take care to prevent attackers from exploiting the monitoring channel. We examine such considerations in the next section.

Security Considerations for Remote Monitoring

Attackers can exploit monitoring mechanisms, just like any other service operating on the network, to obtain sensitive information or to gain unauthorized access to remote systems. When selecting and configuring a monitoring solution, consider the following factors relating to its defenses:

  • How sensitive are the monitoring communications that traverse the network, and should they be encrypted? You might not be concerned about the confidentiality of CPU statistics carried across the internal network, but you might not want to use clear-text SNMP to retrieve a router's running configuration across the Internet.

  • How strong is the authentication scheme that prevents unauthorized entities from accessing monitored resources? Many monitoring solutions rely solely on IP addresses for access control. Additionally, SNMPv1 and SNMPv2c support only the use of community strings, which can be intercepted over the network or brute-forced via tools such as SolarWinds (

  • Do monitoring mechanisms provide write access to observed resources? Even though we have been focusing on read-only aspects of system monitoring, SNMP can also be used to modify settings on remote devices. For example, an attacker with write access to a resource might be able to manipulate a router's ARP table or otherwise change the configuration of the system.

  • Can the monitoring infrastructure be exploited to grant attackers elevated access to the systems? For example, a vulnerability in Big Brother allowed execution of arbitrary commands on the monitoring server with the privileges of the Big Brother user ( Even if the server were restricted to accept status information from only specific IP addresses, systems hosting Big Brother agents could have been used to obtain access to the monitoring server.

Unauthorized access from monitoring agents to the central server is a particularly significant concern because the server is often located in a more secure area than the agents. Consider the configuration in Figure 19.5, where the server on the internal network collects performance and availability information from hosts on the screened subnet. If this scenario were implemented using Big Brother, client modules that run on public servers would be sending notifications to the monitoring server's TCP port 1984 and could exploit the CAN-2000-0450 vulnerability described earlier. (IANA officially assigned port 1984 for Big Brother's client-to-server communications.)

Figure 19.5. The central monitoring server could be exposed to attacks coming from a host on the screened subnet if agents can initiate connections to the server.

To minimize the risk of such attacks, run monitoring services as user accounts with constrained access. Big Brother, for instance, should be set up as a dedicated user without administrative privileges. If possible, also configure the hosts so that connections are initiated from the more secure subnet to limit the possibility of privilege escalation across network security zones. In the case of SNMP, for example, this would involve using the polling mechanisms when observing hosts across subnets as well as limiting trap communications to a single subnet. Monitoring applications might allow you to set up dedicated servers in each subnet for collecting status reports and then link them together; this way, the amount of traffic crossing subnet boundaries is minimized.

Perhaps one of the safest ways to monitor devices is by remotely probing their ports from the central server, as we discussed in the "Accessibility of Network Services" section. This method does not provide detailed information about the local state of the observed machines, but it also does not require that additional software be enabled on remote systems. Moreover, monitoring services that need to be accessible over the network, such as HTTP, DNS, and SMTP, often can be tested over channels that are already open. When a packet-filtering device blocks access to remote services from the network where the monitoring server is located, be sure to open access only to and from IP addresses that are required for monitoring to take place.

Monitoring products out of the box rarely provide many options when it comes to restricting which systems can obtain status information from a remote agent or who can submit status reports to the central server. Most products support restrictions based on IP addresses. To set this up in Big Brother, for example, you need to edit its security file to list addresses of hosts that should be able to communicate with the monitoring server.


Monitoring products can provide supplementary modules that enhance the security of communications between the monitoring components. For example, you can purchase an add-on to HP OpenView Operations, called Advanced Security, to support encryption and cryptographic authentication.

IP-based restrictions form a useful layer for improving the security of the monitoring infrastructure and are most effective when combined with strong authentication in client-to-server communications. Cryptographic authentication is supported in SNMPv3; you should take advantage of it when setting up your monitoring infrastructure. SNMPv3 can also encrypt its messages, which helps to protect their confidentiality. As SNMP agents are beginning to adopt the security mechanisms of SNMPv3, it is becoming easier to find a monitoring system that can support them. Here is a list of some of the products able to securely communicate with SNMPv3 agent modules:

  • HP OpenView and IBM Tivoli, with the help of an add-on called SNMP Security Pack (

  • Castle Rock SNMPc Network Manager (

  • AdventNet SNMP API for implementing SNMP support in Java-based applications (

  • MG-SOFT SNMP management software (

  • Net-SNMP, a popular set of free SNMP utilities for UNIX, utilized by applications such as Big Brother (

By this point in the chapter, we have discussed the fundamental principles of setting up system and network monitoring in a secure and reliable manner. After such infrastructure is in place, administrators can connect to a central server to view the status of the network's hosts and devices. The next step in maintaining a security perimeter is ensuring that administrators are alerted when critical conditions occur and empowering administrators to quickly respond to such events.

    Inside Network Perimeter Security
    Inside Network Perimeter Security (2nd Edition)
    ISBN: 0672327376
    EAN: 2147483647
    Year: 2005
    Pages: 230

    Similar book on Amazon © 2008-2017.
    If you may any questions please contact us: