Implementing a Network Management System (NMS)

There is more to hardening your network than merely implementing firewalls, ACLs, and virus protection. No matter how secure your network is, if it isn t functional and reliable, all your efforts are an exercise in futility. A too-often thrown around buzzword in our industry is proactive. We all want to be proactive instead of reactive; however, very few of us manage to accomplish that goal. Instead, we spend our days (and nights) reacting to security incidents and technical failures, and proactive becomes an almost utopian term that is beyond our grasp. It does not have to be that way, however.

A network management system, if properly implemented, not only can help us achieve a goal of proactive management of our network, but it can also help us to secure our network by providing insight into how it operates. Fundamentally, all networks have identifiable patterns or habits. In fact, I often refer to networks as having a personality. In a sense, they are very much like children. For example, many parents seem to have an almost instinctual knowledge as to what is going on with their kids . The reason for this is pretty simple really: As a parent, you are so in tune with your children that you just know when things are not normal. You know your child s personality and habits, and you know when things aren t in line. This doesn t necessarily mean that what happened is a bad thing, but you know that something is up that warrants further investigation.

A well-implemented NMS can provide the same insight as to the habits and personality of your network. If you spend enough time observing and learning the way your network works, it becomes very easy for you to detect when something is occurring that is out of the ordinary. This does not necessarily mean that something wrong has occurred, but it lets you know that something is up that warrants further investigation ”and at that point you make the turn from being reactive to being proactive.

The International Organization for Standardization (ISO) defined five functional areas for network management to help network administrators understand what to do and how to manage their networks. The acronym FCAPS is used to represent these five areas:

Fault management
Configuration management
Performance management
Accounting or asset management
Security management

We are going to look at each functional area and at how we can use it to increase not only the security but also the reliability of our network infrastructure.

Fault Management

Fault management is the process of trouble detection, trouble logging, trouble notification, and trouble correction. Fault management is probably the most widely implemented area of network management because it provides the functionality we all want ”it informs us of a fault or failure that is causing downtime. Most of us do not have the time or desire to spend all day watching screens to tell us if we have experienced a failure somewhere in the network that could be the result of a security incident or could contribute to a security incident. Instead, we want fault management to be an almost transparent process that is used to monitor the network on our behalf and to alert us in the event that something occurs that requires human intervention.

Two primary mechanisms are used for fault management. The first is the use of proactive polling systems to detect a fault. The second is the use of log monitoring to detect that a noteworthy event has been logged.

Using Polling to Detect Faults

Polling is the classic Is it up or is it down? method of fault management. Most products that do polling use ICMP pings or SNMP polling to detect whether the device being monitored is available. If the monitoring station receives a response, the device is considered up. If it does not, the device is considered down. A number of products are available that you can use to poll your network:

Ipswitch WhatsUp Gold (http://www.ipswitch.com/products/ network-management .html)
HP OpenView (http://www.openview.hp.com/index.html)
Solarwinds Orion (http://www.solarwinds.net/)
Micromuse Netcool (http://www.micromuse.com/sols/ent_net_man.html)

Using Ipswitch WhatsUp Gold for Fault Management Ipswitch WhatsUp Gold is one of the leading fault management products and, in fact, is bundled with CiscoWorks to provide fault management functionality for CiscoWorks. Using WhatsUp Gold to monitor your network is surprisingly easy.

The first step to using WhatsUp Gold to monitor your environment is to build a network map. Once you have started WhatsUp Gold from the Start menu, click File New Map Wizard to start the New Map Wizard. At the Device Discovery Information screen, select Discover and Map Network Devices and then click Next .

At the Device Discovery Methods screen, select either Discover Your Network with SNMP SmartScan or Discover Your Network Using ICMP. If your network supports SNMP, I recommend that you use it (and this example will use it). Otherwise, use ICMP. When you are finished, click Next.

At the SNMP SmartScan screen, enter the appropriate SNMP root device, as shown next. The root device is typically the backbone router from which you want to begin the scan. Enter the SNMP communities you want to use. Also, review the scan depth ”setting this value too large will cause a scan to take a very long time in large networks. Under Active Discovery, select whether you want to keep your network maps current with the latest devices. This will cause WhatsUp Gold to automatically scan the network to update the network map with new devices that come online. If you select Scan at Intervals, the discovery will happen at the specified scan interval (the default is 1440 minutes). If you select Scan at Distributed Intervals, the discovery will be spread throughout the scan interval to minimize the network impact. When you are finished, click Next.

Heads Up

Because automatically scanning will add network devices to your map, this can cause the placement of objects on the map to change. If you do not want the discovery of new devices to change the placement of existing devices, make sure you lock the devices in the map properties while in edit mode. You can do this by right-clicking the devices in the edit map and selecting Lock Position from the menu.

At the TCP/IP Service Scan screen, select the services you want to attempt to scan for and discover. At a minimum, you should scan for interfaces. When you are finished, click Next.

The scan will begin. When the scan is completed, you will be presented with the Scan Results screen. Uncheck the devices you do not want to be added to the map and click Finish.

Once the devices have been added to the map, the next step is to configure the map and save the settings. You can rearrange the layout of the devices on the map by clicking the Edit tab. You can reposition the devices by simply clicking and dragging to move them around. You can link devices together by right-clicking an object and selecting Attach To from the menu and then clicking the device to connect to. When you have finished laying out the topology map the way you want it, select all the objects, right-click, and select Lock Position from the menu. This will ensure that the topology does not change when the network is rescanned or new devices are added. When you are finished, save the map.

Once you have configured the device map, the next step is to configure the alerts. This can be done by selecting the Configure menu and clicking Notifications Library. This will open the Notifications Library screen, shown next. The most commonly configured notifications are SMTPMail, Pager, and Beeper . As the names imply, they allow you to configure e-mail- , pager- , and beeper-based alerts. You can configure e-mail “based alerts by editing the default setting or adding a new mail setting. The benefit of adding a new SMTPMail setting is that you can create custom e- mails to send to different groups of users (for example, to generate an alert that goes to the firewall admins when a firewall fault is detected ). When you are finished configuring your alert notifications, click Close.

The next step is to configure the devices to send alerts based on fault detection. Select the device (or devices), right-click, and select Properties from the menu. This will open the Properties screen for the devices you selected, as shown next. Click Alerts in the navigation bar and click Add. Select the notification you want to use and specify a Trigger value. The Trigger value is how many failed checks must occur before an alert is generated. Although you may have a desire to select 1, keep in mind that fault management uses ICMP, and therefore a low value here could cause false positives for any of a number of reasons. When you are finished configuring the alert, click OK. Repeat the process for the other alerts you want to configure for the device (or devices). When you are finished, click OK.

WhatsUp Gold also provides web-based management, which is particularly useful because it does not require you to be sitting at the machine running WhatsUp Gold to get your network status. By default, the web server is not enabled, but you can change this by clicking the Configure menu and selecting Web Server. This will open the Web Server Properties screen, shown next. Under the General category, check the value for Enable Web Server.

Heads Up

WhatsUp Gold does not support HTTPS as a management protocol; therefore, you might not want to allow this traffic to traverse insecure networks, or you might want to encapsulate the management console traffic in IPsec, as detailed later in this chapter.

Like any time you configure a device to be remotely accessible, you need to take steps to secure the device. The first step to securing your WhatsUp Gold web server is to configure the IP security by clicking IP Security from the navigation bar. Select to deny access to all computers, except those listed, and add the IP addresses of the systems that you want to be able to connect to the WhatsUp Gold web server. The next hardening step is to configure the users who will be allowed to log in by clicking Users in the navigation bar. First, delete the guest user by right-clicking it and selecting Delete. Next, delete the admin user. The next step is to add the users you want to be able to connect. I recommend that you use individual user accounts for each user who will be granted permission. Click Add User. At the Add User screen, enter the username and password and select the maps you want this user to be able to view. When you are finished, click OK. When you have returned to the Users category, select the user and configure the appropriate user security settings, as shown next.

In this case, I have granted the user full access to all aspects of WhatsUp Gold. Next, select a map and configure the Map Level Security Settings area, as shown next. Again, in this case, I have granted the user full rights to the map. Repeat this step for each map that the user has access to. When you are finished adding users, click OK.

The final step of getting WhatsUp Gold set up and running with basic functionality in your environment is to configure it to automatically load maps on startup and to start automatically. You can configure the maps to open on startup by selecting the Configure menu and selecting Program Options. In the Startup category, select Open Maps on Startup and add the maps you want to open, as shown next. When you are finished, click OK.

Next, configure WhatsUp Gold to run as a service (this is not enabled by default). Open a command prompt, change to the WhatsUp Gold program directory, and then enter the following command:

 wugsvc -install

This will install WhatsUp Gold as a service and start the service. If you do not do this, you have to run the WhatsUp Gold application in order to monitor your devices.

At this point, you can safely close the WhatsUp Gold application, and it will continue to monitor your environment. If you need to manage WhatsUp Gold, you can simply so from a web browser, as shown here:

Notice that there are devices in alert right now. For example, I can click 192.168.173.96 to drill down and look at what device is down (in this case, the device local-rtr), as shown next.

Using Logging to Detect Faults

In addition to using tools to detect up/down faults on your network, you can use logging to detect faults or error conditions. This is most effectively done through the use of syslog. We covered the fundamentals of how syslog works and how to configure devices to use syslog in their respective chapters, and now we are going to take a look at how you can configure the syslog server to provide you with the information and alerts you require. We will be doing this using Kiwi Enterprises Syslog Daemon 7.0.3 (Build 174).

Using Display Filters The Kiwi Syslog Daemon supports using up to ten displays. These displays come in particularly handy in allowing you to separate the output of your syslog messages by logging levels. For example, you can send all your debug-level (level 7) messages to Display 7 and send all your informational-level (level 6) messages to Display 6.

Like much of Kiwi Syslog s advanced configuration, this can be accomplished through the use of rules. The steps to configure a display rule are as follows :

Run the Kiwi Syslog Service Manager and select File Properties.
You will notice a default rule. This rule causes all messages to be displayed to the Display 0 output, and it logs all data to a file. You can create a new rule by right-clicking Rules in the tree view and selecting Add Rule. Assign the rule a name , as shown here (in this case, I named it Display 7):
Configure a filter to define what will trigger Kiwi Syslog to use this rule. Right-click Filters under the new rule and select Add Filter. Then name the filter accordingly .
In the Field drop-down list, select Priority. In the Filter Type drop-down list, select Priority. Next, select Debug for all the logging facilities. When you are finished, the screen should look like this:
Right-click Actions in the tree view under the rule you created and select Add Action. Then name the action appropriately.
In the Action drop-down field, select Display.
In the Display Number field, select Display 07, as shown here:
When you are finished configuring the rule, click Apply.

Display 07 should display only debug-level values in the Display 07 output (while continuing to display and log debug-level values according to the default rule). Now if you want to look only at debug-level messages, you can select Display 07, as shown next. Repeat this process for each display you want to customize.

Sending Alerts Although display filters provide an excellent method of separating data to make it easier to locate messages, you probably don t have the time to sit in front of your syslog server waiting for important messages to scroll by. Thankfully, Kiwi Syslog provides alerting functionality that allows you to use rules to define a filter and attach an e-mail alert action. The following steps detail this process:

Open the Kiwi Syslog Service Manager properties as previously detailed.
Right-click Rules, select Add Rule, and name the rule accordingly.
Right-click Filters, select Add Filter, and name the filter accordingly.
In the Field drop-down list, select Message Text. In the Filter Type drop-down list, select Simple or Complex. Simple filters are simple one-line filters. Complex filters allow you perform complex include/exclude matching.
In the Include field, enter the text string for which you want to search. Wherever possible, I like to use syslog event severity levels. For the Cisco PIX Firewall, Cisco maintains a list of all the messages at http://www.cisco.com/en/US/customer/products/sw/secursw/ps2120/products_system_message_guide_chapter09186a00801582af.html. For example, if you want to filter for invalid logon attempts, the severity ID is 6-605004 (information level-event number). Enter the string you want to search for, enclosing it in quotes, and select [C] or [S], as shown next. Pressing [C] tells the filter that the string is case sensitive, whereas pressing [S] tells the filter that the data can appear anywhere within the text. You can enter multiple strings by enclosing each in quotes and using a space as a delimiter .
Right-click Actions in the tree view and select Add Action. Then name the action accordingly.
In the Action field, select E-mail Message.
Enter the appropriate information for the e-mail, as shown here:
The last step is to configure Kiwi Syslog to use an SMTP server. This can be done by clicking E-mail in the tree view.
Configure Kiwi Syslog with the appropriate SMTP mail server and a valid from address. If your SMTP server requires it, configure a username and password. You can also configure the syslog server to send an e-mail in the event that the syslog server itself experiences an alarm condition (this is not the same as configuring an e-mail action in a rule). For example, if the syslog server experiences low disk space, it could send an e-mail. You can also configure the syslog server to send a daily statistics e-mail. I highly recommend configuring this so that you can start to get a feel for the amount of daily syslog events the server logs. Finally, configure the e-mail logging options, as shown next. When you are finished, click Apply.

At this point, any time this event occurs, the syslog server will send an e-mail, as shown here:

Simply repeat this process for any other events for which you want to send messages. I recommend that, at a minimum, you configure e-mail alerts for the following Cisco PIX syslog messages:

All Severity Level 1 messages (use the string "%PIX-1" for the filter)
%PIX-2-106016: Deny IP spoof from (IP_address) to IP_address on interface interface_name.
%PIX-2-106017: Deny IP due to Land Attack from IP_address to IP_address
%PIX-2-106018: ICMP packet type ICMP_type denied by outbound list acl_ID src inside_address dest outside_address
%PIX-2-106020: Deny IP teardrop fragment ( size = number, offset = number) from IP_address to IP_address
%PIX-2-304007: URL Server IP_address not responding, ENTERING ALLOW mode.
%PIX-2-304009: Ran out of buffer blocks specified by url-block command
%PIX-2-316001: Denied new tunnel to IP_address. VPN peer limit (platform_vpn_peer_limit) exceeded
%PIX-3-201008: The PIX is disallowing new connections.
%PIX-3-211001: Memory allocation Error
%PIX-3-211003: CPU utilization for number seconds = percent
%PIX-3-302302: ACL = deny; no sa created
%PIX-3-304003: URL Server IP_address timed out URL url
%PIX-3-304006: URL Server IP_address not responding
%PIX-3-315004: Fail to establish SSH session because PIX RSA host key retrieval failed.
%PIX-3-710003: {TCPUDP} access denied by ACL from source_address/source_port to interface_name:dest_address/service
%PIX-4-106023: Deny protocol src [interface_name:source_address/source_port] dst interface_name:dest_address/dest_port [type {string}, code {code}] by access_group acl_ID
%PIX-4-209003: Fragment database limit of number exceeded: src = IP_address,dest = IP_address, proto = protocol, id = number
%PIX-4-209004: Invalid IP fragment, size = bytes exceeds maximum size = bytes: src = IP_address, dest = IP_address, proto = protocol, id = number
%PIX-4-209005: Discard IP fragment set with more than number elements: src = IP_address, dest = IP_address, proto = protocol, id = number
%PIX-4-401004: Shunned packet: IP_address ==> IP_address on interface interface_name
%PIX-4-402103: identity doesn t match negotiated identity (ip) dest_address= dest_address, src_addr= source_address, prot= protocol, (ident) local=inside_address, remote=remote_address, local_proxy=IP_address/IP_address/port/port, remote_proxy=IP_address/IP_address/port/port
%PIX-4-407001: Deny traffic for local-host interface_name:inside_address, license limit of number exceeded
%PIX-5-111001: Begin configuration: IP_address writing to device
%PIX-5-111003: IP_address Erase configuration
%PIX-5-111004: IP_address end configuration: {FAILEDOK}
%PIX-5-111005: IP_address end configuration: OK
%PIX-5-111007: Begin configuration: IP_address reading from device.
%PIX-5-111008: User user executed the command string
%PIX-5-199001: PIX reload command executed from telnet (remote IP_address).
%PIX-5-304001: user source_address Accessed {JAVA URLURL} dest_address: url.
%PIX-5-304002: Access denied URL url SRC IP_address DEST IP_address: url
%PIX-5-500001: ActiveX content modified src IP_address dest IP_address on interface interface_name.
%PIX-5-500002: Java content modified src IP_address dest IP_address on interface interface_name.
%PIX-5-501101: User transitioning priv level
%PIX-5-502101: New user added to local dbase: Uname: user Priv: privilege_level Encpass: string
%PIX-5-502102: User deleted from local dbase: Uname: user Priv: privilege_level Encpass: string
%PIX-5-502103: User priv level changed: Uname: user From: privilege_level To: privilege_level
%PIX-5-612001: Auto Update succeeded:filename, version:number
%PIX-6-109006: Authentication failed for user user from inside_address/inside_port to outside_address/outside_port on interface interface_name.
%PIX-6-109008: Authorization denied for user user from source_address/source_port to destination_address/destination_port on interface interface_name.
%PIX-6-109009: Authorization denied from inside_address/inside_port to outside_address/outside_port (not authenticated) on interface interface_name.
%PIX-6-109015: Authorization denied (acl=acl_ID) for user ˜user from source_address/source_port to dest_address/dest_port on interface interface_name
%PIX-6-308001: PIX console enable password incorrect for number tries (from IP_address)
%PIX-6-309002: Permitted manager connection from IP_address.
%PIX-6-315011: SSH session from IP_address on interface interface_name for user user disconnected by SSH server, reason: reason
%PIX-6-605004: Login denied from {source_address/source_port serial} to {interface_name:dest_address/service console} for user user
%PIX-6-605005: Login permitted from {source_address/source_port serial} to {interface_name:dest_address/service console} for user user
%PIX-6-606001: PDM session number number from IP_address started
%PIX-6-606002: PDM session number number from IP_address ended
%PIX-6-610101: Authorization failed: Cmd: command Cmdtype: command_modifier
%PIX-6-611101: User authentication succeeded: Uname: user
%PIX-6-611102: User authentication failed: Uname: user
%PIX-6-611311: VNPClient: XAUTH Failed: Peer: IP_address
%PIX-7-111009: User user executed cmd:string

Archiving Data It is not good enough to simply log events in your environment. You have to be able to archive the data, and you have to be able to archive the data in such a manner as to ensure data integrity in the event that the data is needed for future legal actions or reasons. Kiwi Syslog includes an archival system that you can expand to provide for long-term archiving and data integrity. It can be accessed via the Kiwi Syslog Daemon Setup properties, as previously detailed. Right-click Archiving, select Add New Archive Schedule, and name the new archive schedule accordingly. By default, the archive schedule is set to occur every 24 hours. You can select a different schedule if you so desire. Specify the source and destination folders for the logs. For the matching file mask, enter the extension used for your log files. By default, this should be .txt. Kiwi Syslog will archive all files that match the extension you enter. Specify the date format and any file/folder options. If you select to use dated folder names, the archives will be stored in folders based on date. If you select to use filenames, the filename will be prefaced with the date format information.

The last step is to configure the archive system to provide for data integrity. This can be done by running an external program that will calculate a hash file based on the log file contents. At a later date, you can verify the data integrity by checking the hash value through recalculating the hash and verifying whether the two values match (for example, by running the command fsum “c < hash file > against the original hash file generated). If the hash value does not match, the file has been manipulated or changed. If the hash does match, the data integrity of the file is ensured. This can be of critical importance for the legal admission of these files into court . The easiest way to generate the hash value is through the use of a batch file. Although you can use any of a number of command-line hash utilities, I will use Slavasoft FSUM (http://www.slavasoft.com/fsum/). For the batch file to work, fsum.exe must be copied to your path . I recommend copying it to %systemroot%\system32 . You also need to create a batch file (I called it shahash.bat) that contains the following lines:

 @echo off fsum -sha1 C:\progra~1\syslogd\Datedl~1\%1 > C:\progra~1\syslogd\Datedl~1\%1.sha

This will cause FSUM to generate an SHA-1 hash for the file that is fed to it via the command-line arguments from Kiwi Syslog:

Don t forget to encapsulate the file that you want to run in parentheses. In addition, if you do not use the %FileShort command-line variable, you will need to edit the batch file accordingly. At this point, you can archive the files to tape or CD-R with the hash file, and as long as the values precisely match when you run the fsum “c command, you can be confident that the log data has not been changed or tampered with.

Configuration Management

Configuration management is the process of providing consistency in your device configurations and having a mechanism to verify if any changes have occurred. Configuration is largely a process of best practices, with some tools and utilities to help enforce those best practices. Here are the five steps to implementing a successful configuration management system in your environment:

Create standards.
Implement standards.
Maintain documentation.
Validate and audit compliance.
Review standards.

Creating Standards

Standardization is the key to a stable network infrastructure. Standards will not only reduce the network complexity, but they will also reduce the amount of unplanned downtime because everyone knows exactly what they should be doing and how devices should be configured. At a minimum, you need to have the following standards in your organization:

Version control and management This will help ensure that you are running consistent software images in your environment.
IP addressing standards and management You should break up your address space and assign ranges of addresses for your device types. For example, you might require that all subnets function on a /24 address space and reserve .1 “.2 for routers, .3 for HSRP addresses, .5 “.9 for switches, and.10 “.254 for DHCP. This will make troubleshooting your environment much easier because you will know what to expect to be using for each address. You also should subnet in a way that is conducive to route summarization.
Naming conventions You should define naming conventions for all your devices and interfaces, including the use of interface descriptions. This will make it extremely easy for you to identify the exact device and interface that may be experiencing a problem as well as providing a good description of what that interface s function is. For example, you might define your routers using a convention that concatenates the location, building, and floor with the router number ”for example, houb02f14rtr01 to refer to Houston, Building 2, Floor 14, Router 1.
Standard configuration The vast majority of your devices can probably share between 75 “90 percent of their configurations, with the only unique aspects being advanced features, such as certain access lists and interfaces, or device-specific configuration settings. This allows you to copy the standard configuration to any new devices, providing only the unique device settings, to ensure that the devices have a consistent configuration. For example, you should specify full duplex operation for all your network devices except your access devices. Access devices should be configured for half-duplex operation. By default, most switches operate in auto-detection mode. By making a change such as this to part of your standard configuration, it makes it much easier to ensure the change is made on all the devices that are implemented. Using standard configurations will also greatly assist in troubleshooting by allowing you to compare configurations between known good and malfunctioning equipment to determine whether the problem is related to a simple misconfiguration.
Configuration upgrade procedures You need to put all these practices into effect by building configuration upgrade procedures that define how to plan, test, and implement changes in your environment. We will explore change control and upgrade procedures in much greater detail in Chapter 14.

Implementing Standards

Once you have defined the standards that should be used in your environment, you have to implement them. Don t have standards simply for the sake of having them or so that your auditors can check a box on their audit form. Ensure that all the systems that are implemented actually follow the defined standards. This will help ensure that your systems are less susceptible to security threats, because your standards should define the most secure methods of implementation and configuration.

Maintaining Documentation

I have mentioned this throughout this book and will not belabor the point again. You need to maintain documentation of every aspect of your network environment. In fact, you should not allow any changes to be made to your network without accompanying documentation, including the following:

Inventory documentation
Configuration version control
AAA configuration logs
Network topology documentation (both physical and logical)
Security policies and procedures

Validating and Auditing Compliance

Once you have taken the time and effort to define, implement, and document the configurations that should be used in your environment, you should periodically audit your environment to ensure compliance. The objective is to verify that the configuration of your devices complies with the standards you have defined. You can follow a number of methods to verify your configurations and audit your network devices, including using vendor tools and utilities (some very expensive) as well as using homegrown scripts that download and diff the configurations (much cheaper, but time consuming to write and test the scripts). Here s a list of some vendor tools for automated configuration management, compliance, and auditing:

Cisco CiscoWorks (http://www.cisco.com/en/US/customer/products/sw/cscowork/index.html)
OPNET Netdoctor (http://www. opnet .com/products/modules/netdoctor.html)
AlterPoint DeviceAuthority (http://www.alterpoint.com/products/index.html)
Ecora Enterprise Auditor (http://www.ecora.com/ecora/products/jump/auditor1.asp)
Visonael Network Audit (http://www.visionael.com/products/ info /network_audit.pdf)
Cisco Router Audit Tool (http://www.cisecurity.org)

Note	Sample scripts for downloading configurations are provided in Chapter 14.

In addition to implementing tools and utilities that will audit your environment and verify your configurations, you can also use syslog to proactively alert you in the event that an IOS-based device or Cisco PIX firewall configuration has changed. You can do this by configuring rules for Kiwi Syslog, as previously demonstrated, using the following syslog values:

%SYS-5-CONFIG: Configured from [chars]
%SYS-5-CONFIG_I: Configured from [chars] by [chars]
%SYS-5-CONFIG_NV: Nonvolatile storage configured from [chars]
%SYS-5-CONFIG_NV_I: Nonvolatile storage configured from [chars] by [chars]
%PIX-5-111007: Begin configuration: IP_address reading from device.
%PIX-5-111008: User user executed the command string

For example, you could parse for configured from as the search string for all your IOS-based devices and 5-111007 and 5-111007 for your PIX firewalls.

Reviewing Standards

Like all aspects of security, you should periodically review your configuration management policies to ensure not only that they remain current and address recent security issues and updates, but that they are indeed functional standards and are being adhered to.

Performance Management

Performance management is an often-overlooked aspect of hardening your network infrastructure. This is largely due to the fact that many folks look at performance management as being something that is only really useful in ensuring that the network meets the appropriate service-level agreements. Although this is certainly the major reason for implementing performance management, performance management has the additional ability to be a precursor to future potential network problems.

In addition, as you ll recall, I mentioned earlier that learning the habits and personality of your network can help you know what is going on with your network. Performance management is how you learn those habits.

A proper performance management system will monitor, measure, and report on all aspects of your network infrastructure. This is typically done through the use of the following protocols:

Simple Network Management Protocol (SNMP)
RMON
Cisco NetFlow
Cisco IOS Service Assurance Agent (SAA)

Using SNMP for Performance Management

SNMP is the conventional method of providing performance-related data on your network devices. A number of SNMP MIB values are particularly valuable for providing security information, including (listing OID shortnames) the following:

SysUpTime This value provides information about how long the system has been up. It can indicate a compromise or failure condition that caused a reboot.
ifInOctets and ifOutOctets These values provide the total number of octets received or transmitted on the interface, including framing characters .
ifSpeed Although not of particular use by itself, ifSpeed provides the bandwidth of the interface, which can be used to calculate bandwidth utilization. For an example of how to do this, see http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a008009496e.shtml.
ifInDiscards and ifOutDiscards These are packets that were discarded even though no errors had been detected. They can indicate a problem with the device input or output buffers being full. This can be a symptom of a denial of service attack.
ifOutQLen This value represents the length of the output queue, and it can indicate a problem with the buffer getting full, which can be a symptom of a denial of service attack.

Some Cisco-specific MIB values to monitor are listed here:

locIfInputQueueDrops and locIfOutputQueueDrops These values represent the number of packets dropped because the input or output queue was full. They can indicate a potential denial of service that is attempting to flood the network device.
locIfarpInPkts and locIfarpOutPkts These values represent the ARP traffic that is being seen by the device. It is normal to see this value increment as devices ARP for resources. If these values make a marked increase or remain consistent (that is, they never increase), this could be a symptom of a denial of service.
busyPer, avgBusy1, and avgBusy5 These values list the CPU busy percentage for the last five seconds, one minute, and five minutes, respectively, and can be an indicator that something is causing excessive CPU utilization. This was a symptom of CodeRed and many of the worms that resulted in ARP-based denial of service attacks.

You can use a number of network management systems to monitor your network devices using SNMP, including the following:

Multi Router Traffic Grapher (MRTG) (http://people.ee.ethz.ch/~oetiker/webtools/mrtg/)
PATROL DashBoard (http://www.bmc.com/products/products_services_detail/0,,0_0_0_22,00.html)
Concord eHealth (http://www.concord.com/solutions/ent/network/)
HP OpenView (http://www.openview.hp.com)
Micromuse Netcool (http://www.micromuse.com/sols/ent_net_man.html)

Using RMON, Cisco NetFlow, and SAA for Performance Management

RMON, Cisco NetFlow, and SAA pick up where SNMP leaves off. SNMP does an excellent job of showing you what is happening on a device or an interface. RMON, Cisco NetFlow, and SAA take it a step further and display the traffic flows throughout the network enterprise. This allows you to correlate the network traffic so that you can view the entire conversation between two hosts throughout your network. For example, you can identify a host that is attempting to communicate with all the hosts on your network by seeing that multiple traffic flows are coming from this single host. This can be a symptom of a worm attempting to replicate throughout your network. With the exception of MRTG, all the previously mentioned tools will display RMON, Cisco NetFlow, and Cisco SAA data. In addition, BMC Software offers PATROL Visualis to provide topology mapping to support these protocols, and in the open-source arena you have NTOP (http://www.ntop.org/) as the counterpart to MRTG.

Accounting or Asset Management

Accounting or asset management is the process of measuring your network usage so that individuals or groups of users can be charged appropriately based on their share of the network utilization. Although not a traditional security process, the same Cisco NetFlow and IP Accounting data (as detailed in Chapter 8) can be used as a very basic indication of what your users are doing. Some vendors of accounting software include the following:

Evident Software (http://www.evidentsoftware.com/)
Amdocs XACCT Solutions (http://www.xacct.com/)
NetScout nGenius Probes (http://www.netscout.com/products/probes_home.asp)

Security Management

Security management is really what this entire book has focused on. From hardening your network devices to implementing AAA to controlling network access through the use of access control lists, security management is the goal and objective. I will refer you to any of the other chapters of this book for more security management information.