The problem areas of the Security Monitor can be categorized as follows:
The Security Monitor can be installed through either of the two following scenarios:
The installation procedures for different versions of the Security Monitor are listed in the following link and are not be discussed here.
You must install the Common Services first before proceeding with the Security Monitor installation. Core provides authentication and user management, and a host of other services. As mentioned before, when both Security Monitor and the IDS/IPS MC are installed on the same system, sensors configured to send their events to Security Monitor can be imported from the IDS/IPS MC. Otherwise, every sensor monitored by Security Monitor needs to be added manually. PostOffice devices must be in the device table to receive those events. Syslog devices do not need to be in the device table, but if they are, the device name will be displayed in the event viewer and reports.
Troubleshooting techniques related to the installation of Security Monitor are similar to those of IDS/IPS MC which is discussed in Chapter 18, "Troubleshooting IDM and IDS/IPS Management Console (IDS/IPS MC)" under the section entitled "Installation and upgrade issues." Therefore, the troubleshooting techniques are not discussed here.
Issues with Launching
To launch Security Monitor, first log in to Security Monitor server from your Web browser: http://localhost:1741.
Then, launch the Security Monitor by clicking VPN/Security Management Solution > Monitoring Center > Security Monitor.
When you launch, Security Monitor will land on the device table as shown in Figure 22-2.
Figure 22-2. Security Monitor Shown After the Login
Following is the list of tabs available for different purposes in the Security Monitor GUI:
There might be several reasons for being unable to launch the Security Monitor or the CiscoWorks Common Services in general. The sections that follow explain some of these reasons.
If you cannot launch CiscoWorks Common Services or the Security Monitor with the IP address, try the following:
Issues with Enabling SSL
When SSL is enabled for CiscoWorks Common Services, its certificate must be selected for the Security Monitor. Otherwise you will get a Java error when launching the event viewer and the untrusted server cert chain error message will be displayed. A certificate is created when SSL is turned on in CiscoWorks Common Services. Since the Event Viewer uses the Java plug-in provided by the Cisco Works Common Services server, the Security Monitor server needs knowledge of this certificate before the Event viewer applet is launched.
Work through the following steps to enable SSL:
If you are stopping and starting the daemon manager from the Process Management folder of the CiscoWorks Common Services GUI fails, you can perform this task from the service control applet or by using the net stop crmdmgtd and net start crmdmgtd commands in a command prompt window.
When you try to login for the first time, CiscoWorks Common Services shows a security alert for the CA root certificate. If this happens, view the certificate and install it before proceeding. If this warning does not appear, it probably means that you have already installed the certificate. You can proceed without installing the certificate by clicking Yes. There are no negative affects to this acceptance; however the warning message may be displayed again.
Getting Internal Server Error While Opening Security Monitor
If you receive an Internal Server Error message, first be sure that you have enough space on the drive where you have installed Security Monitor. For example, if VMS is installed at E:\PROGRA~1\CSCOpx, then you must have enough space on the E:\ drive.
MDCSupport Zip file does not show how much disk space is available for drive E:. However, SysEventLog.evt will contain messages that E:\is at or near capacity. On the IDS_Receiver.log, you might see the following message:
Error writing Audit Log Msg: Security Msg INSERTER ERROR: SybaseESql::commit: Disk full 'Fatal error: disk full E:\PROGRA~1\CSCOpx\MDC\Sybase\DB\IDS\idsmdc.log' -- transaction rolled back
So, if you run into the disk space issue, to resolve the problem free up some disk space on drive E:\.
Security Monitor Takes a Long Time to Launch
The Security Monitor database is the file install-dir\CSCOpx\MDC\Sybase\Db\IDS\idsmdc.db. The idsmdc.log file in the same directory is used to keep track of database changes. It is important to maintain these files so that they do not become too large or consume all available disk space. Refer to the section entitled "Database Maintenance Issues" in this chapter for additional details on how to maintain the Database.
Page Cannot Be Found Error While Trying to Launch Security Monitor
You need to analyze the SysEventlog.evt file if you receive the "Page cannot be found" error message when trying to bring up the Security Monitor. You may receive multiple errors in the file (one caused by another).
An example of such a problem is that the CiscoWorks Common Services Tomcat Servlet Engine service failed to start due to the following error:
The service did not start due to a logon failure.
The probable cause of the problem is that Common Services is installed by user "X". Then, user "X" was either deleted from the computer, or more likely, his password was changed.
To resolve the issue, go into Start > Settings > Control Panel > Administrative Tools > Services. Then double-click on CiscoWorks Common Services, click on the Log On tab, click the This Account radio button, and browse to the user account. Then provide the password for that account. Next, stop and restart the services.
IDS/IPS MC Launches But Security Monitor Does Not
During the installation or upgrade, if the modification of the XML files that help to launch Security Monitor is corrupted, you might run into this problem: IDS/IPS MC may be working but there might be issues with the Security Monitor. The best way to determine the problem to analyze the installation log files. The file responsible for launching the Security Monitor is ids-monitor.xml. If the file is indeed corrupted, copy this file from another Security Monitor server that is working.
Security Monitor Behaves Strangely
If your Security Monitor behaves very strangely in terms of launching, reporting and so on, you need to look primarily at the Tomcat stderr.log file that contains stack traces from the Java code. You need to send these stack traces to Cisco Support, so that the development team can analyze the traces to find the root cause of the problem. The stack traces are found at . . .\CSCOpx\MDC\tomcat\logs
The license issue for Security Monitor is the same as for IDS/IPS MC. Hence, for details on licensing issues with Security Monitor, refer to the "Licensing Issues" section in Chapter 18.
Device Management Issues
Depending on the types of monitoring devices, you can populate the device tables of Security Monitor by importing the devices from IDS/IPS MC, or you can add them manually if you do not want to import them from IDS/IPS MC. It is important to understand that devices that are imported from the IDS/IPS MC cannot be edited by the Security Monitor. Also, if these devices are deleted from the IDS/IPS MC, they will not automatically be deleted from the Security Monitor. For a monitoring device that uses Syslog, you do not have to add the device in the device table. However, if you want to see the name of the device instead of the IP address on the Event Viewer or the reports, you need to add the device. In this section, we will look into the configuration and troubleshooting aspects of adding sensors to the Security Monitor.
Importing IDS Sensors from IDS/IPS MC
Work through the steps that follow to import an IDS Sensor from IDS/IPS MC to the Security Monitor:
As mentioned before, monitored devices that are imported from the IDS/IPS MC are managed by the IDS/IPS MC. Their settings cannot be changed by the Security Monitor. There is a check box that is enabled by default in the IDS/IPS MC Add Device Page that will automatically add the device to the Security Monitor device table.
Adding Other Devices
Security Monitor can receive events from other hosts or devices such as PIX Firewall, IOS IPS, and Cisco Security Agents Management Console (CSA MC), and so on. The type of protocol used depends on the device type (IDS 3.x sensors use PostOffice, IDS 4.x uses RDEP, IPS 5.x uses SDEE, CSA MC uses RDEP, Router/PIX can use Syslog, and so on). The device table can contain different types of devices. Only sensor devices that are added to IDS/IPS MC can be imported; all other device types are added manually. Unlike IDS Sensor Import, sensors that are added manually may be deleted from the table.
Work through the steps that follow to add a device to the device table:
Follow these steps to configure the IOS device for logging:
IEV and Security Monitor Connect with Sensor
After importing or adding the sensor, go to Monitor > Connections to check the connection status. (Note that connection status is not displayed for devices that send events via Syslog.) Refer to Table 22-1 for connection status information and the reason for failure. If you see connection status other than "Connected" or "ConnectedTLS", then you know you have a problem with connectivity between the Security Monitor and the sensor. Use Table 22-1 as a reference to find the cause of the problem. For example, if your connection status shows "Indeterminate", as per Table 22-1, this means that the "IDS_Receiver process is not running". To resolve this issue, you might want to go to Server Configuration > Administration > Process Management > Process Status to check the status of the IDS_Receiver process on the VMS server. If the process has stopped, then restart it. If the process continues to stop automatically, then try the following:
Depending on the log, you might analyze the issue further and resolve it. Work through the steps that follow as a general troubleshooting technique to isolate and fix the problem that occurs when connecting to a sensor using the IEV or Security Monitor:
Notifications are performed by two daemonsIDS_Analyzer (for Event Rules) or IDS_DbAdminAnalyzer (for Database Rules) and IDS_Notifier. The Analyzer is the component that watches the incoming event, or database events, and triggers the notification. The Notifier is the component that builds and sends the notification. The notification can be sent to the console or by e-mail. If both of the Analyzer and Notifier daemons are running, then you need to determine if the Event Rules are indeed supposed to be triggered. Review the filter criteria of the event or database rules, and make necessary changes. To debug this issue further, collect the following information:
A more detailed case study on e-mail notification is analyzed in the "Case Studies" section of this chapter.
Event Viewer Issues
Event Viewer is used to view the events in almost real time. This window allows you to retrieve the events from the database based on the criteria you define.
Launching the Event Viewer
Follow these steps to launch the Event Viewer from the Security Monitor:
Event Viewer has the following two limitations:
Using the Event Viewer
Column headers and cell data depend on the type of data being displayed. You can drag columns and expand and collapse columns on the Event Viewer within the limitations of the user's Event Viewer preferences. When you drag a column or click a table of contents item, the system will do a round trip; this takes a variable amount of time (depending on the number of records in the database and the number of events being displayed). The column settings are reset by the launching of the Event Viewer in the version earlier than 2.0, but with version 2.0 and above, the settings are retained.
Generating Events for Test
For testing, events may be generated several ways. A simple form of event generation process is to generate the large Internet Control Message Protocol (ICMP) packet to a host that belongs to a VLAN where the sniffing port of the Sensor is connected. To generate a Large ICMP event, open a command prompt on the server and type: C:\> ping 10.1.1.2 t l 5000
Large ICMP events show up when the Event Viewer is refreshed.
You need to ensure that the Large ICMP event is configured with a severity high enough to be sent. You can verify this either by CLI on the sensor or on the IDS/IPS MC. The signature severity setting must be greater than the minimum level configured to be sent in the IDS/IPS MC.
What you see in the Event Viewer depends on several factors.
You can expand columns and refresh events to update the view. One important point to note is that the event viewer Table of Contents is not used for navigation. New events are not necessarily displayed immediately.
This section analyzes the issues that you might experience with the Security Monitor Event Viewer.
Cannot Load Security Event Viewer
If the Security Monitor Event viewer is not coming up, look at the IDS_EvsServer.log file. You will see a message such as: ERROR: NrMainWidget::createDaemonMgrListener, daemon manager reported errors. This is usually caused by a problem with the Java plug-in. Follow these steps:
No Events on Event Viewer from IDS/IPS Sensor Are Visible
If the Event Viewer on the Security Monitor does not show any events, then the problem can be either on the sensor or on the Security Monitor or even on both. Work through the steps that follow to troubleshoot the issue:
When the sensor is configured to generate events for lower severity and the Security Monitor is configured to pull lower severity events, the number of events sent to the VMS server increases dramatically. This could cause database issues if you do not perform regular database maintenance.
No Syslog Events from PIX Firewall Or IOS Router in Security Monitor Event Viewer Are Visible
Problems with being unable to see events on the Security Monitor Event Viewer are similar to those discussed in the previous section on the IDS sensor's events issues. If you cannot see any events on the Security Monitor Event Viewer from the PIX firewall or IOS router, the problem might be either on the Monitoring device or on the Security Monitor itself. Work through the steps that follow to troubleshoot this issue:
Report Generation Issues
You can generate a static report based on different criteria and send out the report as e-mail in different formats. More details about generating and managing reports can be found at the following location:
If you find any issues with reporting, analyze the log file <cw-install>\log\IDS_ReportScheduler.log. This section examines the two primary problems that can occur with report generation:
Report Generation Fails
If report generation fails, analyze the log IDS_ReportScheduler.log file. Work through the following steps to troubleshoot report generation failure issues:
If the Security Monitor does not fail immediately but hangs instead, then follow the directions in the next section.
Report Fails to Complete
If you have a huge database full of events, report generation may take hours, as the Security Monitor may take hours to query and pull the events from the database. If so, the report generation process may seem to be hung. If you analyze the file <cw-install>\log\IDS_ReportScheduler.log, you may see messages similar to those shown in Example 22-3.
Example 22-3. Error When Report Does not Completes
The solution to this problem is to re-adjust the default pruning value so that the alarms are deleted at a lower value (see the "Database Maintenance Issues" section, which is discussed next in this chapter).
Database Maintenance Issues
The Security Monitor receives data from different devices and stores the data in records in database tables for viewing and creating reports. The Security Monitor is designed to capture and store data flowing into its receiver process at high rates (on the order of 50 IDS/ IPS events per second) for a sustained period or at even higher rates (up to 500 IDS/IPS events per second) for a short burst period of up to five minutes. Storing large amounts of data has its drawbacks, however, as both disk space requirements and query time increase with the amount of data stored.
Care must be taken to ensure that the Security Monitor's database environment is maintained. Devices sending information to the Security Monitor must be configured properly so that they do not send unwanted messages at the Security Monitor receiver and overload it with data. Be aware of these requirements and monitor the following:
The Security Monitor comes with several tools to help you keep the database size manageable. In particular, three default database rules are installed with the Security Monitor for maintaining the size of three database tables. These default rules are configured to:
The sections that follow explain how to use the tools efficiently to maintain the database for the healthy operation of IDS/IPS MC and Security Monitor.
Proactive Measures Immediately After Installing the Security Monitor
It is extremely important to use the following proactive measures to avoid issues with the database:
Redirect Archive Files Away from the Database Disk
When the database is pruned, by default it creates an archive of the pruned data in a flat file that is stored in ~\CSCOpx\MDC\secmon\AlertPruneData. Monitor this directory very closely, because it can grow rapidly. Therefore, we recommend that you change the pruning directory to a network share, so that you do not have to worry about maintaining this directory. To change the directory, go to Admin -> System Configuration -> Prune Archive Location in Security Monitor version 2.0.
If you are running a version of Security Monitor that is earlier than 2.0, work through the steps that follow to redirect the archive files away from the database disk:
Redirect Backup Files Away from the Database Disk
A backup copy of the Sybase database is on the installation disk by default. Regularly scheduled backups can quickly consume a large amount of disk space, adversely affecting the performance of the installed management and monitoring centers. To prevent this problem, move the default location of the backups to a separate local disk or to a mounted network drive. Follow these steps for redirecting backup files away from the Database Disk:
Create a New Database Rule
In Security Monitor versions earlier than 2.0, when you delete data from the Event Viewer in Security Monitor, the records are marked for deletion. To remove these records from the database, create a new Database Rule that executes once daily to delete the records that are marked for deletion. Follow these steps to create a new database rule:
Reactive Measures During Run Time
At run time, you must periodically monitor the system for the following four items:
Flow Rates of Events and Syslog Messages
The receiver process on Security Monitor is responsible for receiving the events from various devices. You must not exceed the limit of the receiver capacity. Flow rates for the various devices are measured continually by the Security Monitor and are available for viewing in the 24 Hour Metrics report. To run the 24 Hour Metrics report, follow these steps:
Each line indicates a five-minute period, and each row contains the timestamp for the time at which the measurement was logged. Pay particular attention to any measurement that is out of spec. (Security Monitor documentation specifies that it can handle a sustained rate of less than 50 IDS/IPS events per second45,000 in 15 minutes. Syslog events are not specified in the documentation, but a good rule of thumb is that the system performance will start to suffer when syslog events approach 25 messages per secondor about 22500 messages in 15 minutes). If you find measurements out of spec, you should determine the cause and make necessary changes on the configuration of the monitoring devices, so that the events or the syslog messages are reduced.
Monitoring the Size of Log Files
Log files are used by the Security Monitor for error messages and state information, and for temporary data storage. Because log files reside on the same disk as the database, it is necessary to monitor their size and periodically move them off the system or delete them to ensure that the database has enough space to operate. There are two types of log files: CMF log files and IDS/IPS log files. There is no GUI method available for monitoring IDS/IPS log file sizes, so users need to monitor their sizes by other means. IDS/IPS log files are located in the ~CSCOpx/log directory and all begin with the characters "IDS_". Any IDS/IPS log file larger than 50,000 bytes should be deleted or moved away from the database disk.
You can monitor the size of CMF log files in the GUI by following these steps:
This page displays a list of the CMF log files and their sizes, and the File System Utilization for the database disk. The sizes of any of the log files that have grown larger than their recommended maximum size are listed in the table in red. These files may need to be deleted or moved to another computer. Both CMF and IDS/IPS log files are in the ~CSCOpx/log directory. Follow these steps to move or delete files from this directory:
Monitoring Database File Size
You can monitor the size of the database by monitoring the disk utilization from the Server Configuration > Administration > Log File Status page. The File System Utilization column indicates the percentage of the database disk that is currently being used. The actual percentage of the disk that can be used without causing system degradation depends on the available resources. As a good rule of thumb, this number should not exceed 80 percent because the database will need extra space during operation to maintain its transaction log. The Security Monitor database is stored in two files in the ~CSCOpx\MDC\Sybase\Db\IDS subdirectory: idsmdc.db and idsmdc.log. You can find out the size of the database by analyzing the IDS_DbAdminAnalyzer.log as shown in Example 22-4.
Example 22-4. Determining Database Size
To maintain the database, use the pruning techniques that are discussed in the following section.
Under normal operation, when events are flowing into the Security Monitor and the default database rules are in place, the system will prune excess data (the oldest data) automatically. The time at which this occurs depends upon the number of events that have been received and the sizes of the syslog, IDS/IPS events, and audit log tables. Whenever the pruning utility runs, it logs a message to the audit log table at the start of the process and again when it is completed. The user should check the audit log for these pruning messages to be sure that data is being pruned from the database as expected. In particular, if the pruning process is started but not completed, users should be aware of a problem. Starting from Security Monitor 2.0, the database pruning is handled automatically by a database pruning daemon. By default, it will prune the database when the volume reaches 2,000,000 events. This default value can be changed by logging into Security Monitor and going to Admin > Data Management > Database > Pruning Configuration.
To troubleshoot issues pertaining to pruning, follow these steps to run the Pruning subsystem report:
The pruning report lists a message, including a time stamp for each time the pruning utility started and was completed. If the user believes that the database should have been pruned but the process did not run, here are a few things to check:
Pruning trims records from tables in the DB. It does nothing, however, to the actual DB files, which are managed by the DB server. The Sybase DB, used by IDS/IPS MC/SecMon, does not give back space to the file system when table records are deleted; instead it keeps them as available space within the current DB file. Hence the DB file will not shrink in size even though data was purged from it. The sizes of the database files might be reduced by running the dbcompact utility, which will shrink the database by the amount of free space created when records were deleted. Database files themselves are never pruned.
The trick is to set up the DB record pruning trigger points as desired by the customer. When those trigger points are reached, the record counts in the DB will essentially remain at those counts, as pruning deletes older records to make room for new ones. The size of the DB should not grow by much past this point. Once at that point, you can reset the file management DB size limit value to that based on your actual steady-state size. If the DB grows to this new limit, then the dbcompact utility can be run again.
It is important to note that the pruning, by going to IDS/IPS MC > admin > data mgmt > databases > pruning configuration, affects the record counts of tables in the DB, not the size of the DB file. The other screen, file management, is used so that you can be alerted to files that are growing beyond a preset limit, which might require you to act.
If the database grows too large, you must run dbcompact to reduce the size of the dataset. The program is run by typing IDSdbcompact. You can find instructions for running the utility at: http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/mon_sec/secmon12/ug/ch07.htm#1824
Starting from Security Monitor version 2.0, database rules are now only used to set up e-mail notifications. With 2.0 and subsequent releases, the Security Monitor's pruning algorithm is no longer triggered by a database rule. Pruning (and archiving) now runs continuously once the pruning threshold has been reached. When archiving is enabled, event data is written to the archive files and pruned from the database at the same rate and at the same time that it is being received.