Categorization of Problem Areas

The problem areas of the Security Monitor can be categorized as follows:

• Installation issues

• Unable to launch issues

• Licensing issues

• Device management issues

• Event viewer issues (real time)

• Report generation issues

• Notification Issues (e-mail notification)

• Database maintenance issues

Installation Issues

The Security Monitor can be installed through either of the two following scenarios:

• Security Monitor only If you have multiple monitoring devices that need to send events to the Security Monitor, it is recommended to install the Security Monitor on a different server than the IDS/IPS MC. Installing only the Security Monitor on a different server gives you better management and performance of the Security Monitor. If the Security Monitor database is corrupted, you do not need to uninstall and re-install the IDS/IPS MC. This is required if both IDS/IPS MC and Security Monitor are installed on the same server, because both use the same Sybase database). Note that you only need to uninstall and reinstall the Security Monitor without using the old database, and then import the devices from IDS/IPS MC.

• Both IDS/IPS MC and Security Monitor The advantage of installing the IDS/IPS MC and Security Monitor on the same system is the ease of management, as both of the components use the same database and share other files.

The installation procedures for different versions of the Security Monitor are listed in the following link and are not be discussed here.

http://www.cisco.com/en/US/products/sw/cscowork/ps3990/prod_installation_guides_list.html

You must install the Common Services first before proceeding with the Security Monitor installation. Core provides authentication and user management, and a host of other services. As mentioned before, when both Security Monitor and the IDS/IPS MC are installed on the same system, sensors configured to send their events to Security Monitor can be imported from the IDS/IPS MC. Otherwise, every sensor monitored by Security Monitor needs to be added manually. PostOffice devices must be in the device table to receive those events. Syslog devices do not need to be in the device table, but if they are, the device name will be displayed in the event viewer and reports.

Troubleshooting techniques related to the installation of Security Monitor are similar to those of IDS/IPS MC which is discussed in Chapter 18, "Troubleshooting IDM and IDS/IPS Management Console (IDS/IPS MC)" under the section entitled "Installation and upgrade issues." Therefore, the troubleshooting techniques are not discussed here.

Issues with Launching

To launch Security Monitor, first log in to Security Monitor server from your Web browser: http://localhost:1741.

Then, launch the Security Monitor by clicking VPN/Security Management Solution > Monitoring Center > Security Monitor.

When you launch, Security Monitor will land on the device table as shown in Figure 22-2.

Figure 22-2. Security Monitor Shown After the Login

Following is the list of tabs available for different purposes in the Security Monitor GUI:

• Devices Add, import, and edit devices.

• Configuration Configure Event Rules. This tab is new to Security Monitor 2.0. The Event Rule configuration is performed under the Admin tab on the version earlier than 2.0.

• Monitor Launch Event Viewer, and determine device status.

• Reports Generate, schedule, and view reports.

• Admin Configure database rules, system configuration, and event viewer preferences.

There might be several reasons for being unable to launch the Security Monitor or the CiscoWorks Common Services in general. The sections that follow explain some of these reasons.

DNS Issues

If you cannot launch CiscoWorks Common Services or the Security Monitor with the IP address, try the following:

• Enter the IP Address of the Security Monitor manually on the browser, without using its cache (clear the browser cache).

• Try connecting to the Security Monitor from a browser in a different machine and see if that helps.

  If both fail, add the Security Monitor server's hostname to the hosts file on your PC with the IP address, and browse to the Security Monitor via its hostname: http://hostname:1741

Issues with Enabling SSL

When SSL is enabled for CiscoWorks Common Services, its certificate must be selected for the Security Monitor. Otherwise you will get a Java error when launching the event viewer and the untrusted server cert chain error message will be displayed. A certificate is created when SSL is turned on in CiscoWorks Common Services. Since the Event Viewer uses the Java plug-in provided by the Cisco Works Common Services server, the Security Monitor server needs knowledge of this certificate before the Event viewer applet is launched.

Work through the following steps to enable SSL:

Step 1.	Go to Server Configuration > Administration > Security Management > Enable/Disable SSL.
Step 2.	In the new page, fill out the appropriate certificate information before clicking Submit. This certificate information may be presented to you when the Event Viewer Applet is launched.
Step 3.	Go to VPN and Security Management Solution > Administration > Configuration > Certificate.
Step 4.	In the next window, choose Common Services Certificate and click Finish.
Step 5.	Restart the CiscoWorks Common Services Daemon Manager service.

Note

If you are stopping and starting the daemon manager from the Process Management folder of the CiscoWorks Common Services GUI fails, you can perform this task from the service control applet or by using the net stop crmdmgtd and net start crmdmgtd commands in a command prompt window.

When you try to login for the first time, CiscoWorks Common Services shows a security alert for the CA root certificate. If this happens, view the certificate and install it before proceeding. If this warning does not appear, it probably means that you have already installed the certificate. You can proceed without installing the certificate by clicking Yes. There are no negative affects to this acceptance; however the warning message may be displayed again.

Getting Internal Server Error While Opening Security Monitor

If you receive an Internal Server Error message, first be sure that you have enough space on the drive where you have installed Security Monitor. For example, if VMS is installed at E:\PROGRA~1\CSCOpx, then you must have enough space on the E:\ drive.

MDCSupport Zip file does not show how much disk space is available for drive E:. However, SysEventLog.evt will contain messages that E:\is at or near capacity. On the IDS_Receiver.log, you might see the following message:

Error writing Audit Log Msg: Security Msg INSERTER ERROR: SybaseESql::commit: Disk full 'Fatal error: disk full E:\PROGRA~1\CSCOpx\MDC\Sybase\DB\IDS\idsmdc.log' -- transaction rolled back 

So, if you run into the disk space issue, to resolve the problem free up some disk space on drive E:\.

Security Monitor Takes a Long Time to Launch

The Security Monitor database is the file install-dir\CSCOpx\MDC\Sybase\Db\IDS\idsmdc.db. The idsmdc.log file in the same directory is used to keep track of database changes. It is important to maintain these files so that they do not become too large or consume all available disk space. Refer to the section entitled "Database Maintenance Issues" in this chapter for additional details on how to maintain the Database.

Page Cannot Be Found Error While Trying to Launch Security Monitor

You need to analyze the SysEventlog.evt file if you receive the "Page cannot be found" error message when trying to bring up the Security Monitor. You may receive multiple errors in the file (one caused by another).

An example of such a problem is that the CiscoWorks Common Services Tomcat Servlet Engine service failed to start due to the following error:

The service did not start due to a logon failure. 

The probable cause of the problem is that Common Services is installed by user "X". Then, user "X" was either deleted from the computer, or more likely, his password was changed.

To resolve the issue, go into Start > Settings > Control Panel > Administrative Tools > Services. Then double-click on CiscoWorks Common Services, click on the Log On tab, click the This Account radio button, and browse to the user account. Then provide the password for that account. Next, stop and restart the services.

IDS/IPS MC Launches But Security Monitor Does Not

During the installation or upgrade, if the modification of the XML files that help to launch Security Monitor is corrupted, you might run into this problem: IDS/IPS MC may be working but there might be issues with the Security Monitor. The best way to determine the problem to analyze the installation log files. The file responsible for launching the Security Monitor is ids-monitor.xml. If the file is indeed corrupted, copy this file from another Security Monitor server that is working.

Security Monitor Behaves Strangely

If your Security Monitor behaves very strangely in terms of launching, reporting and so on, you need to look primarily at the Tomcat stderr.log file that contains stack traces from the Java code. You need to send these stack traces to Cisco Support, so that the development team can analyze the traces to find the root cause of the problem. The stack traces are found at . . .\CSCOpx\MDC\tomcat\logs

Licensing Issues

The license issue for Security Monitor is the same as for IDS/IPS MC. Hence, for details on licensing issues with Security Monitor, refer to the "Licensing Issues" section in Chapter 18.

Device Management Issues

Depending on the types of monitoring devices, you can populate the device tables of Security Monitor by importing the devices from IDS/IPS MC, or you can add them manually if you do not want to import them from IDS/IPS MC. It is important to understand that devices that are imported from the IDS/IPS MC cannot be edited by the Security Monitor. Also, if these devices are deleted from the IDS/IPS MC, they will not automatically be deleted from the Security Monitor. For a monitoring device that uses Syslog, you do not have to add the device in the device table. However, if you want to see the name of the device instead of the IP address on the Event Viewer or the reports, you need to add the device. In this section, we will look into the configuration and troubleshooting aspects of adding sensors to the Security Monitor.

Importing IDS Sensors from IDS/IPS MC

Work through the steps that follow to import an IDS Sensor from IDS/IPS MC to the Security Monitor:

Step 1.	Click the Import button under the Device Table.
Step 2.	Select the sensors you want to import from the MC.
Step 3.	Click the Import Sensors button
Step 4.	Now the Monitored Device table will show the imported sensors.

As mentioned before, monitored devices that are imported from the IDS/IPS MC are managed by the IDS/IPS MC. Their settings cannot be changed by the Security Monitor. There is a check box that is enabled by default in the IDS/IPS MC Add Device Page that will automatically add the device to the Security Monitor device table.

Adding Other Devices

Security Monitor can receive events from other hosts or devices such as PIX Firewall, IOS IPS, and Cisco Security Agents Management Console (CSA MC), and so on. The type of protocol used depends on the device type (IDS 3.x sensors use PostOffice, IDS 4.x uses RDEP, IPS 5.x uses SDEE, CSA MC uses RDEP, Router/PIX can use Syslog, and so on). The device table can contain different types of devices. Only sensor devices that are added to IDS/IPS MC can be imported; all other device types are added manually. Unlike IDS Sensor Import, sensors that are added manually may be deleted from the table.

Work through the steps that follow to add a device to the device table:

Step 1.	Click the Add button under the Device table.
Step 2.	Enter in the IP Address of your device.
Step 3.	Select the device type.
Step 4.	Click Apply when ready; you should see a new entry in the table.
Step 5.	If the IOS device uses PostOffice, check the Uses Postoffice Settings check box and enter the appropriate PostOffice data; otherwise, if the IOS device logs syslog, Uses Postoffice should be left unchecked.
Step 6.	The NAT address (if filled in) refers to the address of the device from the Security Monitor Server's perspective.

Follow these steps to configure the IOS device for logging:

Step 1.	Use Telnet to access the IOS router.
Step 2.	In configuration mode, enter the commands shown in Example 22-1. Example 22-1. Configuration for Forwarding Syslog to the Security Monitor c2600# c2600#configure terminal c2600(config)#logging on c2600(config)#logging host 10.1.1.40 c2600(config)#exit c2600#write memory Building configuration... [OK] c2600#
Step 3.	Save your configuration and exit the Telnet window.

IEV and Security Monitor Connect with Sensor

After importing or adding the sensor, go to Monitor > Connections to check the connection status. (Note that connection status is not displayed for devices that send events via Syslog.) Refer to Table 22-1 for connection status information and the reason for failure. If you see connection status other than "Connected" or "ConnectedTLS", then you know you have a problem with connectivity between the Security Monitor and the sensor. Use Table 22-1 as a reference to find the cause of the problem. For example, if your connection status shows "Indeterminate", as per Table 22-1, this means that the "IDS_Receiver process is not running". To resolve this issue, you might want to go to Server Configuration > Administration > Process Management > Process Status to check the status of the IDS_Receiver process on the VMS server. If the process has stopped, then restart it. If the process continues to stop automatically, then try the following:

• Check the install-dir/CSCOpx/log/IDS_Receiver.log file.

• Run an audit log report that shows the IDS_Receiver subsystem messages.

Depending on the log, you might analyze the issue further and resolve it. Work through the steps that follow as a general troubleshooting technique to isolate and fix the problem that occurs when connecting to a sensor using the IEV or Security Monitor:

Step 1.	Ping to sensor's IP address fails. If the ping to the sensor fails, check the network configuration. Run the setup command on the sensor to ensure that the sensor's IP address and gateway are set correctly. Also be sure that no router, switch or firewall is configured to interface with the sensor that may be blocking the traffic. If the network configuration is correct, verify that the sensor does not have an IP address conflict with another host on the network. Linux automatically prevents the command-and-control Ethernet port from activating if it detects an address conflict with another host. To check this, run the show interfaces command from the CLI. The output should say command-control interface is up. If the output says command-control interface is down, then there is a hardware issue, a cabling issue, or an IP address conflict.
Step 2.	Ping succeeds but SSH fails to connect or the connection is refused. In IDS 4.x, if ping succeeds but SSH fails, be sure the sensor's access list is configured to accept the user's address. This will have to be done from the CLI (run setup). If the access list is correct, be sure the sensor's SSH, Telnet, or web server ports are open in the firewall. Unlike IDS version 4.x, in IPS version 5.x and above if the sensor's ACL does not permit the host you are pinging from, the ping will fail. When you run setup, be sure that you include the IP address of the management station from where you are pinging the sensor. Also, be sure that the certificate Security Monitor you are using is valid (see Chapter 18 under the section entitled "Verifying If the IDS/IPS MC (Apache) Certificate Is Valid").If the certificate is not valid, follow the procedure "Regenerating IDS/IPS MC (Apache) Certificate" as explained in Chapter 18.
Step 3.	Sensor can be accessed via SSH but not via the Web browser. If the sensor can be accessed through SSH, verify that you are accessing the correct port on the sensor (that is, http rather than https). This can be verified by logging into the CLI and displaying settings for the web server service. To verify the https connection, you can try to access Event Server directly through a Web browser through the following URL: http(s)://<sensor's IP address>/cgi-bin/event-server
Step 4.	Access to the port and IP is right, but I am still being refused. If you are correctly addressing the sensor, verify that the web server is still running by using the show version CLI command. If the web server is no longer running, look for a bug on Cisco.com for the specific version you are running. If an upgrade is available for the sensor, proceed with the upgrade. If the issue persists, run show tech-support and get the output file to Cisco Support. Restart the sensor.
Step 5.	The web server process is running, but I am still unable to connect. If the CLI indicates that the web server is still running, verify that the firewall has an open port for the sensor. Be sure that the firewall device between the port and the server allows SSL (TCP/443) and is open. If there is an NAT device, be sure to use the translated Sensor IP.
Step 6.	I can connect and am getting a login prompt but fail on authentication. Check to see if logins to the account have been disabled due to the failed login limit being reached. The sensor provides the configuration option to limit the number of consecutive failed login attempts. Once this limit is reached, the account becomes locked until it is administratively unlocked. This option is disabled by default. It can be enabled in the CLI. To determine if a failed login attempt limit is enabled, enter the authentication service configuration mode and show settings. If the attemptLimit is greater than zero, the failed login attempt limit is set to this value. Set the attemptLimit value to zero to disable this account locking feature. This feature is required to satisfy the government's Common Criteria for security devices. To check the failed login count, log into the service account, if possible. From the service account shell, run the command pam_tally. The output shows the number of failed login attempts for each account. To reset the count, run pam_tally--reset. This command resets the failed login counts.
Step 7.	If directly accessing the event server or IDM succeeds, Security Monitor should be able to connect to the sensor. If it is unable to do so, delete the sensor and re-add.
Step 8.	If removing and reading does not resolve the connectivity problem, then reboot the server, add/remove the sensor.
Step 9.	If the problem persists, compact the database in the database issues section.
Step 10.	If everything else fails, upgrade to the latest patch/version of Security Monitor to avoid any bug. As a last resort you may need to uninstall and reinstall the software.

Notification Issues

Notifications are performed by two daemonsIDS_Analyzer (for Event Rules) or IDS_DbAdminAnalyzer (for Database Rules) and IDS_Notifier. The Analyzer is the component that watches the incoming event, or database events, and triggers the notification. The Notifier is the component that builds and sends the notification. The notification can be sent to the console or by e-mail. If both of the Analyzer and Notifier daemons are running, then you need to determine if the Event Rules are indeed supposed to be triggered. Review the filter criteria of the event or database rules, and make necessary changes. To debug this issue further, collect the following information:

• Run "Subsystem Report" from the "Audit Log" report group and make the following choices: IDS_Analyzer (if problem is with event-related issues); IDS_DbAdminAnalyzer (if database related issues); and IDS_Notifier for the sub-system on the filter settings page of the report wizard. Also set the Time/Date filtering to cover the approximate time that you first noticed the problem.

• Look for messages in the corresponding log files of the daemons (for example, IDS_Analyzer.log and IDS_Notifier.log for events rules).

A more detailed case study on e-mail notification is analyzed in the "Case Studies" section of this chapter.

Event Viewer Issues

Event Viewer is used to view the events in almost real time. This window allows you to retrieve the events from the database based on the criteria you define.

Launching the Event Viewer

Follow these steps to launch the Event Viewer from the Security Monitor:

Step 1.	Click the Monitor tab and select the Event Viewer option.
Step 2.	Select the Filter options.
Step 3.	Click the Launch Event Viewer button as shown in Figure 22-3. Figure 22-3. Launch Event Viewer Screen for Security Monitor
Step 4.	After loading the applet and the data, the Event Viewer is displayed. This page represents the filter used to screen data that is displayed in the Event Viewer.

Event Viewer has the following two limitations:

• A user-configurable limitation on the maximum number of events to be shown in a grid (up to 32 million) with a default of 100,000. If there are more events than the limit, the oldest events are shown (this is done to allow the oldest data to be deleted from the grid).

• A hard-coded limitation exists on the number of rows to be displayed. This limitation depends on what data is being viewed (generally about 250000 rows).

Using the Event Viewer

Column headers and cell data depend on the type of data being displayed. You can drag columns and expand and collapse columns on the Event Viewer within the limitations of the user's Event Viewer preferences. When you drag a column or click a table of contents item, the system will do a round trip; this takes a variable amount of time (depending on the number of records in the database and the number of events being displayed). The column settings are reset by the launching of the Event Viewer in the version earlier than 2.0, but with version 2.0 and above, the settings are retained.

Generating Events for Test

For testing, events may be generated several ways. A simple form of event generation process is to generate the large Internet Control Message Protocol (ICMP) packet to a host that belongs to a VLAN where the sniffing port of the Sensor is connected. To generate a Large ICMP event, open a command prompt on the server and type: C:\> ping 10.1.1.2 t l 5000

Large ICMP events show up when the Event Viewer is refreshed.

Note

You need to ensure that the Large ICMP event is configured with a severity high enough to be sent. You can verify this either by CLI on the sensor or on the IDS/IPS MC. The signature severity setting must be greater than the minimum level configured to be sent in the IDS/IPS MC.

What you see in the Event Viewer depends on several factors.

• The launch filter

• The level of expansion

• Timing

• The viewing preferences set for the user

You can expand columns and refresh events to update the view. One important point to note is that the event viewer Table of Contents is not used for navigation. New events are not necessarily displayed immediately.

Troubleshooting Steps

This section analyzes the issues that you might experience with the Security Monitor Event Viewer.

Cannot Load Security Event Viewer

If the Security Monitor Event viewer is not coming up, look at the IDS_EvsServer.log file. You will see a message such as: ERROR: NrMainWidget::createDaemonMgrListener, daemon manager reported errors. This is usually caused by a problem with the Java plug-in. Follow these steps:

Step 1.	Verify that you are running a supported Web browser, and have the proper cookies, Javascript, or that Java is enabled, and verify that the client operating system is supported.
Step 2.	Remove older versions of Java. If this does not work, remove all versions of Java, and CiscoWorks Common Services will prompt you to download the version that it is compatible with.
Step 3.	Uninstall all "damaged" objects as reported in the: Tools > Internet Options > General Tab > Settings > View Objects panel.
Step 4.	Delete all the cookies, temporary Internet files, history, and so on from Internet Explorer (Tools > Internet Options > General Tab > Delete Files).
Step 5.	Shut down all open Web browsers.
Step 6.	Delete the Java jar cache (Start > Settings > Control Panel > Java Plug-In, click on the Cache tab and press the Clear button).
Step 7.	Be sure the proper Java plug-in is used. Start > Settings > Control Panel > Java Plug-In, click on the Advanced tab and be sure use Java plug-in default is selected.
Step 8.	Add the Security Monitor server's hostname to the hosts file with the IP address, and browse to the Security Monitor server via its hostname http://hostname:1741.

No Events on Event Viewer from IDS/IPS Sensor Are Visible

If the Event Viewer on the Security Monitor does not show any events, then the problem can be either on the sensor or on the Security Monitor or even on both. Work through the steps that follow to troubleshoot the issue:

Step 1.	Verify that Security Monitor shows the IDS/IPS Sensor connection status as Connected or ConnectedTLS.
Step 2.	Be sure that the sensor is sending the events to the Security Monitor (see the "Events Issues" section in Chapter 14, "Troubleshooting IDS/IPS Sensor").
Step 3.	Be sure to choose the proper filter when you bring up the Event Viewer. Otherwise, you may be blocking the alerts from being shown.
Step 4.	By default, Medium is the lowest severity level for alerts sent from a sensor to Security Monitor. This means that only Medium and High severity alerts are shown in the Event Viewer. So, if the sensor is not generating and sending any Medium or High Severity events, then Security Monitor will not see anything. This could be because none of the signatures are configured to report High or Medium severity. Turn signatures 2000 and 2004 to High severity and see if they make it to Security Monitor. If you want to see lower severity events, in Security Monitor, you can click the Devices tab. Then click the radio box next to the sensor name and click Edit. In this new window, there will be a drop-down box entitled Minimum event level. Set this to the appropriate value.

Note

When the sensor is configured to generate events for lower severity and the Security Monitor is configured to pull lower severity events, the number of events sent to the VMS server increases dramatically. This could cause database issues if you do not perform regular database maintenance.

No Syslog Events from PIX Firewall Or IOS Router in Security Monitor Event Viewer Are Visible

Problems with being unable to see events on the Security Monitor Event Viewer are similar to those discussed in the previous section on the IDS sensor's events issues. If you cannot see any events on the Security Monitor Event Viewer from the PIX firewall or IOS router, the problem might be either on the Monitoring device or on the Security Monitor itself. Work through the steps that follow to troubleshoot this issue:

Step 1.	Ensure that the PIX or IOS router is set up correctly to send syslog messages to Security Monitor (see Example 22-1 for syslog configuration on the IOS router).
Step 2.	On Security Monitor go to Admin > System Configuration > SYSLOG Settings to check that the setting matches the UDP port number that the device uses. You do not need to check the Forward Syslog Messages box unless you want to forward the events to another syslog server.
Step 3.	To ensure that the device name appears in the Event Viewer, add the device into Security Monitor from the Devices tab. When you start the Event Viewer, ensure that the Event Type is set to one of the syslog types. The PIX Security Summary should show you all syslog data received. To access the PIX Security Summary, in Security Monitor go to the tab Monitor, and select Events (the Event Viewer can be launched from there). Before launching the Event Viewer, select PIX Security Summaries in the drop-down menu for the Event Type field.
Step 4.	Ensure that the CWCS syslog service on the VMS server has started correctly. If this service fails to start correctly, check the size of the install-dir\Program Files\CSCOpx\log\syslog.log file. If this file is quite large, clear the file or move it to an alternate location, and then restart the service.

Report Generation Issues

You can generate a static report based on different criteria and send out the report as e-mail in different formats. More details about generating and managing reports can be found at the following location:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/mon_sec/secmon20/ug/ch06.htm

If you find any issues with reporting, analyze the log file <cw-install>\log\IDS_ReportScheduler.log. This section examines the two primary problems that can occur with report generation:

• Report generation fails

• Report fails to complete

Report Generation Fails

If report generation fails, analyze the log IDS_ReportScheduler.log file. Work through the following steps to troubleshoot report generation failure issues:

Step 1.	Be sure that the IDS_ReportScheduler process is running. Restart the process and re-run the report. If the process keeps dying and report generation keeps failing, apply the latest patch from the Cisco.com Web site.
Step 2.	If the IDS_ReportScheduler is running but you are still having issues with report generation, the problem might be with the TEMP directory. Find the location of the temp directory by going to Start > Run, then type cmd, and press Enter. Type Set and press Enter. You will see a list of all the variables in use, and one of them will be the TEMP variable which points to the temp folder. Another quick way to find the path to the TEMP directory is to type %TEMP% in the command prompt. The report is temporarily created in the TEMP directory.
Step 3.	Be sure the TEMP directory is not full. If it's full, then go to Start > Run > type %TEMP%. This will take you to the TEMP directory. Delete all the files in that directory and re-run the report.
Step 4.	If the system fails to allow access to the temporary (TEMP) directory that is used during the report generation process, you will see the messages similar to those shown in Example 22-2. Example 22-2. Error Message in IDS_ReportScheduler.log Variable Query: "SELECT COUNT(*) FROM tmpTableIdsTopSrcDestPairsAlarm" getQueryVariableValue: var value "1729811" Variable Query: "SELECT COUNT(*) FROM tmpTableIdsTopSrcDestPairsAlarm WHERE summary_count > '0'" getQueryVariableValue: var value "0" Variable Query: "SELECT SUM(summary_count) FROM tmpTableIdsTopSrcDestPairsAlarm" getQueryVariableValue: var value "0" File error: The system cannot find the path specified File error: The system cannot find the path specified getTimeSqlCondition: result is (event_time >= '1047136363531000000') SqlResultTable.setResults: converting direct to XML ResultSet err: null java.lang.NullPointerException
Step 5.	If you have restrictive permissions on the directory in which the TEMP files are created, the report generation will fail. So, be sure to provide the Read/Write permission to the TEMP directory for the user account that is used to run the CiscoWorks Common Services.

If the Security Monitor does not fail immediately but hangs instead, then follow the directions in the next section.

Report Fails to Complete

If you have a huge database full of events, report generation may take hours, as the Security Monitor may take hours to query and pull the events from the database. If so, the report generation process may seem to be hung. If you analyze the file <cw-install>\log\IDS_ReportScheduler.log, you may see messages similar to those shown in Example 22-3.

Example 22-3. Error When Report Does not Completes

!SqlResultTable.setResults: converting direct to XML Cannot access a pooled DB connection java.lang.Exception: Cannot access a pooled DB connection query: DECLARE LOCAL TEMPORARY TABLE tmpTableIdsSrcDestPairsAlarm (severity tinyint NULL, attacker_locality VARCHAR(17) NULL, victim_locality VARCHAR(17) NULL, sig_id unsigned int NULL, sub_sig_id unsigned int NULL, event_time unsigned bigint NULL, attacker_address unsigned int NULL, victim_address unsigned int NULL, orig_app_ip_addr unsigned int NULL, summary_count unsigned int NULL) ON COMMIT PRESERVE ROWS query: INSERT INTO tmpTableIdsSrcDestPairsAlarm SELECT severity, attacker_locality, victim_locality, sig_id, sub_sig_id, event_time, attacker_address, victim_address, orig_app_ip_addr, summary_count FROM Correlated_Alerts WHERE ((1 & flags) != 1) SqlResultTable.setResults: converting direct to XML SqlResultTable.setResults: done. ! There are the same log entries for another report being generated but then the entries ! stop. SqlResultTable.setResults: converting direct to XML IDS_ReportScheduler rx broadcast msg: jrm;IDS_Backup;3;0 IDS_ReportScheduler rx broadcast msg: jrm;IDS_DbAdminAnalyzer;3;0 IDS_ReportScheduler rx broadcast msg: jrm;IDS_DeployDaemon;3;0 ! It looks like while it was converting some data into the finished XML format !the process was stopped, possible by the daemon manager. !From IDS_DbAdminAnalyzer.log file contents, this Security Monitor has almost !2 million alarms. When the number of records approaches that level reports !can take a long time to run to completion. DbAdminAnalyzerApp.execute: done testing rules. Wed Mar 26 19:23:52 EST 2003 getDatabaseFileSize: result is 1907507200 getHddFreeSpace: result is 1,978,601,472 getTableRecordCount: ALERT_TABLE has 42427 records getTableRecordCount: SECURITY_MSGS has 1830148 records 

The solution to this problem is to re-adjust the default pruning value so that the alarms are deleted at a lower value (see the "Database Maintenance Issues" section, which is discussed next in this chapter).

Database Maintenance Issues

The Security Monitor receives data from different devices and stores the data in records in database tables for viewing and creating reports. The Security Monitor is designed to capture and store data flowing into its receiver process at high rates (on the order of 50 IDS/ IPS events per second) for a sustained period or at even higher rates (up to 500 IDS/IPS events per second) for a short burst period of up to five minutes. Storing large amounts of data has its drawbacks, however, as both disk space requirements and query time increase with the amount of data stored.

Care must be taken to ensure that the Security Monitor's database environment is maintained. Devices sending information to the Security Monitor must be configured properly so that they do not send unwanted messages at the Security Monitor receiver and overload it with data. Be aware of these requirements and monitor the following:

• Amount of data (number of records) stored in the tables.

• Amount of space left on the database disk.

• Rate at which data is flowing into the Security Monitor's receiver.

The Security Monitor comes with several tools to help you keep the database size manageable. In particular, three default database rules are installed with the Security Monitor for maintaining the size of three database tables. These default rules are configured to:

• Run when the size of a specific table exceeds a set level (for example, 2,000,000 syslog alarms).

• Prune table data out of the database.

• Write the pruned records data out to archive files.

The sections that follow explain how to use the tools efficiently to maintain the database for the healthy operation of IDS/IPS MC and Security Monitor.

Proactive Measures Immediately After Installing the Security Monitor

It is extremely important to use the following proactive measures to avoid issues with the database:

• Redirect archive files away from the database disk

• Redirect backup files away from the database disk

• Create a new database rule

Redirect Archive Files Away from the Database Disk

When the database is pruned, by default it creates an archive of the pruned data in a flat file that is stored in ~\CSCOpx\MDC\secmon\AlertPruneData. Monitor this directory very closely, because it can grow rapidly. Therefore, we recommend that you change the pruning directory to a network share, so that you do not have to worry about maintaining this directory. To change the directory, go to Admin -> System Configuration -> Prune Archive Location in Security Monitor version 2.0.

If you are running a version of Security Monitor that is earlier than 2.0, work through the steps that follow to redirect the archive files away from the database disk:

Step 1.	Click the Admin tab and select Database Rules from the sub-area bar; the Database Rules table displays.
Step 2.	Select the Default Pruning rule and click Edit.
Step 3.	Click Next to go to the Choose the Actions page.
Step 4.	The archive directory is specified in the Argumentslist under Execute a Script. After installation, the argument contents look something like this:-wc:\PROGRA~1\CSCOpx\MDC\Sybase\DB\IDS\AlertPruneData This string specifies the root directory location where the archived files are stored. To redirect the archive files to another location (for example, d:\ArchiveData), change the argument string to this:-wd:\ArchiveData. Note that after installation, the root directory always points to the same disk where the database files are located. You must redirect the files away from this disk. You can specify a mounted network drive to redirect the archive files to a different computer.
Step 5.	Click Finish.
Step 6.	Repeat steps 1-5 for the other two rules (Default Syslog Pruning and Default Audit Log Pruning).

Redirect Backup Files Away from the Database Disk

A backup copy of the Sybase database is on the installation disk by default. Regularly scheduled backups can quickly consume a large amount of disk space, adversely affecting the performance of the installed management and monitoring centers. To prevent this problem, move the default location of the backups to a separate local disk or to a mounted network drive. Follow these steps for redirecting backup files away from the Database Disk:

Step 1.	From the CiscoWorks Server Desktop, select VPN/Security Management Solution > Administration > Common Services > Preferences and the Administrative preferences page displays.
Step 2.	Type a new path in the Backup Directory field. You should point the path to another local disk or to a mounted network drive.
Step 3.	Click Apply and a confirmation dialog displays.
Step 4.	Click Yes to confirm the change, and then click OK to return to the Administrative Preferences page.

Create a New Database Rule

In Security Monitor versions earlier than 2.0, when you delete data from the Event Viewer in Security Monitor, the records are marked for deletion. To remove these records from the database, create a new Database Rule that executes once daily to delete the records that are marked for deletion. Follow these steps to create a new database rule:

Step 1.	Click the Admin tab and select the Database Rules sub-area bar item.
Step 2.	Click the Add button.
Step 3.	Name the rule, add a description, and check Daily Beginning; then enter the time that you want the rule to take effect and click Next.
Step 4.	On the Choose the Actions page, check Execute a Script, select PruneMarkedForDeletion.pl from the drop-down list, and click Finish.

Reactive Measures During Run Time

At run time, you must periodically monitor the system for the following four items:

Flow Rates of Events and Syslog Messages

The receiver process on Security Monitor is responsible for receiving the events from various devices. You must not exceed the limit of the receiver capacity. Flow rates for the various devices are measured continually by the Security Monitor and are available for viewing in the 24 Hour Metrics report. To run the 24 Hour Metrics report, follow these steps:

Step 1.	Click the Reports tab and select Generate from the sub-area bar.
Step 2.	Select IDS/IPS Alarms from the Report Group drop-down box.
Step 3.	Click to the second page, choose the 24 Hour Metrics report radio button, and click Select.
Step 4.	Click Finish to run the report immediately.
Step 5.	After the notification displays, click OK and select IDS/IPS Alarms from the Report Group drop-down box. The report displays in the list (you may need to refresh the page a few times).
Step 6.	To see the report, select it from the list and click View. The 24 Hour Metrics report lists two measurements for each type of event: - The number that was received in the last 15-minute period - A running total

Each line indicates a five-minute period, and each row contains the timestamp for the time at which the measurement was logged. Pay particular attention to any measurement that is out of spec. (Security Monitor documentation specifies that it can handle a sustained rate of less than 50 IDS/IPS events per second45,000 in 15 minutes. Syslog events are not specified in the documentation, but a good rule of thumb is that the system performance will start to suffer when syslog events approach 25 messages per secondor about 22500 messages in 15 minutes). If you find measurements out of spec, you should determine the cause and make necessary changes on the configuration of the monitoring devices, so that the events or the syslog messages are reduced.

Monitoring the Size of Log Files

Log files are used by the Security Monitor for error messages and state information, and for temporary data storage. Because log files reside on the same disk as the database, it is necessary to monitor their size and periodically move them off the system or delete them to ensure that the database has enough space to operate. There are two types of log files: CMF log files and IDS/IPS log files. There is no GUI method available for monitoring IDS/IPS log file sizes, so users need to monitor their sizes by other means. IDS/IPS log files are located in the ~CSCOpx/log directory and all begin with the characters "IDS_". Any IDS/IPS log file larger than 50,000 bytes should be deleted or moved away from the database disk.

You can monitor the size of CMF log files in the GUI by following these steps:

• Select the Server Configuration drawer from the CiscoWorks browser.
  
  
• Open the Administration folder and select Log File Status.
  
  

This page displays a list of the CMF log files and their sizes, and the File System Utilization for the database disk. The sizes of any of the log files that have grown larger than their recommended maximum size are listed in the table in red. These files may need to be deleted or moved to another computer. Both CMF and IDS/IPS log files are in the ~CSCOpx/log directory. Follow these steps to move or delete files from this directory:

Step 1.	Close all browser windows that are accessing CiscoWorks.
Step 2.	Stop the CiscoWorks Daemon Manager service on the Security. Monitor the server (either through the Windows Service panel or by using net stop from the command line). This will stop several associated processes and may take a few minutes.
Step 3.	If you are deleting or moving Syslog.log, you must also stop the CWCS syslog service.
Step 4.	Copy the files you want to move to another drive. (You may specify a mapped network drive to move them to another computer.)
Step 5.	Delete the log files.
Step 6.	If you stopped it, restart the CWCS syslog service. (You can use the services panel or "net start" from the command line.)
Step 7.	Restart the CiscoWorks Daemon Manager. It might take a few minutes.

Monitoring Database File Size

You can monitor the size of the database by monitoring the disk utilization from the Server Configuration > Administration > Log File Status page. The File System Utilization column indicates the percentage of the database disk that is currently being used. The actual percentage of the disk that can be used without causing system degradation depends on the available resources. As a good rule of thumb, this number should not exceed 80 percent because the database will need extra space during operation to maintain its transaction log. The Security Monitor database is stored in two files in the ~CSCOpx\MDC\Sybase\Db\IDS subdirectory: idsmdc.db and idsmdc.log. You can find out the size of the database by analyzing the IDS_DbAdminAnalyzer.log as shown in Example 22-4.

Example 22-4. Determining Database Size

*-- DatabaseRule ------------------------- ID:            2 Name:          Default Syslog Pruning Desc:          Default pruning for syslog table. Available:     0 Used:          0 IDS Count:     0 Syslog count:  2000000 Total count:   0 Exec Daily At: 2004-08-08 07:34:00.0 Is Exec Daily: false ------------------------------------------ isSyslogRecordCountTrigger: rule is asserted insertNotification: inserted Notification record ID 199 DbAdminAnalyzerApp.execute: done testing rules. Wed Mar 26 19:23:52 EST 2005 ! The following two lines show the database and disk space used getDatabaseFileSize: result is 1907507200 getHddFreeSpace: result is 1,978,601,472 getTableRecordCount: ALERT_TABLE has 42427 records getTableRecordCount: SECURITY_MSGS has 1830148 records 

To maintain the database, use the pruning techniques that are discussed in the following section.

Pruning

Under normal operation, when events are flowing into the Security Monitor and the default database rules are in place, the system will prune excess data (the oldest data) automatically. The time at which this occurs depends upon the number of events that have been received and the sizes of the syslog, IDS/IPS events, and audit log tables. Whenever the pruning utility runs, it logs a message to the audit log table at the start of the process and again when it is completed. The user should check the audit log for these pruning messages to be sure that data is being pruned from the database as expected. In particular, if the pruning process is started but not completed, users should be aware of a problem. Starting from Security Monitor 2.0, the database pruning is handled automatically by a database pruning daemon. By default, it will prune the database when the volume reaches 2,000,000 events. This default value can be changed by logging into Security Monitor and going to Admin > Data Management > Database > Pruning Configuration.

To troubleshoot issues pertaining to pruning, follow these steps to run the Pruning subsystem report:

Step 1.	Click the Reports tab and select Generate from the sub-area bar.
Step 2.	Select Audit Log from the Report Group box.
Step 3.	Choose the Subsystem report and click Select.
Step 4.	Click Select All next to Event Severity, choose IDS_Database Prune from the Subsystem selection box, and click Next.
Step 5.	Click Finish on the next page to run the report immediately.
Step 6.	Select Audit Log from the Report Group and the report displays in the list (you may need to refresh the page a few times).
Step 7.	Select the report and click View.

The pruning report lists a message, including a time stamp for each time the pruning utility started and was completed. If the user believes that the database should have been pruned but the process did not run, here are a few things to check:

• Is the IDS_DbAdminAnalyzer process running? IDS_DbAdminAnalyzer process determines that the trigger condition has been met. If it is not running, the rule will not take effect. (You can stop and restart processes from the Server Configuration drawer, Process Management folder.)

• Is the IDS_Notifier process running? The IDS_Notifier runs the specified script which attempts to run the pruning utility; if it is not running the script will not be executed. (You can stop and restart processes from the Server Configuration drawer, Process Management folder.)

• Are any other instances of pruning running? Only one instance of the pruning utility can run at a time, so if someone used the utility from the command line to perform another task (for example, archiving data) the rule would attempt to take effect but the pruning would not run. Be sure that no other instances of the pruning utility are running.

Pruning trims records from tables in the DB. It does nothing, however, to the actual DB files, which are managed by the DB server. The Sybase DB, used by IDS/IPS MC/SecMon, does not give back space to the file system when table records are deleted; instead it keeps them as available space within the current DB file. Hence the DB file will not shrink in size even though data was purged from it. The sizes of the database files might be reduced by running the dbcompact utility, which will shrink the database by the amount of free space created when records were deleted. Database files themselves are never pruned.

The trick is to set up the DB record pruning trigger points as desired by the customer. When those trigger points are reached, the record counts in the DB will essentially remain at those counts, as pruning deletes older records to make room for new ones. The size of the DB should not grow by much past this point. Once at that point, you can reset the file management DB size limit value to that based on your actual steady-state size. If the DB grows to this new limit, then the dbcompact utility can be run again.

It is important to note that the pruning, by going to IDS/IPS MC > admin > data mgmt > databases > pruning configuration, affects the record counts of tables in the DB, not the size of the DB file. The other screen, file management, is used so that you can be alerted to files that are growing beyond a preset limit, which might require you to act.

DB Compact

If the database grows too large, you must run dbcompact to reduce the size of the dataset. The program is run by typing IDSdbcompact. You can find instructions for running the utility at: http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/mon_sec/secmon12/ug/ch07.htm#1824

Database Rules

Starting from Security Monitor version 2.0, database rules are now only used to set up e-mail notifications. With 2.0 and subsequent releases, the Security Monitor's pruning algorithm is no longer triggered by a database rule. Pruning (and archiving) now runs continuously once the pruning threshold has been reached. When archiving is enabled, event data is written to the archive files and pruned from the database at the same rate and at the same time that it is being received.