Analysis of Problem Areas

After you install and initialize (with the setup command on the sensor) the sensor, the preliminary configuration of the sensor can be performed with IDM. By default the IDM is installed and turned on in the sensor. So, to access the sensor using IDM, you need only to open a supported version of the browser and point to the sensor's IP address as https://sensor_ip. As mentioned before, IDM can be used either as a primary management tool for the sensor (it can manage only one sensor) or for preliminary configuration, so that IDS/IPS MC can import the sensor configuration. Because IDM code runs on the sensor itself, most of the problems involve the IDM being unable to launch the IDM using a Web browser. This section explores the possible causes of not being able to launch IDM and goes into the details of IDS/IPS MC troubleshooting by covering the following topics:

• Important procedures and techniques

• Inability to access sensor using IDM

• IDS/IPS MC installation and upgrade issues

• IDS/IPS MC licensing issues

• Issues with importing sensors with IDS/IPS MC

• Signature or service pack upgrade issues with IDS/IPS MC

• Configuration deployment issues with IDS/IPS MC

• Database maintenance (pruning) issues with IDS/IPS MC

Important Procedures and Techniques

This section discusses some of the important procedures and techniques, which are essential for troubleshooting issues with importing a sensor, upgrading a signature or service packs, or deploying a configuration.

Verifying Allowed Hosts on the Sensor

Before you attempt to import the configuration from the sensor into IDS/IPS MC, your IDS/IPS MC must be one of the allowed hosts on the sensor. You can configure trusted hosts in two ways. Before you attempt to add the IDS/IPS MC network or host as a trusted host on the sensor, check to see if it is in the trusted list as shown in Example 18-1.

Example 18-1. Network Lists that Are in the Sensor

sensor# show configuration | include access accessList ipAddress 171.0.0.0 netmask 255.0.0.0 accessList ipAddress 172.16.0.0 netmask 255.255.0.0 accessList ipAddress 10.0.0.0 netmask 255.0.0.0 accessList ipAddress 20.1.1.0 netmask 255.255.255.0 sensor# 

Another way to verify allowed hosts (access-list) configuration is shown in Example 18-2.

Example 18-2. Network List Configured on the Sensor

sensor# configure terminal sensor(config)# service host !The following line is allowed only on version IDS 4.x sensor(config-Host)# networkParams !If you are running IPS 5.0 the above line should be replaced by the line below. The show !settings output is from the IPS 5.0 sensor(config-Host)# network-settings sensor(config-Host-net)# show settings    network-settings    -----------------------------------------------       host-ip: 172.16.171.13/26,172.16.171.1 default: 10.1.9.201/24,10.1.9.1       host-name: sensor13 default: sensor       telnet-option: enabled default: disabled !The following are the lists of network that are allowed to the sensor       access-list (min: 0, max: 512, current: 3)       -----------------------------------------------          network-address: 10.0.0.0/8          -----------------------------------------------          network-address: 171.0.0.0/8          -----------------------------------------------          network-address: 172.0.0.0/8          -----------------------------------------------       -----------------------------------------------       ftp-timeout: 300 seconds <defaulted>       login-banner-text:  <defaulted>    ----------------------------------------------- sensor(config-hos-net)# 

If the desired host or network is not listed in the allowed host table, add in the sensor with the procedure described in the next sections.

Adding Allowed Hosts on the Sensor

The following are two ways to add trusted hosts on the sensor:

• Add an allowed host by running the setup command on the sensor.

• Add an allowed host manually on the sensor.

Adding an Allowed Host By Running setup Command on a Sensor

You can add the trusted hosts by running the setup command on the sensor as shown in Example 18-3.

Example 18-3. Adding a Trusted Host Using the setup Command

sensor# setup    --- System Configuration Dialog --- At any point you may enter a question mark '?' for help. User ctrl-c to abort configuration dialog at any prompt. Default settings are in square brackets '[]'. Current Configuration: networkParams ipAddress 172.16.171.14 netmask 255.255.255.192 defaultGateway 172.16.171.1 hostname sensor14 telnetOption enabled !These are the hosts allowed in the current configuration accessList ipAddress 171.0.0.0 netmask 255.0.0.0 accessList ipAddress 171.69.0.0 netmask 255.255.0.0 accessList ipAddress 172.16.0.0 netmask 255.255.0.0 exit timeParams offset -480 standardTimeZoneName PDT summerTimeParams active-selection recurringParams recurringParams summerTimeZoneName PDT exit exit exit service webServer general ports 443 exit exit Current time: Mon Nov 15 16:02:12 2004 Setup Configuration last modified: Mon Nov 15 14:21:22 2004 Continue with configuration dialog?[yes]: Enter host name[sensor14]: Enter IP address[172.16.171.14]: Enter netmask[255.255.255.192]: Enter default gateway[172.16.171.1]: Enter telnet-server status[enabled]: Enter web-server port[443]: !This is where you have the option to choose if you want to add additional allowed hosts !to the ACL of the sensor Modify current access list?[no]: yes Current access list entries:   [1] 171.0.0.0 255.0.0.0   [2] 171.69.0.0 255.255.0.0   [3] 172.16.0.0 255.255.0.0 Delete: 2 Delete: ! You are adding all hosts from 10.0.0.0 network to be able to connect to the sensor with ! the following line Permit: 10.0.0.0 255.0.0.0 !In the following line, you are leaving it blank and hit Enter which indicates that you !have no more hosts to add Permit: Modify system clock settings?[no]: The following configuration was entered. networkParams ipAddress 172.16.171.14 netmask 255.255.255.192 defaultGateway 172.16.171.1 hostname sensor14 telnetOption enabled accessList ipAddress 171.0.0.0 netmask 255.0.0.0 accessList ipAddress 172.16.0.0 netmask 255.255.0.0 accessList ipAddress 10.0.0.0 netmask 255.0.0.0 exit timeParams offset -480 standardTimeZoneName PDT summerTimeParams active-selection recurringParams recurringParams summerTimeZoneName PDT exit exit exit service webServer general ports 443 exit exit [0] Go to the command prompt without saving this config. [1] Return back to the setup without saving this config. [2] Save this configuration and exit setup. Enter your selection[2]: Configuration Saved. *16:04:23 PDT Mon Nov 15 2004 Modify system date and time?[no]: sensor# 

Adding an Allowed Host Manually on a Sensor

You can add the trusted hosts on the sensor using CLI. Example 18-4 shows how to add the trusted host.

Example 18-4. Adding a Trusted Host Manually

sensor# configure terminal sensor(config)# service Host sensor(config-Host)# networkParams sensor(config-Host-net)# accessList ipAddress 20.1.1.0 netmask 255.255.255.0 sensor(config-Host-net)# exit sensor(config-Host)# exit Apply Changes:?[yes]: sensor(config)# exit sensor# 

Note

Make sure that the assigned ipAddress correlates to the netmask. A good example of an ipAddress is 10.1.1.0 255.255.255.0, and a bad example is 10.1.1.1 255.255.255.0.

Verifying the SSH and SSL Connection Between IDS/IPS MC and a Sensor

From the MC, open a command shell and change to {INSTALL_DIR}/CSCOpx/MDC/bin/ids directory.

Try plink -ssh username@sensorIP.

It should prompt you for the sensor's password.

Enter the password.

It may prompt you to accept a changed or new SSH

fingerprint. Answer Yes.

You should now have a CLI prompt. Enter show version.

All the processes except CLI should report "Running". Exit.

Resolving SSH and SSL Connection Problems Between IDS/IPS MC and a Sensor

If IDS/IPS MC and the sensor are in two different networks, be sure that the network devices between IDS/IPS MC and the sensor are allowing SSH (TCP/22) and SSL (TCP/443) traffic in both directions. If both IDS/IPS MC and the sensor are in the same network, be sure that there is no Layer 2 firewall. If there is, open the ports (TCP/22, TCP/443) that are necessary for communication in both directions between the IDS/IPS MC and the sensor. Open a DOS prompt on the IDS/IPS MC, and try to connect to the sensor using plink -ssh username@ip_address_of_sensor, which will ensure that the communication is good between the IDS/IPS MC and the sensor.

Verifying If the Sensor Processes Are Running

To verify if all the processes are running on the sensor, use SSH or Telnet to access the sensor and execute show version. Example 18-5 shows the show version output on IDS/IPS sensor 4.x.

Example 18-5. Output of show version Command from IDS/IPS Sensor 4.x

sensor# show version Application Partition: Cisco Systems Intrusion Detection Sensor, Version 4.1(4)S145 OS Version 2.4.18-5smpbigphys Platform: IDS-4250 Sensor up-time is 9 days. Using 1123102720 out of 1980493824 bytes of available memory (56% usage) Using 3.6G out of 15G bytes of available disk space (26% usage) MainApp           2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running AnalysisEngine    2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running Authentication    2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running Logger            2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running NetworkAccess     2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running TransactionSource 2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running WebServer         2004_Apr_15_15.03 (Eng4e)   2004-09-09T15:08:54-0500 Running CLI               2004_Apr_15_15.03 (Release) 2004-04-15T15:11:59-0500 Upgrade History:   IDS-sig-4.1-4-S145.rpm.pkg 01:18:11 UTC Fri Feb 18 2005 Recovery Partition Version 1.2 - 4.1(1)S47 sensor# 

Example 18-6 shows the output of the show version command, which shows all the processes are running on IPS version 5.0.

Example 18-6. Output of show version Command from IDS/IPS Sensor 5.x

sensor# show version Application Partition: Cisco Intrusion Prevention System, Version 5.0(1)S149.0 OS Version 2.4.26-IDS-smp-bigphys Platform: IDS-4235 Serial Number: CGTWJ11 No license present Sensor up-time is 16 days. Using 780869632 out of 923549696 bytes of available memory (84% usage) system is using 17.3M out of 29.0M bytes of available disk space (59% usage) application-data is using 31.2M out of 174.7M bytes of available disk space (19% usage) boot is using 40.5M out of 75.9M bytes of available disk space (56% usage) application-log is using 530.5M out of 2.8G bytes of available disk space (20% usage) MainApp          2005_Mar_04_14.23  (Release)  2005-03-04T14:35:11-0600 Running AnalysisEngine   2005_Mar_04_14.23  (Release)  2005-03-04T14:35:11-0600 Running CLI              2005_Mar_04_14.23  (Release)  2005-03-04T14:35:11-0600 Upgrade History:   IDS-K9-maj-5.0-1- 22:16:00 UTC Thu Mar 04 2004 Recovery Partition Version 1.1 - 5.0(1)S149 sensor# 

Verifying That the Service Pack or Signature Level Sensor Is Running

After connecting to the sensor, you can execute show version command and look under the Upgrade History: to get the actual version information of the sensor. In Example 18-6, the sensor is shown to be running version 5.0(1) signature 149.

Verifying the Service Pack or Signature Level on IDS/IPS MC

If you are running IDS/IPS MC version 2.0 or above, you can find out the latest signature or service pack applied to the IDS/IPS MC by looking at the bottom of the first page of IDS/IPS MC. Before IDS/IPS MC version 2.0 or above, this information is not available. The only way to find the latest signature or service pack on the earlier version is by adding a dummy sensor on the IDS/IPS MC. Even on version 2.0 or above, though, you can find the latest version of the signature or service packs. To find the exact version of signature that the sensor is running (this may be a lower version than the latest version that IDS/IPS MC has), add a dummy sensor on the IDS/IPS MC. To determine the current version for IDS/IPS MC (in a version earlier than 2.0), work through the following steps:

Step 1.	Go to the Device page and try to add one sensor (it can also be a non-existing sensor). Do not choose the autodiscovery option.
Step 2.	In the second screen of the wizard, you will be asked to select the version of the sensor you want to add. The latest versions you see in this screen are the versions that are known to the IDS/IPS MC. Figure 18-1 shows how to find the version information with a dummy sensor. Figure 18-1. Checking the Version Information
Step 3.	You do not need to finish adding the sensor and can just click Cancel.

Note

The IDS/IPS MC upgrade is not cumulative as it is on the sensor. You need to apply all the signature upgrades to the server for the signature to be recognized. It is not possible to downgrade the IDS/IPS MC server.

Verifying That the IDS/IPS MC (Apache) Certificate Is Valid

For successful secured communication between IDS/IPS MC and the sensor, the Apache certificate must be valid on the IDS/IPS MC. You can verify this by using the following procedure:

Step 1.	Open a DOS command prompt.
Step 2.	Change the directory to ssl by executing the following command: C:\>cd Program Files\CSCOpx\MDC\Apache\conf\ssl
Step 3.	Type the command keytool printcert file server.cert. This command provides the Apache Certificate details, including when it will expire. Note that part of the VMS installation creates an Apache certificate that expires in one year. Example 18-7 shows a certificate that does not expire until March 17, 2009. Example 18-7. The Certificate Does Not Expire Until March 17, 2009 C:\cd Program Files\CSCOpx\MDC\Apache\conf\ssl>keytool printcert file server.cert Owner: CN=wk2 Issuer: CN=Cisco Security Agent Root CA Serial number: 4059c5a0 Valid from: Thu Mar 18 09:52:02 CST 2004 until: Tue Mar 17 09:52:02 CST 2009 Certificate fingerprints: MD5: 7E:02:00:F3:43:57:3B:B3:F0:16:C1:F6:B3:77:F2:9F SHA1:42:8A:6A:54:16:6E:7A:13:F6:82:F5:97:72:D6:D6:F5:DD:D1:E2:04 C:\cd Program Files\CSCOpx\MDC\Apache\conf\ssl>

Regenerating IDS/IPS MC (Apache) Certificate

If you have an expired certificated or corrupted certificate that is verified by the procedure explained in the preceding section, you can regenerate the Apache certificate using the following procedure:

Step 1.	Stop the CiscoWorks Daemon Manager.
Step 2.	Open a DOS command prompt.
Step 3.	Type the command cd install-dir/CSCOpx/MDC/Apache.
Step 4.	If you want to generate a certificate that has a validity period of longer than one year, edit the file gencert.bat and change the value 365 located at the end of the file to a larger value. For example, 3650 is about 10 years.
Step 5.	Type the command gencert. This command creates a new certificate.
Step 6.	Start the CiscoWorks Daemon Manager.

Resolving Issues with the IDS/IPS Sensor Being Unable to Get the Certificate

After you regenerate the certificate, it is important to refresh the trusted host. In addition, if the sensor is unable to get the certificate from IDS/IPS MC, work through the steps that follow to resolve this issue:

Step 1.	Remove the existing certificate of IDS/IPS MC by using the following commands: sensor# configure terminal sensor(config)# no tls trusted-host ip-address [ip address of vms server]
Step 2.	Add the IDS/IPS MC by using the following command: sensor(config)# tls trusted-host ip-address [ip address of vms server]
Step 3.	If the previous two steps do not resolve the issue, log in to the service account, and 'su -' to the root. The procedure for creating a "service" account is explained in the section entitled "Creating a Service Account" under "Manually Updating the Signature Level on a Sensor." Then log in to the sensor with the "service" account and change to login as root. Then execute the commands shown in Example 18-8. Example 18-8. Procedure for Removing the Certificate to Resolve the Trusted Host Issue -bash-2.05b$ su - Password: -bash-2.05b# /etc/init.d/cids stop Shutting down CIDS: Remove cidmodcap: Remove cidmodcap node: -bash-2.05b# cd /usr/cids/idsRoot/etc/cert -bash-2.05b# mv mytestca.cer mytestca.save -bash-2.05b# rm -f *.cer -bash-2.05b# mv mytestca.save mytestca.cer -bash-2.05b# cd .. -bash-2.05b# rm -f curTrustedCertificatesConfig.xml -bash-2.05b# /etc/init.d/cids start Checking kernel allocated memory: [ OK ] Load cidmodcap: Create node: Starting CIDS: -bash-2.05b#
Step 4.	After making the changes, reboot the sensor to be safe.
Step 5.	Once the sensor is backed up, log in with the "Admin" account and add a trusted host as explained in Step 2.

Changing the VMS Server IP Address

Work through the steps that follow if you have changed the IP address of the VMS Server:

Step 1.	Stop the service daemon manager.
Step 2.	Use Notepad to open install-dir\MDC\etc\ids\xml\SystemConfig.xml.
Step 3.	Change the <HostIP> value to the new IP address.
Step 4.	Save the modified file.
Step 5.	Copy the modified file to install-dir\MDC\Tomcat\vms\ids-config\web-inf\classes\com\cisco\nm\mdc\ids\common\SystemConfig.xml and install-dir\MDC\Tomcat\vms\ids-monitor\web-inf\classes\com\cisco\nm\mdc\ids\common\SystemConfig.xml.
Step 6.	Use Notepad to open install dir\PostOffice\etc\routes.
Step 7.	Change the host name and IP address to the new values.
Step 8.	Save the modified file.
Step 9.	Start the service daemon manager.

Manually Updating the Signature Level on the Sensor

In some rare circumstances, IDS/IPS MC upgrade may fail, so you might need to upgrade the signature or service packs of the sensor manually. To perform the manual upgrade, create a service account to transfer the upgrade file to the sensor from an FTP server, then run the upgrade command from the sensor itself.

Creating a Service Account

Work through the steps that follow to create the service account:

Step 1.	Log in to the sensor using an "admin" account, and the prompt will look like following: sensor#
Step 2.	Enter into the configuration with the following command: sensor# configure terminal
Step 3.	Create a user (this user name does not have to be service) with the service privilege by using the following command: sensor(config)# username service privilege service password xxx

Note

Only one service account can be configured on a sensor. The root user will have the same password as the service account.

Update Locally Over ftp/scp

Use the following technique if you cannot update the signature from the network (if it is failing). This technique downloads the update to the sensor and then upgrades from the local file system.

Step 1.	If you do not have one, create a service account (refer to the previous section for instructions on creating a service account).
Step 2.	Log in using the "service" account (and do not "su" to root). The prompt will look like: bash-2.05a$
Step 3.	Connect to the FTP server: bash-2.05a$ ftp ftp_server_address
Step 4.	Set up an FTP client to use a "binary" mode to get the file: ftp> bin
Step 5.	Check that the file is in the FTP server: ftp> ls 227 Entering Passive Mode . 125 Data connection already open; Transfer starting. -rwxrwxrwx 1 owner group 13280279 Aug 28 14:44 IDS-K9-min-4.1-1-S47.rpm.pkg -rwxrwxrwx 1 owner group 2061291 Aug 28 14:47 IDS-sig-4.0-2-S47.rpm.pkg -rwxrwxrwx 1 owner group 2120589 Oct 20 18:26 IDS-sig-4.1-1-S53.rpm.pkg -rwxrwxrwx 1 owner group 2124411 Oct 20 19:43 IDS-sig-4.1-1-S54.rpm.pkg 226 Transfer complete.
Step 6.	Retrieve the file (you may copy and paste the file name from the previous output): ftp> get upgrade_file
Step 7.	Close the FTP connection and quit from FTP client: ftp> close ftp> quit
Step 8.	Check that the file is there. You should see the upgrade file that you downloaded from the FTP server. bash-2.05a$ ls
Step 9.	Log out of the service account. bash-2.05a$ exit
Step 10.	"Copy" the file from the server itself over scp. This will ensure that no network settings could be affecting the upgrade.
Step 11.	Log in using the "cisco" account; the prompt will look like: sensor#
Step 12.	Enter configure terminal mode: sensor# configure terminal
Step 13.	Create the key: sensor(config)# ssh host-key [sensor_ip_address]
Step 14.	Type "yes" to accept the key.
Step 15.	Apply the upgrade as follows: sensor(config)# upgrade scp:// User: service_account_user_name Server's IP Address: sensor_ip_address Port[22]: File name: upgrade_file Password: ********** Warning: Executing this command will apply a signature update to the application partition. Continue with upgrade? : yes

Unable to Access the Sensor Using IDM

If you are unable to access the sensor to configure it using IDM (by Web browser) follow these steps to troubleshoot problem:

Step 1.	Ping to sensor's IP address fails. If the ping to the sensor fails, check the network configuration. Run the setup command on the sensor to be sure the sensor's IP address and gateway are set correctly. Also be sure there are no routers, switches, or firewalls configured to interface with the sensor that may be blocking the traffic. If the network configuration is correct, verify that the sensor does not have an IP address conflict with another host on the network. Linux automatically prevents the command and control Ethernet port from activating if it detects an address conflict with another host. To check this, run the show interfaces command from the command line interface (CLI). The output should be "command-control interface is up." If the output is "command-control interface is down," then there is a hardware issue, a cabling issue, or an IP address conflict.
Step 2.	Ping succeeds, but SSH fails to connect or connection is refused. Be sure the sensor's access list is configured to accept the user's address. This will have to be done from the CLI (run setup). If the access list is correct, be sure the sensor's SSH or Telnet and web server ports are open in the firewall.
Step 3.	Sensor can be accessed via SSH but not via the Web browser. If the sensor can be accessed through SSH, verify that you are accessing the correct port on the sensor (that is, http versus https). This can be verified by logging into the CLI and showing settings for the web server service.
Step 4.	Access to the right port and right IP is still being refused. If you are correctly addressing the sensor, verify that the web server is still running by using the show version CLI command. If the web server is no longer running, look for a bug on Cisco.com for the specific version you are running. If an upgrade is available for the sensor, proceed with the upgrade. If the issue persists, run show tech-support and send the output file to Cisco Support. Restart the sensor.
Step 5.	Web server process is running, but you are still unable to connect. If CLI indicates that the web server is still running, verify that the firewall has an open port for the sensor.
Step 6.	You can connect and are getting a login prompt but authentication fails. Check to see if logins to the account have been disabled due to the failed login limit being reached. The sensor provides the configuration option to limit the number of consecutive failed login attempts. Once this limit is reached, the account becomes locked until it is administratively unlocked. This option is disabled by default. It can be enabled in the CLI. To determine if a failed login attempt limit is enabled, enter the authentication service configuration mode and use the command show settings. If the attemptLimit is greater than zero, then the failed login attempt limit is set to this value. Set the attemptLimit value to zero to disable this account locking feature. This feature is required to satisfy the government's Common Criteria for security devices. To check the failed login count, log into the service accountif possible. From the service account shell, run the command pam_tally. The output shows the number of failed login attempts for each account. To reset the count, run pam_tally --reset. This command will reset the failed login counts.

IDS/IPS MC Installation and Upgrade Issues

IDS/IPS MC is installed as a component on top of Common Services. You must have the Common Services installed before you proceed with the IDS/IPS MC installation.

The installation procedure for the different versions of IDS/IPS MC are listed in the following link and are not discussed here.

http://www.cisco.com/en/US/products/sw/cscowork/ps3990/prod_installation_guides_list.html

Here are some issues that you may experience while installing the IDS/IPS MC:

• Installing IDS/IPS MC without administrator rights You must install IDS/IPS MC with a user who has administrator rights. We highly recommend that you install all the VMS components using the same user who has administrator rights.

• Terminal services are running during installation Because of interoperability issues, you must not be running Terminal services during IDS/IPS MC installation. We recommend disabling it during installation or upgrading IDS/IPS MC.

• Using special characters for passwords During the installation we recommend not using special characters for the different passwords. This causes confusion and installation failure.

• No Domain Name System (DNS) record for the VMS Server You must have a DNS record for the IP address of the VMS server with the name of the VMS Server. If you do not have the DNS Server configured for the IP address, be sure to modify your hosts file on the VMS server and on the client PC where you access the VMS Server. If you do not have the DNS record, you may be able to get to the login page, but may not be able to log in, because the web server points the link of subsequent pages to the name, not to the IP address.

• Corrupt IDS/IPS MC software or corrupt installation If you have corrupt IDS/IPS MC software or if the software installation is corrupted, you might receive one of the following messages:

  - Unable to retrieve package name (idsmdc_c).

  - Unable to retrieve package version (idsmdc_c).

  - Unable to register IDS/IPS MC Resource with CiscoWorks Common Services Client Registrar.

  - Unable to initialize IDS/IPS MC Configuration Manager Database.

  To resolve the issue, download the IDS/IPS MC from the Cisco Web site, and reinstall the IDS/IPS MC.

• Process dies or hangs Go to Server Configuration > Administration > Process Management > Process Status and be sure all the processes are running as shown in Table 18-1. If any of the processes dies, restart the service.

Table 18-1. Processes That Must Be Running for IDS/IPS MC

Processes	Description
Apache	Checks that the web server is running properly
ASANYs_SqlCoreDB	Checks that the SQL database is running properly
Daframework	Checks that the device agent is running properly
Fms	Checks that the database is running properly
IDS_Analyzer	Checks that the service that processes event rules and requests user-specified notifications when appropriate is running properly
IDS_Backup	Checks that the service that backs up and restores the IDS/IPS MC and Security Monitor database is running properly
IDS_DbAdminAnalyzer	Checks that the service that periodically applies active database rules to the current state of the server is running properly.
IDS_DeployDaemon	Checks that the service that manages all configuration deployments is running properly
IDS_EvsServer	Checks that the service that manages most Event Viewer data processing is running properly
IDS_Notifier	Checks that the service that receives notification requests (script, e-mail, and/or console) from other subsystems and performs the requested notification is running properly
IDS_Receiver	Checks that the service that receives IDS/IPS and syslog events and stores them in the database is running properly
IDS_ReportScheduler	Checks that the service that generates all scheduled reports is running properly
JRunProxyServer (Windows only)	Checks that the Java servlet engine is running properly
Lm	Checks that the FMS database lock manager is running properly
NRPostOfficeD	Checks that the post office service, which enables communication to IDS/IPS sensors, is running properly
WebServer	Checks that the Apache web server is running properly

IDS/IPS MC Licensing Issues

You must have a valid IDS/IPS MC license in addition to the common services license for IDS/IPS MC to function fully (note that both licenses can be in the same file). You may experience two issues with IDS/IPS MC trial or purchased license file:

• Corrupted license

• Expired license

Corrupted License

VMS or IDS/IPS MC license files may be corrupted either at the time of generation by Cisco Systems, Inc. licensing department or in transfer vie e-mail. In either case, if the license is corrupted, you need to get another license by sending an e-mail to [e-mail].

Sometimes, the VMS server gives false messages while importing a valid license file. So, it is extremely important to verify that the license file is indeed corrupted. The only way this can be verified is by uploading the license file to a known working VMS server. The Cisco support team can verify this for you. However, if the license happens to be good for a known working VMS server, then you might be running into one of the following issues for which you are getting the corrupted license message:

• Running terminal services If you are running terminal service on the VMS Server while importing the license, it will give you a false message, such as Invalid License. This is because there are known incompatibilities between Sybase SQL Anywhere and Windows Terminal Services. Sybase SQL Anywhere is used by CiscoWorks Common Services. Details about the problem are available on the Microsoft Web site:

  http://www.microsoft.com/windows2000/docs/W2kTSApCmpt.doc

  Under this circumstance, disable the terminal service and then import the license.

• Database corruption or database not running You might get a license corruption message if either the VMS or IDS/IPS MC database is corrupted or not running. If so, you must be sure that the IDS/IPS MC or VMS server database is running properly.

Determining If a License Is Expired

There is another licensing issue on IDS/IPS MC that you might run intothe expired license. The expired license needs to be replaced with a valid license. A trial license should not be replaced by another trial license. To determine the status of the license, go to VPN/Security Management Solution > Administration > Common Services > Licensing Information. Click on Select and choose the license file from your PC. Click on Upload. If you have an expired license, you will see the message as shown in Figure 18-2.

Figure 18-2. Error Message Seen with an Expired License

Importing Sensor Issues with IDS/IPS MC

IDS/IPS MC must add the sensor before you can configure it. This section examines the configuration steps on the IDS/IPS MC to add the sensor and troubleshooting steps, should you run into import problems.

Configuration Steps

Work through the steps that follow to configure IDS/IPS MC to import the configuration from the sensor:

Step 1.	Open IDS/IPS MC.
Step 2.	If you decide to create a device group, click on the Devices tab. Click on Group Option and click the Create Subgroup button. Provide a Group name and description and click OK.
Step 3.	Click the Devices tab and the Sensor option.
Step 4.	From the Group pull down box, select the group you just created.
Step 5.	Click Add to launch the Add Sensor Wizard to add a new IDS/IPS sensor to that group.
Step 6.	Fill out the sensor identification information. It should be similar to what is shown on the screen in Figure 18-3. Note that IDS/IPS MC uses this information to try to discover the sensor settings. Figure 18-3. Sensor Information in the Import on Sensor cisco is the username for this setup. You may have a different username, but be sure that the user has administrator privileges. The password here is ciscotac123. Be sure you do not check the SSH keys check box.
Step 7.	Click Next when ready. Since the IDS/IPS MC is set up to discover sensor settings, you should automatically see the version field populated.
Step 8.	When everything looks satisfactory, click Finish to import your sensor. You should now see your sensor in the list.

More details on how to add a sensor can be found at the following link:

http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/mgt_ids/idsmc20/ug/ch04.htm

Troubleshooting Steps

If you are unable to import the sensor configuration, work through the steps that follow to isolate and resolve the issue:

Step 1.	Be sure that the IDS/IPS MC machine can access the IDS/IPS sensor using SSH. The IDS/IPS MC communicates with the IDS/IPS sensor over SSH (22) and HTTPS (443). Be sure that both ports can communicate. Verify this with the procedure in the section entitled "Verifying the SSH and SSL Connection Between IDS/IPS MC and a Sensor" under "Important Procedures and Techniques." If you run into a problem with the SSH and or SSL connection with the sensor, follow the procedure entitled "Resolving SSH and SSL Connection Problems Between IDS/IPS MC and a Sensor" in the same section.
Step 2.	Be sure all the sensor services are running. All the processes must be running on the sensor for successful import of the sensor by the IDS/IPS MC. To verify if all the processes are running, execute the show version command (refer to "Verifying If the Sensor Process Are Running"). All the processes must have "running" status except the last process, which is the CLI process. If these are not all running, this will cause the import to fail. You can restart the services by logging into the service account and issuing /etc/init.d/cids restart. Unless you are experiencing a bug in the code, the service should restart.
Step 3.	Be sure that the IDS/IPS sensor version is supported by the IDS/IPS MC. There are many versions of IDS/IPS MC and the related sensors. You must ensure that the IDS/IPS Sensor version is supported by the IDS/IPS MC. To find out which IDS/IPS sensor versions are supported by different versions of IDS/IPS MC, refer to the following link: http://www.cisco.com/en/US/products/sw/cscowork/ps3990/products_device_support_tables_list.html
Step 4.	Be sure you have exactly the same signature or service pack level on the IDS/IPS MC and on the sensor. You must have the same service pack or signature levels on both the IDS/IPS MC and sensor. So, it is important to verify the version of IDS/IPS MC and sensor you are running. (Refer to the sections entitled "Verifying That the Service Pack or Signature Level Sensor Is Running" and "Verifying the Service Pack or Signature Level On IDS/IPS MC" under the "Important Procedures and Techniques" section.) Software versions must match exactly. For example, if you run version 4.1(4) S105 on the IDS/IPS sensor, you need to ensure that you apply IDS/IPS-sig-4.1-4-S105.zip to IDS/IPS MC. There is a common misconception that if you have a higher version of signature on the IDS/IPS MC, you can import the sensor that is running a lower version. This is untrue. Actually, the version needs to match to the signature level exactly for the import to work. For example, if you have IDS/IPS MC version 4.1(4)s120, you cannot import a sensor running 4.1(4)s105 unless the IDS/IPS MC is also running the same version, which is 4.1(4)s105. If you try to import the sensor configuration of a different version, it will warn you and will abort the import. If your IDS/IPS MC does not have the same signature or service pack level, you can download the latest signature from the following link: http://www.cisco.com/cgi-bin/tablebuild.pl/idsmc-ids4-sigup
Step 5.	Be sure that the username and password for the IDS/IPS sensor are correct. When adding a sensor, you must enter the username and password that is defined on the sensor for the IDS/IPS MC to be able to import the configuration from the sensor. You can execute the show users command on the sensor to identify the users who are defined on the sensor.
Step 6.	Be sure the SSL certificate on the IDS/IPS MC has not expired. If the SSL certificate on the IDS/IPS MC expires, you will not be able to import the configuration from sensor. You can verify if the certificate is expired with the keytool tool. (Refer to the section entitled "Verifying If the IDS/IPS MC (Apache) Certificate Is Valid" under "Important Procedures and Techniques.") If your certificate has expired, refer to "Regenerating IDS/IPS MC (Apache) Certificate" under "Important Procedures and Techniques" to generate the certificate.
Step 7.	If everything else fails, a reboot may be required for the server to resolve this import issue.

Signature or Service Pack Upgrade Issues with IDS/IPS MC

You can upgrade the sensor with a signature or service pack with the IDS/IPS MC. The sensor can be upgraded manually from the CLI as well (see "Manually Updating the Signature Level on the Sensor" under the section entitled "Important Procedures and Techniques"). If you decide to upgrade the sensor manually (which is not a very common practice), IDS/IPS MC can be upgraded itself.

The sections that follow discuss the upgrade procedure and troubleshooting steps.

Upgrade Procedure

Before proceeding with the upgrade it is recommended that you read the "ReadMe" file for the corresponding signature or service pack upgrade files. As an alternative, you can read the information on Cisco.com to find the details of the upgrade procedure and additional details specific to the upgrade file. This section explains the generic step-by-step procedure on how to perform the upgrade from IDS/IPS MC 2.x:

Step 1.	Download the signature or service pack update file (.zip extension) to the <install-dir>/MDC/etc/ids/updates directory on the IDS/IPS MC server from the following location: http://www.cisco.com/cgi-bin/tablebuild.pl/mgmt-ctr-ids
Step 2.	Open IDS/IPS MC from the CiscoWorks2000 Server desktop.
Step 3.	Go to the Configuration > Updates page.
Step 4.	Click on the Update Network IDS/IPS Signature link and then click the Submit button.
Step 5.	On the Update Network IDS/IPS Signatures page, select the appropriate Update File from the drop-down list and then click the Apply button.
Step 6.	In the next window, check the sensor or sensors that you want to upgrade. If you only want to upgrade the IDS/IPS MC, you do not have to select any sensor. Click the Next button.
Step 7.	In the next window, you can either choose to run the upgrade with the Immediate option or can schedule the upgrade to run later with the Scheduled option. This Scheduled option is only available on IDS/IPS MC version 2.0.
Step 8.	Click Next, and then Finish.

With IDS/IPS MC version 2.x, you can download the signature or service update file directly from Cisco.com with appropriate login credentials. Alternatively, you can download the file into a central local server and configure IDS/IPS MC in your network to pull the file from the local server automatically. Both these configuration options are available under Admin > System Configuration > Automatic Signature download.

By default, there is a maximum of 10 threads for performing an upgrade at one time by IDS/IPS MC. IDS/IPS MC takes a list of sensors and splits them between the 10 threads. So, if 30 sensors are selected for upgrade, each thread gets 3 sensors. IDS/IPS MC starts all threads and waits until they are all finished. Each thread gets the first sensor of the three selected sensors and starts the upgrade process. For every sensor upgrade, IDS/IPS MC logs in to the sensor and determines its version, and then logs out. IDS/IPS MC then decides if it needs to proceed with the upgrade. If IDS/IPS MC decides that the sensor needs to be upgraded, IDS/IPS MC executes the command over an SSH connection to the sensor, as shown in Example 18-9.

Example 18-9. Commands Executed by the IDS/IPS MC to Sensor for Upgrading Signature

sensor# configure terminal sesnor(config)# tls trusted-host ip-address <ipaddress of MC> port <https port of MC apache> yes sensor(config)# upgrade https://<ip address of MC>/vms/sensorupdate/IDS-sig-4.1-4-S97.rpm.pkg Yes sensor(config)# exit sensor# exit 

If the update fails, IDS/IPS MC tries to upgrade all over again, up to three times. If the sensor hangs at any point (no response), IDS/IPS MC waits 30 minutes before declaring update failed.

Troubleshooting Steps

After initiating the signature or service pack upgrade process, it is important to verify that the update has been applied successfully to the sensor. There are two ways to accomplish this:

• Log in to the sensor and execute show version to verify what version the sensor is upgraded to.

• Run an Audit Log Report (see "Audit Reports" under the section entitled "Diagnostic Commands and Tools") in IDS/IPS MC and check to see if the IDS/IPS upgrade is successful.

Before delving into the details of the troubleshooting steps, go through the Audit log report as shown in Example 18-10 for a successful signature upgrade only to the IDS/IPS MC server itself.

Example 18-10. Audit Reports for IDS/IPS MC Upgrade

2004-01-31 13:39:45 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL Signatures for the Host Based IDS have been added to the Signature Registry. ! This is the message indicating that IDS MC signature upgrade 4.1(3)S61 is being applied 2004-01-31 13:39:45 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL Signatures for Sensor version 4.1(3)S61 have been added to the Signature Registry. 2004-01-31 13:39:33 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update for version 4.1(3)S61 has processes Services completed. ! The following line confirms the signature upgrade addition to the IDS MC. 2004-01-31 13:39:33 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The meta-data provided in the Signature Update for version 4.1(3)S61 has been added to the system. 2004-01-31 13:39:15 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL A sensor update for version 4.1(3)S61 has started. 

Example 18-11 shows a sample Audit Log when sensor is upgrade is requested from IDS/IPS MC.

Example 18-11. Audit Reports for Sensor Upgrade b IDS/IPS MC Upgrade

2004-01-31 13:54:20PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update for version 4.1(3)S61 has completed. 2004-01-31 13:54:19 PST 0.0.0.0 Shared service processes Common Java System Services information admin Changes for Sensor vpn-ids saved and lock released. 2004-01-31 13:54:17 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The signature with the id of 5322.0 has been moved to a different micro-engine because of a version change. The micro-engine parameter tunings for this signature have been removed since they are not compatible with the new micro-engine. 2004-01-31 13:54:12 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The sensor inet-ids has been updated and its configuration modified to match the update. 2004-01-31 13:54:11 PST 0.0.0.0 Shared service processes Common Java System Services information admin Changes for Sensor inet-ids saved and lock released. 2004-01-31 13:54:06 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update process has completed for the sensor named vpn-ids. 2004-01-31 13:54:06 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update of sensor vpn-ids completed successfully. 2004-01-31 13:54:01 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update process has completed for the sensor named inet-ids. 2004-01-31 13:54:01 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update of sensor inet-ids completed successfully. 2004-01-31 13:53:54 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update was transferred to the sensor named vpn-ids. 2004-01-31 13:53:54 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The transfer of the update to sensor vpn-ids succeeded. 2004-01-31 13:53:54 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The transfer of the update to sensor inet-ids succeeded. 2004-01-31 13:53:54 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL The update was transferred to the sensor named inet-ids. 2004-01-31 13:53:53 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL Update of sensor vpn-ids started. 2004-01-31 13:53:53 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL Update of sensor inet-ids started. 2004-01-31 13:53:49 PST 0.0.0.0 Shared service processes Sensor Version Import information vpn-ids.OrganizationName:Successfully imported sensor version from the sensor 2004-01-31 13:53:40 PST 0.0.0.0 Shared service processes Sensor Version Import information inet-ids.OrganizationName: Successfully imported sensor version from the sensor 2004-01-31 13:53:35 PST 0.0.0.0 Shared service processes Common Java System Services information INTERNAL A sensor update for version 4.1(3)S61 has started. 

Once you become familiar with the successful signature or service pack upgrade audit log, work through the following steps to troubleshoot any issues with signature or service pack upgrade failure:

Step 1.	Make sure the sensor and IDS/IPS MC fulfills the minimum software version requirements. To be able to upgrade to the version you desire on the sensor or IDS/IPS MC, ensure that you have the minimum version required for the upgrade. This information can be found from the corresponding ReadMe file of the signature or the service packs file.
Step 2.	Be sure the necessary ports are open for IDS/IPS MC to IDS/IPS communication. The VMS server will ssh to the sensor on port 22, and then the IDS/IPS will pull the signature update down from the IDS/IPS MC via port 443, so be sure both ports are open between the two systems. If they are not, the job will hang. This can be verified with the procedure explained in the section entitled "Verifying SSH and SSL Connection between IDS/IPS MC and the Sensor" under the "Important Procedures and Techniques" section. If you run into a problem with SSH and or SSL connection with the sensor, follow the procedure entitled "Resolving SSH and SSL Connection Problems Between IDS/IPS MC And Sensor" in the same section.
Step 3.	Be sure the filenames that you downloaded have not changed since downloading. Be sure not to change the name of the files you downloaded from www.cisco.com and added to the ~CSCPpx/mdc/etc/ids/updates directory. The files in this directory should be Zip files. The Zip files contain the .rpm.pkg files that will be sent to the sensor. You should not edit the filenames or contents. You can download the zip files from the following location: http://www.cisco.com/pcgi-bin/tablebuild.pl/mgmt-ctr-ids-ids4updates
Step 4.	Be sure the time on the sensor and the IDS/IPS MC is in sync. Check the IDS/IPS sensor's time by issuing show clock detail from the command line. Also note the VMS server's current time. The times need to be the same for the update to occur correctly.
Step 5.	Be sure the sensor's processes are all running. If all the process are not running, you may see the following error message in the audit log: Sensor xxx: Signature Update Process The update of the sensor xxx failed during the update script process, msg=An error occurred while trying to determine the sensor version. Detail = Error occurred while communicating with xxx.xx.x.x: Connection refused: connect An error occurred while running the update script on the sensor named xxx. Detail = The update of the sensor xxx failed during the update script process, msg=An error occurred while trying to determine the sensor version. Detail = Error occurred while communicating with xxx.xx.x.x: Connection refused: connect Check to see that the necessary processes are running on the sensor. You can do this by logging into the sensor via CLI and issuing a show version command. You should see the processes below listed in the "Running" state. (See the section entitled "Verifying if the Sensor Processes Are Running" under "Important Procedures and Techniques"). If the processes are not running, or you cannot log in at all, try logging into the service account (if you have created one) and manually restarting the services.
Step 6.	Be sure the Apache certificate is valid The IDS/IPS MC (Apache) certificate must be valid for the successful secured communication between IDS/IPS MC and sensor. You can verify the certificate with the procedure explained in the section entitled "Verifying That the IDS/IPS MC (Apache) Certificate Is Valid" under "Important Procedures and Techniques." If the certificate is invalid, you can correct the problem by regenerating the certificate. (See the section entitled "Regenerating IDS/IPS MC (Apache) Certificate" under "Important Procedures and Techniques."
Step 7.	Be sure that the VMS server is listed as trusted in the configuration of the IDS/IPS sensor. Execute show tls trusted-hosts on the sensor and see if the IP address of your VMS server is listed in the output. If it is not listed, that means that the sensor is unable to get the certificate from the VMS server. The solution is explained in the next step. If you run into this problem, you will see errors such as that shown in AuditLogFirst.log file: 58639,2004-12-21 08:05:52.183,0,0,2,21000,1002,21005,2,21004,,'RDEP Collector (th-sns-3) parsed an evError: errSystemError MainApplication::processExecUpgradeCtlTrans errSystemError The host is not trusted. Add the host to the system''s trusted TLS certificates.'
Step 8.	Be sure the ids sensor is getting the certificate. If you change the IP address of the VMS server, or regenerate the Apache certificate, you may have difficulties in getting a new updated certificate. To resolve the certificate issue, follow the procedure explained in "Resolving Issues With IDS/IPS Sensor Being Unable To Get The Certificate" under the "Important Procedures and Techniques" section.
Step 9.	Be sure the IDS/IPS MC version settings for the sensor are in sync with the sensor version. Be sure that the IDS/IPS MC configuration for the sensor is in sync with the sensor's actual version. You also need to make sure they are in the same signature level. If the sensor version is out of sync with what IDS/IPS MC expects, the upgrade will fail. For example, if you import a sensor and then upgrade the sensor's signature pack manually, the IDS/IPS MC will fail when it tries to update it further. To re-sync the version number of the sensor with IDS/IPS MC 2.0, simply select your sensor in the table of contents and then go to Configuration-Settings-Identification, and you will see what version the IDS/IPS MC thinks that the sensor currently has. Click the Query sensor button to update the IDS/IPS MC with the current IDS/IPS version information.
Step 10.	Try re-importing the sensor into IDS/IPS MC. Sometimes it becomes necessary to delete the sensor from the IDS/IPS MC configuration, and then re-import the sensor to resolve the signature or a service pack upgrade issue.
Step 11.	Be sure the correct certificate is selected on the VMS server. In version IDS/IPS MC 2.0 or 2.0.1, be sure you have not configured VMS to use the "CiscoWorks certificate" for communications as opposed to the default "Common Services Certificate" under VPN/Security Management Solution > Administration > Configuration > Certificate. To work around this problem, select Common Services Certificate instead of CiscoWorks certificate. This is a problem only on IDS/IPS MC 2.0 or 2.0.1 with IDS/IPS 4.x sensors.
Step 12.	Be sure the VMS server has only one network card. A multi-homed machine has multiple Network Interface Card (NIC), each configured with different IP addresses. To run CiscoWorks Common Services on a multi-homed machine, there are two requirements. - First, all IP addresses must be configured in DNS. - Second, because of restrictions with CORBA, only one IP address can be used by the client or browser to access the server. You must select one IP address as the external address, with which the client will log in to the CiscoWorks server. To select an IP address, modify the gatekeeper file located in NMSROOT\lib\vbroker\gatekeeper.cfg. Replace every instance of external-IP-address with the external IP address you choose, and remove the "#" character, from the following: - #vbroker.gatekeeper.backcompat.callback.host=external-IP-address - #vbroker.se.exterior.host=external-IP-address - #vbroker.se.iiop_tp.host=external-IP-address - #vbroker.se.interior.host=external-IP-address After modifying the gatekeeper file, restart the Daemon Manager by entering: net start crmdmgtd
Step 13.	Be sure the VMS server IP address is unchanged. Ensure that the IP address of the VMS server running IDS/IPS MC has not changed since it was installed. IDS/IPS MC issues the upgrade command to the sensor and indicates where the update package exists. The old IP address is issued unless IDS/IPS MC is modified with the new IP address information (see "Changing the VMS Server IP Address" under the section entitled "Important Procedures and Techniques").
Step 14.	Take corrective action if a signature update has failed, leaving the virtualSensor.xml file incomplete. If a signature update is made to the sensor but the sensor did not apply the update completely, the Audit log will report an error message similar to the following: "Error while pushing files to the sensor...an exception occurred during deploy...Signature Conversion failed, Signature (xxx) from SensorConfig is not a custom signature and is not found in the sensor XML..." This indicates that a signature upgrade failed for some reason, and that you will have to re-apply the signature update. Verify if you are having this problem by looking at the /usr/cids/idsRoot/etc/VS-Config/virtualSensor.xml file. This file shows all the signatures that are configured on the sensor. You can grep this file for signature (for example "S146") to match against the latest signature showing with show version output. If you find that a sensor is running S146, yet there are missing S146 signatures, you should downgrade the sensor with the downgrade command, and then re-apply the signature update; or you can upgrade to a more recent signature version and it will overwrite the missing values.
Step 15.	Update from two digit signature pack to three digits (for example S91->S127). At the time of writing this book, up to version IDS/IPS MC 2.0, IDS/IPS MC cannot push down the update from a two-digit signature pack to a three-digit one. The solution is to manually update the signature level of the IDS/IPS sensor (see "Manually Updating the Signature Level on the Sensor" under the section entitled "Important Procedures and Techniques") and then re-import the sensor into IDS/IPS MC.
Step 16.	Be sure the signature pack is available on the IDS/IPS MC. To send the signature update to the sensor, that particular signature update must be available on the IDS/IPS MC. Check to be sure that the signature update is installed on the IDS/IPS MC. Otherwise it cannot be pushed to the sensor.
Step 17.	Check the Network Address Translation (NAT) between the IDS/IPS MC and the sensor. If you are running Network Address Translation (NAT) between the IDS/IPS MC and the IDS/IPS sensor, ensure that IDS/IPS MC Version 1.2.3 or later is installed with the latest patch.
Step 18.	Check to see if the connection between IDS/IPS MC and the sensor is slow. If the link between IDS/IPS MC and the IDS/IPS sensor is slow, ensure that IDS/IPS MC is running version 1.2 or later. In earlier versions of IDS/IPS MC, an update to an IDS/IPS sensor must be transferred within 30 minutes or the update fails. If desired, you can edit .../CSOCpx/MDC/Tomcat/vms/ids-config/WEB-INF/classes/com/cisco/nm/mdc/ids/common/sigupdate/updateinfo.xml file and change the TransferTimeout from the default as shown in Example 18-12 to your desired value. Example 18-12. Timeout Value in updateinfo.xml File !30 minutes for each update try <TransferTimeout>1800</TransferTimeout> !Retry update 3 times <TransferTries>3</TransferTries> !Wait 30 seconds before trying update again <TransferWaitBetweenTries>30</TransferWaitBetweenTries> !3.x only <SP_UpdateScriptTimeout>1800</SP_UpdateScriptTimeout> !3.x only <SU_UpdateScriptTimeout>600</SU_UpdateScriptTimeout> !Max number of concurrent updates to do <MaxThreads>10</MaxThreads> !after actual "upgrade https..." is done, how long between !tries to see if what version of !sensor is <CheckUpdate>120</CheckUpdate> Be sure not to set a timeout for too long a period. If the timeout is too long, the update process may take a long time, and if there is a failure, the IDS/IPS MC will not be able to start another signature update until the timeout period is completely finished.
Step 19.	Check for IDS/IPS MC Database connection problems. If you experience issues with the database connection, you may be unable to upgrade the IDS/IPS sensor or the IDS/IPS MC itself, and you may be unable to access many components within IDS/IPS MC. Ensure that IDS/IPS MC version 1.2.3 or later is installed with the latest patch.

Configuration Deployment Issues with IDS/IPS MC

There are several XML files (for example virtualSensor.xml, analysisEngine.xml, and Hosts.xml, etc.) that the IDS/IPS MC imports from the sensor to build the configuration of the sensor in IDS/IPS MC. Once you make any changes on the sensor, depending on the version of IDS/IPS MC you are running, deployment of the configuration is performed in different ways. This is explained in the following section:

• Pre-version IDS/IPS MC 2.x behavior Before IDS/IPS MC version 2.x, the sensor configured is deployed using SSH. IDS/IPS MC uses the CLI processor to push the configuration changes over the SSH connection command-by-command on the CLI parser of the sensor. It is as if you are manually typing the commands after making the SSH connection. For this reason, the processing time with configuration deployment with earlier versions were very long.

• IDS/IPS MC Version 2.x With version IDS/IPS MC 2.x, the configuration deployment method has changed. Now, IDS/IPS MC makes the changes, generates all the necessary XML files with the changes made, and then imports the XML files from the sensor to compare them with the IDS/IPS MC local file using the MD5 hashing algorithm. Because of the changes in IDS/IPS MC local XML files, this hashing algorithm result will be different. At this point, instead of sending the commands to the CLI parser of the sensor using SSH, IDS/IPS MC pushes all the XML files to the sensor to overwrite the existing ones with these modified XML files created by IDS/IPS MC.

Even though the configuration deployment process is different behind the scenes, the configuration and troubleshooting steps are the same on different versions, as discussed in the section that follows.

Configuration Steps

After the sensor is imported to the IDS/IPS MC, you may make the necessary changes to the sensor configuration on IDS/IPS MC and then generate the configuration before pushing it to the sensor. To do so, work through the following steps:

Step 1.	Make changes to a sensor by going to Configuration->Settings, and choosing your sensor under Object Selector. Then make the changes you desire. For example, choose a signature (SigID 2000 is "ICMP Echo Request") for testing purposes.
Step 2.	Check the box next to the signature and click Edit, make changes, and then click OK.
Step 3.	Go to the tab entitled Configuration->Pending to save the pending configuration. Choose your sensor (if it is there) and click the check box next to its name. Then click Save.
Step 4.	Go to the Deployment tab and choose the Generate Option. Click the check box next to the name of the sensor you wish to change and click Generate. It will present a pop-up window. Click OK to proceed, then Close to close the window.
Step 5.	Go to the Deployment tab and choose Approve. If your sensor is listed there, click the check box next to its name and click Approve.
Step 6.	Go to the Deployment tab and choose Deploy and then Submit. Choose your sensor from the list and click Deploy. Choose your configuration and click Next. Then click Next for an immediate deploy, and click Finish.
Step 7.	To quickly check to see if and when the deployment is successful, go to the Configuration > History page. In IDS/IPS MC version 2.x, you can also click the Real time progress Viewer icon (it is a page with a small red 'x' in the upper-right corner of the screen, next to Actions and Notifications).

Note

A deployment to multiple sensors may take several minutes depending on the version you are running. If you are running a version earlier than IDS/IPS MC 2.x, it might take an hour or more. So, it is extremely important that you do not reset or stress the IDS/IPS MC server, and the sensor.

Troubleshooting Steps

After pushing out the configuration changes to the sensor from IDS/IPS MC, the next step is to ensure that the configuration changes applied to the sensor are successful. You can verify this in three ways (not all versions have all three options):

• Using Progress Viewer After a deployment job is requested, click the Real time progress Viewer icon (it is a page with a small red 'x' in the upper-right corner of the screen, next to "Actions and Notifications." Wait about four minutes for the sensor to show up at the top of the list; the configuration change status should appear. Click Refresh to refresh the screen. This option is only available on IDS/IPS MC 2.x. If the deployment fails, click the check box next to the sensor and click Show message. The message will contain some details on failure.

• Using the History page This option is available on all versions of IDS/IPS MC. Check the status of the deployment by going to Configuration > History. From this page, you can find out the status of individual sensors.

• Using the Configuration Deployment report If you have many sensors deployed, it is better to generate a Configuration Deployment report. This feature is available on all IDS/IPS MC versions. You can generate this report by going to Reports > Definitions > Create > Detailed Sensor Configuration Deployment Report > Next > Next > Run Now. This is the simplest form of the report generation process. There are other options available with this process.

Once you have identified a problem with Configuration deployment by using one of the previously listed methods listed, the next step is to analyze the problem through generating the following necessary log files:

• Using Progress Viewer As discussed before, using Progress Viewer, you can check not only the status of the deployment, but can get the details on the reasons for the failure. If the deployment fails, click the check box next to the sensor and click Show message. The message will contain details on the reasons for the failure.

• Using Audit Log You can generate an Audit Log to find additional details on deployment as discussed in the "Diagnostic Commands and Tools" section. To generate an audit log report, click the Reports tab and then Definitions. Press the Create button and choose Audit Log Report, then Next. Choose all the options on this screen and click Next. Then click Run Now under Finish Options and click Finish. Once this report is completed, it will show up under Reports and Completed. Give it several minutes to complete. Once it is done, you can view the report or send it offline.

Using the points you learned in the previous discussion, you can delve into the details of some probable causes of deployment failure:

• Required Process not running on IDS/IPS MC You might run into a problem when you have no sensors in Deployment selection even though you have made changes to the sensor configuration. This may happen if the IDS_DeployDaemon process is not running. Because, when a deployment is started, this daemon checks the devices that have pending configuration changes to be deployed and shows them in the Deployment page. If this daemon is not running, this task is not performed, so no devices are shown, even if there are changes waiting to be deployed. To resolve this problem, start the process by going to Server Configuration > Administration > Process Management > Process start and start the process.

• Version Mismatch between IDS/IPS MC and a sensor Usually you should not run into this problem, unless you manually upgraded the sensor after the sensor configuration was imported. To verify the version information of the sensor, execute the show version command on the sensor. Then on the IDS/IPS MC, select the sensor from the Object Selector, then go to Configuration > Settings > Identification, and look for version information. If the version does not match, then click Query Sensor on the same page to retrieve the current sensor version information. And then click the Apply button to save the changes. If the Query Sensor fails, most likely the problem is that you have applied the signature or service pack upgrade on the sensor only. Therefore, to match the sensor upgrade, you must upgrade your IDS/IPS MC to the corresponding version as well. Then perform the query again.

• XML file corrupted VirtualSensor.xml on the sensor file may become corrupted due to failure of signature or service pack upgrade. If it is, your configuration deployment will fail as well. If you check the output of the show messages button on the Progress Viewer page within IDS/IPS MC, it might say the following:

  A signature update failed, leaving the virtualSensor.xml file incomplete "Error while pushing files to the sensor...an exception occurred during deploy...Signature Conversion failed, Signature (xxx) from SensorConfig is not a custom signature and is not found in the sensor XML..." 

  To resolve this issue, re-apply the signature update and then deploy the configuration.

Database Maintenance (Pruning) Issues

Both IDS/IPS MC and Security Monitor share the same database, Sybase for Audit Log, NIDS, CSA, and firewall logs. If the Security Monitor is installed on the same server as IDS/IPS MC, it is very important to manage the database for the logs, so that it does not outgrow the capacity of the server. The method used for database maintenance is called Pruning. Pruning, and database maintenance in general, is discussed in greater detail in Chapter 22, "Troubleshooting IEV and Security Monitors." So, those topics will not be covered in this chapter. However, this section discusses the configuration for Pruning, which is specific to IDS/IPS MC only. On IDS/IPS MC version 1.x, the default Pruning is accomplished by executing scripts, which have changed in version 2.x. There are three default Pruning rules in IDS/IPS MC version 1.x: IDS/IPS Events, Syslog Events, and Audit Logs. These rules can be edited and are found under Admin > Database Rules. Note that the same default Pruning rules are configured as the default for the Security Monitor on 1.x. In reality, the IDS/IPS Log and Syslog are received inserted into the database by the Security Monitor, and IDS/IPS MC only inserts the Audit Log to the database. So, in version IDS/IPS MC 2.x, the options available for modification are for Audit Log and Progress Viewer under Admin > Database Management > Database > Pruning Configuration.