The Router MC troubleshooting issues can be categorized as follows:
The sections that follow discuss each of these issues in detail.
Installation and Upgrade Issues
If Router MC Installation fails on the Windows platform, you need to analyze the log in the C:\directory. This is where you can find the log in the format of C:\Ciscoworks_setupxxx.log (for example, the C:\Ciscoworks_setup001.log) file. The file name with the highest number is the most recent log file. Following are some of the probable causes of Router MC Installation/Upgrade failure on Windows platform:
Installation directory already exists If the installation directory is not removed completely during a previous uninstall operation, the new installation process will fail. You may receive an error message indicating that the iosmdc installation subdirectory already exists under the Common Services installation directory. If so, you cannot install the Router MC.
To resolve the issue, you need to delete the iosmdc installation directory, and then start the installation process again. If some of the files are locked and cannot be deleted, restart your system, and then delete those files.
Insufficient disk space to invoke Router MC If you do not have sufficient disk space on the system on which you are trying to install Router MC, you may receive a message such as this:
Router MC Installation file cannot be unzipped due to insufficient disk space.
To install the Router MC, you must click Cancel, free up sufficient disk space, and then try again.
Application server not removed after Router MC uninstall When Router MC is uninstalled from a server on which QoS Policy Manager (QPM) is installed, the Application Server service, which is a required process for Router MC, is not removed even after removing the QPM. The Application Server service must not be left installed on a server after Router MC is uninstalled. Otherwise, re-installation of Firewall MC will fail.
To resolve this issue, after uninstalling both Router MC and QPM, remove the Application Server open up DOS prompt and go to \CSCOpx\MDC\Shared\Services\EJB_Server. Then run the following command to delete the Application Server from the list in Windows.
Exe2Service.exe REMOVE AppSrv
In addition, run the following command to remove the Application Server service from the Daemon Manager:
pdreg -u AppSrv
You might encounter problems with bringing up Common Services due to one of the following reasons:
Required process not running If you cannot log in to the Router MC, you should first verify that all processes required are running by selecting Server Configuration > Administration > Process Management > Process Status from the CiscoWorks desktop. Refer to the "Router MC Processes" section to find information on the processes that are pertinent to Router MC. If any of the processes are not running, you can start the processes by selecting Server Configuration > Administration > Process Management > Start Process. If you cannot start the required services from the CiscoWorks desktop, stop and restart all services from the command line on the server with the following commands:
net stop crmdmgtd net start crmdmgtd
You can perform the same task by selecting Start > Settings > Control Panel > Administrative Tools > Services. If the problems persist, try rebooting the server.
Changing Windows account password If you install Common Services and Router MC using a specific admin account and password in Windows, and then change that Windows account password, the installed processes fail to start due to the change in password.
To resolve this issue, change the password for all processes so that they match the current password of the Windows account within which they were installed. Common Services processes include tomcat, fms, lm, and daframework.
Port conflict issues Common Services use port 1741. So, if there is another application running on the server using the same port, Common Services will not be able to start.
Under this circumstance, you must change the port for other applications, and then restart the server system.
Changed database password If you change the Router MC Database password by selecting VPN/Security Management Solution > Administration > Configuration > Database Credentials in the CiscoWorks desktop, you must restart the Router MC server. If try to start Router MC without restarting the Router MC server, the connection to the database will be lost.
Internal error on login Immediately after the installation, if you try to log in to Router MC, you may receive an internal error message. This may be because services for Router MC require a few minutes to start.
Close your Router MC Browser window, then log into Router MC again from the CiscoWorks desktop. If this does not solve the problem, log out of Router MC and CiscoWorks. On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services window. After a few minutes, log into CiscoWorks and Router MC again. If the problem persists, reboot the Router MC server.
Following are some of the problems that you may experience while working with Router MC on your browser:
Unsupported version of Java Runtime Environment (JRE) If you are running an unsupported version of Java Runtime Environment (JRE), you may get Java Runtime error messages while working with Router MC. To check the version of Java you are running, go to Start > Settings > Control Panel > Java Plug-in from the Windows taskbar. Select the Advanced tab. Under Java Runtime Environment, click on the drop-down menu and be sure the supported version of JRE is installed. For Router MC Version 1.3.1, JRE Version 1.4.1_2 is supported as per the "Product Overview" link of Router MC Version 1.3.x.
Go thorough the Release notes of the Router MC Version, and search by Java to see if there is any known issue on a specific JRE version. For example, if you are running Router MC Version 1.3.1, there is a known problem with the JRE version 1.4.2_06 as per the Release note for Router MC Version 1.3.1, as shown in the following link:
Popup blocker You must turn off the popup blocker on your browser to allow Router MC to come up. If the popup blocker is on, the Router MC Window will not come up at all.
Security warning popup When using Java plug-in 1.4.x, a security warning popup appears when you first navigate to a page that uses the Object Selector. If you try to navigate to another page or perform an action while the security warning popup is loading, the browser may freeze. To resolve this issue, close the browser and log out of CiscoWorks. Then log into CiscoWorks and Router MC again. The next time you get the security warning popup, wait a few seconds for it to load, and then click yes or always to accept the certificate. Refer to the following link for more details: http://www.cisco.com/univercd/cc/td/doc/product/rtrmgmt/cw2000/cw2000_b/vpnman/vms_2_2/rmc13/useguide/u13_gst.htm#wp1383679
User interface disappears while working While working with Router MC, the user interface might suddenly disappear, and you might get a message that you can no longer access Router MC.
When working in Workflow Disabled mode, only one Router MC Session per user can be open on the same server at any given time. If you are working with Router MC and another user with the same username and password logs in, your access to Router MC is blocked.
To resolve the issue, close your browser and log into Router MC again and you will observe the following:
- If you log in with the same username and password, you will reopen your previous session with all your changes, and the other users' sessions will be terminated.
- If you log in with a different username and password, a new session of Router MC will open and you will not see the changes you made in your previous session unless you saved and deployed them.
When working in Workflow Disabled mode, it is recommended to have only one session per username and password. If necessary, create new users from the Cisco Works desktop. Go to Server Configuration > Setup > Security > Add Users and add a user with the appropriate permissions. Click the Help button in the Adding Users page for more information.
Certification problem with Netscape Browser If you encounter a certification problem when trying to log in using the Netscape browser, the downloaded certificate might be corrupted. You can correct the problem by removing the certificate from the browser certificate storage as follows:
- For Netscape on Windows 2000
Go to Edit > Preferences > Privacy and Security > Certificates > Manage Certificates > Authorities, and delete the certificate that matches the Router MC server. Then, restart Netscape.
- For Netscape on Solaris
Go to Edit > Preferences > Privacy and Security > Certificates > Manage Certificates > Web Sites, and delete the problematic certificate. Then, restart Netscape.
Now if you bring up the browser and try to log in to CiscoWorks Common Services, you will be prompted to download a new certificate.
Authentication can play a role in Router MC in one or both of these ways, which are detailed in the sections that follow:
Authentication Issues with the Router MC
Once you log in to Router MC, you might see that many buttons in the User Interface are grayed out. The probable cause is that you do not have the correct user privileges to perform the tasks associated with the grayed-out buttons, or for the group or device currently selected in the object selector. For example, assume you have privileges to manage devices but some functions in the Devices tab are disabled. This might be because a group or device for which you have restricted privileges is selected in the object selector.
To resolve this issue, verify your user permissions in the CiscoWorks desktop or in ACS (depending on what method is being used for user authentication).
Authentication Issues with the Managed Device Using SSH
Router MC communicates with the devices using SSH. The Troubleshooting steps might differ depending on how you have configured devices to authenticate SSH connection. If AAA is implemented for SSH, refer to Chapter 9, "Troubleshooting AAA on IOS Routers" for troubleshooting details.
Activity and Job Management Issues
The sections that follow outline some of the issues that you may experience with the Activities.
Dangling locks If you experience a problem with Activity being locked by user X, even though user X is not logged in, or has closed the application, then there might be a dangling connection in the server.
Dangling connections occur because browsing is connectionless, and therefore you cannot be sure if a user simply closes a browser window. Users must either explicitly close activities or use the Logout (top-right of every screen) to close the session completely with the server. If a user closes all windows without closing activities or logging out, the administrator can use Close from the Activities page, or an implied Logout will occur after the session time-out time has passed.
Inability to access an activity after closing and re-opening the Router MC If you close Router MC using the browser's close button and you do not log out of Router MC, your activity remains open. When you log into Router MC again, you will be unable to access it.
Wait until the activity times out or ask your administrator to close the activity for you. In general, always close your open activity before closing Router MC.
Inability to create a new activity An activity of the same name might already exist in the system. Be sure you give the new activity a unique name.
Some activities do not display in the list of activities In rare cases, previously existing activities are no longer visible in the user interface. You need to log out of Router MC and CiscoWorks. On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. After a few minutes, log into CiscoWorks and Router MC again.
Changes made within an activity were retained after the activity was deleted When you delete an activity, all the configurations that were made within the activity should be deleted. However, changes that are made to a device group name, device model, or device IOS version are not activity sensitive. This means that if the activity under which the changes were made is deleted, the changes will still be committed to Router MC's database.
To resolve the issue, create a new activity and reverse the changes.
Object is locked by another activity Another user might be configuring that device or device group in a different activity, in which case it would be locked to other users. A red lock icon next to the selected object indicates that the object is locked by another activity.
Point your mouse at the red lock icon next to the selected object. A tool tip will indicate which activity is locking the object.
Device Import Issues
The sections that follow outline problems with importing a configuration from a file or a live router into Router MC:
Import of configuration file fails If you try to import device configuration from a file, and if the file size is greater than 583 kilobytes, the import will fail. Hence, if you have a larger configuration, import it from a live device.
Import of live device(s) fails Importing the configuration from a live device might fail due to one or more of the following problems:
- Connectivity problem
To find out if you have connectivity problems by pinging the router from Router MC. If this is successful, try to establish an SSH connection with the router using a third party SSH client (puTTY client, for example, which can be downloaded free from the Internet). If this is not successful, the problem could be with the authentication failure. Or you might have mis-configured or not configured SSH on the router.
- SSH Configuration problem
Be sure that you have SSH configured on the device. If SSH is not configured, configure it as per the following link:
- SSH version mismatch
If SSH version 2 is enabled on the device and Router MC supports only SSH version 1, you will run into SSH version mismatch issues. To resolve these issues, check which SSH version is enabled on the device by typing show ip ssh at the command line. If Version 2 is enabled, change to Version 1 using the command: ip ssh version 1.
- Login information is incorrect
If the login credential defined in the Router MC is incorrect, you will have login problem with the device, which will in turn cause the import process to fail. Revise the login information and correct any problem that you may have with either username or password. If Authentication, Authorization and Accounting (AAA) is configured on the router, run the debug for AAA on the router (refer to Chapter 9, "Troubleshooting AAA on IOS Routers") to correct any authentication problems.
Group lock problem When selecting the target group, if you get the message group x is locked by another activity, then you need to select a different group, or approve/reject the locking activity.
Device already exists If you are trying to import a device that has the same hostname or IP address as an existing device, import will fail. Note that you cannot import the same device via different interface.
Do not import a spoke via its inside (private) interface. This will cause a connection loss to the spoke while configuring it. It is recommended to import a spoke via its external interface, where VPN tunnel terminates.
Some imported devices do not show up in the device hierarchy If devices you imported into Router MC are no longer visible in the user interface, you must log out of Router MC and CiscoWorks. On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. After a few minutes, log into CiscoWorks and Router MC again. Note that if you restart CW2000 Daemon Manager, the other management console also will be restarted.
Configuration Generation and Deployment Issues
The lists that follow outline issues that you may encounter while generating or deploying of device configuration from Router MC:
Configuration generation failure The configuration may fail if the current configuration of the device is edited manually after import and either has no exclamation marks in it or does not end with the command end.
Live deployment may fail If due to login failure or timeout problems, the devices are not accessible when you are trying to update the current configuration, live deployment will fail. This may happen if the IOS does not support the CLI that is generated by Router MC. For example, if the Router MC pushes a configuration with 3DES encryption, but the device only supports DES, the configuration deployment will fail.
The VPN policy changes were not deployed to the device Router MC might be set up to manage pre-shared keys only. In this case, only pre-shared key configurations are generated and deployed even though other VPN policy configuration changes have been made.
Go to Configuration > IKE > Pre-shared Key. A note at the bottom of the Pre-shared Key page indicates whether the application is set up to manage pre-shared keys only. You can change this setting from here, if necessary.
Required device does not appear in the device tree When creating a Job, if the required device does not appear in the Device tree to select, it can be for either of the following reasons:
- The activity within which the device was imported has not been approved; therefore, the device is not yet included in the database. In this case, cancel the job creation, approve the activity, and create the job again. The device should now be available for selection.
- The device is already included in another job that has not yet been deployed. In this case, cancel the job creation, deploy or reject the other job to free its devices, and create the job again. The device should now be available for selection.
Deployment fails because router mc is unable to log into the device The reasons for this failure are the same as those described in the section entitled "Import of Live Device(s) Fails" under "Device Import Issues." Therefore, follow the same troubleshooting steps as those described under "Import of Live Device(s) Fails."
Deployment hangs When deploying IPsec without generic routing encapsulation (GRE), if the job deployment remains in progress for a long time (approximately 10 minutes) and then fails, the causes could be as follows:
- Connection to the spoke is lost because the same interface is defined as both the VPN interface (where tunnel terminates) and the inside interface. Redefine either the VPN interface or the inside interface on the device by selecting Configuration > Settings.
- The interface IP address used to identify the device for import is the same as the inside interface. Delete the device from the inventory and then import it again using the IP address of its external interface as the device name. Select the Devices tab to access the options for device management and import.
Job action does not end and it remains in transient status A job action that does not end and remains in transient status (for example, generating, deploying, and so on) prevents the job from being deployed. The job remains active and the devices in the job cannot be included in other jobs.
This might be due to the failure of one of the services associated with Router MC.
On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. This causes the job action to fail, thereby freeing its devices to be included in other jobs. After restarting the Daemon Manager, log into CiscoWorks and Router MC again.
Some of the hub configurations are not shown in the View Configurations Page under the Configuration tab Peer-oriented policies (such as pre-shared key and tunnel policies) are defined on spokes. Router MC mirrors these policies on the hub and generates the required configurations for the relevant hubs. This is done during job creation, not during policy definition. Therefore, the proposed configurations for the hub cannot be seen in the View Configurations page under the Configuration tab.
Define the required policies, approve the activity, and then create a job that includes the spokes associated with the hub for which you want to see configurations. The View Configurations page under the Deployment tab will show the configurations that Router MC has generated for the hub, which are based on the policies defined on its assigned spokes.
Aggressive mode commands are not deployed to the device, although they appear in the View Configurations page If there is an existing pre-shared key defined on the device, the Aggressive mode commands might not be written to the device. Router MC generates the commands and deploys them, but they are not reflected in the device's show-run.
Reboot the device. Then, re-import the device into Router MC, and deploy to the device.
Database Management Issues
The sections that follow cover some problems that you might encounter with Backup/Restore operation:
Backing up and Restoring Database
You should back up the database regularly so that you have a safe copy of the CiscoWorks Common Services database, which includes Router MC. The next two sections discuss both the backing up and restoration of the Router MC Database.
Database Backup Procedure
To back up the database, work through the steps that follow:
Select VPN/Security Management Solution > Administration > Common Services > Backup Database from the navigation tree. The Backup Database page displays.
Specify the path to the directory in which you want to store the backup.
To send an e-mail to a designated recipient each time the database is backed up, select the E-mail Notification check box and enter an e-mail address in the field.
To back up the database immediately, check Immediate.
To back up the database at a specific date and time, deselect Immediate, and define the Start Date, and Start Time time.
To schedule a backup at regular intervals, enter a value in the Repeat After field, and select Days, Hours, or Minutes from the list. To limit the number of times the database backup occurs, enter a value in the Limit Occurrences field under Frequency.
To back up the database according to the settings you have made, click Finish.
Click OK to close the message.
Database Restore Procedure
You can restore the database from an existing backup. The backup contains data from all installed CiscoWorks Common Services client applications. Because user account information is not backed up, you cannot use Restore to recover deleted accounts. Additionally, license information is not restored; the license in effect when the restore is performed remains in effect after the restore.
Troubleshooting Router MC Backup/Restore Operations
Following are some of the possible problems you may encounter with the Router MC Backup/Restore operations:
Router MC is not included in the restore operation This problem occurs if you perform a second restore and you have not rebooted the server after the first restore. After restoring data, reboot the Router MC server.
The restore operation failed but all recent changes were lost When you start a restore, the current database is cleared. Therefore, even if the restore fails, your most recent changes within the Router MC GUI will not be retained. Hence, back up your database before starting the restore. If the first fails, try to run a second restore, or try to restart the CW2000 Daemon Manager, and then run another restore operation.
An internal error displays when launching Router MC after a backup and restore operation After performing a backup and restore operation, if you get an internal error when launching Router MC, reboot the Router MC server to resolve the issue.
Router MC crashes after a database password change This may occur if you change the database password using Cisco Works Common Services, and you do not restart the server that is running Router MC.
To resolve this issue, after making a database password change, reboot the Router MC server.