This section describes the tools that you can use to monitor CallManager and your network:
Cisco CallManager Serviceability
CallManager Serviceability provides alarms, traces, component version information, service activation and control, Quality Reporting Tool reports, and performance
CallManager Serviceability, shown in Figure 6-5, provides the following features:
Figure 6-5. Example of a Screen in CallManager Serviceability
The CallManager system generates alarms to notify you of various events. The alarms are categorized into different levels, as described in Table 6-1. As of CallManager release 4.1, only error levels Debug and Error trigger alarms. You can configure the system to have alarms forwarded to selected monitors, and for each monitor, you can also choose to capture alarms of certain levels. These settings are configured in the Alarm Configuration screen in CallManager Serviceability for each service (such as Cisco CallManager, Cisco TFTP, and so on). Alarm Configuration also enables you to apply a given configuration to all nodes in the CallManager cluster. The
Table 6-1. Alarm Event Levels
You can view detailed alarm descriptions in CallManager Serviceability. Click
Alarm > Definitions
to view these definitions. A simple search dialog box is provided that allows you to search for and view alarms, their descriptions, and recommended actions. After a search has returned a list of alarms, you can click any part of the alarm description to see detailed information about that alarm. The detailed information also includes an explanation of any reason codes that are given as part of the alarm. For example, an alarm generated by a transient device will have a reason code that explains why the device is
Figure 6-6 shows the Alarm Detail screen after a specific alarm (in this case, SDLLinkOOS) has been clicked from the Alarm Message Definitions screen. You can view detailed alarm information for any alarm by clicking it from the Alarm Message Definitions screen.
Figure 6-6. Alarm Details Window
Traces are diagnostic tools that can help you determine the cause of a problem. Running a trace can generate information you might need to identify or isolate the source or symptoms of a problem. The
There are two kinds of traces:
To generate good, usable trace information, all clocks should match on all CallManager-
SDI traces (also known as CCM traces) are useful for diagnosing most problems you might encounter with the CallManager system. SDI traces log different types of runtime events related to CallManager -including device
For example, a user reports that a dialed number results in reorder tone. You can turn on SDI trace and trace only the phone and gateway involved to learn why CallManager cannot connect the
You can choose to log SDI traces in XML format or in standard text-based format. When XML trace formatting is selected, the resulting trace logs can be used with Trace Analysis feature; however, Cisco recommends leaving trace levels set to text-based format for readability reasons.
SDI tracing is enabled by default; however, the default logging level is Error, which provides limited debugging information beyond critical system failures.
SDI Trace Output
SDI traces generate files (for example, ccm000000000.txt) that contain traces of CallManager activities. These traces provide information about the CallManager initialization process, registration process, call flow, digit analysis, and related devices, such as Cisco IP Phones, gateways, gatekeepers, and more. This information can help you isolate problems when troubleshooting CallManager.
The trace files are stored in the following default location:
Cisco recommends changing the default file location for trace files to the F: drive on dual-processor CallManager servers equipped with extra hard
If a trace is enabled, a new trace file is started each time CallManager restarts or when the designated number of lines or minutes has been reached. Although SDI traces do
The Cisco Technical Assistance Center (TAC) and Cisco engineers use SDL traces to diagnose difficult problems. The only time you should change the SDL trace configuration is when directed to do so by TAC. SDL trace logs state transitions only for CallManager and Cisco CTI Manager. Cisco engineers use SDL traces to find the cause of an error. You are not expected to understand the information contained in an SDL trace. However, while working with TAC, you might be asked to change the SDL trace configuration and provide the resulting trace files to TAC.
Before gathering any SDL traces in CallManager Serviceability, you must enable SDL tracing in CallManager Administration. The Cisco TAC representative requesting the trace can
Trace configuration allows you to specify the criteria for SDI tracing. Click Trace > Configuration to select the server you want to trace, and then select the service on that server, such as Cisco CallManager, the Database Layer, CTI Manager, and so on. Figure 6-7 shows the Trace Configuration screen with the default values selected. You can custom configure the trace by selecting the level of trace, the trace fields, and device names (if applicable). Table 6-2 provides a list of the debug trace level settings.
Figure 6-7. SDI Trace Configuration Page
Table 6-2. Debug Trace Levels
CallManager Serviceability provides the option of
By default, trace results are saved in a file. You can choose to save traces in TXT format, where up to 10,000 lines can be saved and the file can be
After a trace starts, it continues until you turn it off. When tracing
Troubleshooting Trace Settings
The Troubleshooting Trace Settings page enables you to easily configure trace settings for troubleshooting most services without having to configure each service one at a time. The Troubleshooting Trace Settings page uses trace levels based on Cisco TAC recommendations. Figure 6-8 shows the Troubleshooting Trace Settings page in CallManager Serviceability.
Figure 6-8. Troubleshooting Trace Settings Page
To apply the TAC-recommended trace settings for a service, select the check box corresponding to the service for which you want to enable traces. You can choose Select all Nodes for a Service to enable traces for a particular service on all nodes in the cluster or use Check all Services for a Node to select all the services on a particular node. When Troubleshooting Trace Settings are enabled, you cannot manually change trace settings until you disable Troubleshooting Trace Settings.
Trace Analysis allows you to perform post-filtering on an XML trace file so that only the relevant information displays. Because trace files can be large and
With Trace Analysis, you can specify collection criteria such as the following:
You can filter the trace so that only the pertinent information displays. The following information can be displayed or filtered out of the trace:
In practice, administrators who are proficient in reading CallManager trace files find that reading and filtering through a raw text-based trace file is much quicker and easier than using Trace Analysis. Also, Trace Analysis is very slow for large amounts of trace data. Future CallManager releases might see XML tracing and the Trace Analysis features removed.
Trace Analysis runs as a low-priority task so that it does not
Before a service can be used on a CallManager node in a cluster, it must be activated, which you can do on the Service Activation page in CallManager Serviceability (
Tools > Service Activation
). Activating or
You should never change the service startup type for any Cisco service directly from the Windows Services Administrative Tool. Always use Service Activation to change the startup type.
CallManager Serviceability enables you to start and stop services by clicking
Tools > Control Center
. The Control Center also indicates whether the services on the CallManager nodes are currently running or
The Quality Reporting Tool (QRT) service enables you to add a
QRT Viewer enables you to select a server in the cluster and a date/time range. After clicking
, a page showing all QRT reports submitted during that timeframe displays, including details such as the date and time of the report, the problem being reported, the device from which the report was submitted, and voice quality statistics (packet loss, jitter, and so on) if the problem was
Figure 6-9 shows a sample QRT report indicating that a user was unable to place a call and received a fast busy (reorder) tone. The report shows the date and time the problem occurred, device name, phone number, and IP address.
Figure 6-9. QRT Report
Serviceability Reports Archive
CallManager Serviceability monitors various aspects of a CallManager cluster and generates nightly reports in PDF format. You can access these reports from the Serviceability Reports Archive in CallManager Serviceability ( Tools > Serviceability Reports Archive ).
Each day, five reports are generated:
The Serviceability Reports Archive data is presented in a graphical format that enables you to view trends or any sudden events that might have occurred on a given day. For example, you might look at a report and see that CPU utilization is always high between 3 and 4 a.m., but there is no significant call activity during this time. You can then use this information to investigate what might be
By default, the Serviceability Reports Archive stores the last seven days of data. You can increase this in CallManager Administration ( Service > Service Parameters > select a server > Cisco Serviceability Reporter > RTMT Report Deletion Age ). You can also configure the time that the reports are generated. By default they are generated at 12:30 a.m.; you can modify the time in CallManager Administration ( Service > Service Parameters > select a server > Cisco Serviceability Reporter > RTMT Report Generation Time ).
In addition to these PDF reports, the raw performance data used to generate the reports is available in C:\Program Files\Common Files\Cisco\Longs\RTMTLogger on the Publisher. You can open these files in Microsoft Performance or any other software that supports the CSV file format (for example, Microsoft Excel).
Component Version Information
You can check the versions of installed components by clicking Help > Component Versions . Version information proves useful when you are trying to verify whether you have the latest version of an installed component.
Learn More About Cisco CallManager Serviceability
Detailed information about CallManager Serviceability is available at the following location:
Real-Time Monitoring Tool (RTMT)
The Real-Time Monitoring Tool (RTMT) is a client plug in that enables you to view real-time serviceability information for a CallManager cluster from a remote PC. RTMT provides real-time clusterwide system monitoring, performance monitoring, and device monitoring. You can monitor CallManager and Windows 2000 performance objects and counters in chart view or table view using the Performance tab in RTMT. You can also set thresholds and generate alerts that can be sent to you as e-mail or pop-up messages. Alerts can be configured for CallManager on a per-node basis, or on phones, gateways, ports, Cisco TFTP, and much more.
RTMT includes several
Figure 6-10. RTMT Summary Screen
RTMT has two main tabs on the bottom left:
RTMT View Tab
The View tab has seven categories:
Each category includes one or more
The Summary category only has one
The Server category has three subcategories that provide information about server resources such as CPU and memory. The three subcategories are as
The CallProcess category has four subcategories that provide call processing statistics such as number of calls active on the system. The four subcategories are as follows:
The Service category has three subcategories that provide information about TFTP and directory services as well as heartbeat information for various services. The three subcategories are as follows:
The Device category has two subcategories that provide information on registered devices and enable you to search for specific devices on the cluster. The two subcategories are as follows:
The CTI category has two subcategories that provide information on CTI devices such as number of open devices and a CTI search similar to the device search on the Device category. The two subcategories are as follows:
The Performance category has only a single subcategory, called Performance. This category provides access to the same counters available in Microsoft Performance and enables you to configure alerts based on those counters.
Figure 6-11 shows various counters in table view in the RTMT window. RTMT provides the current, minimum, maximum, and average value for each counter. You can see descriptions of every counter by right-clicking the counter and selecting Counter Description .
Figure 6-11. Monitoring Performance Counters in RTMT
With RTMT, you can select and monitor real-time performance counters that you specify in your CallManager cluster. You can configure multiple tabs under the Performance category. You can configure the view for each tab to be either graph or table format. The graph format provides 6 panes per category,
Note that CallManager-related statistics must be enabled for RTMT to collect data. These statistics are enabled by default in CallManager and can be turned on or off in the Service Parameters Configuration screen in CallManager Administration ( System > Service Parameters > select a server > Cisco CallManager > StatisticsEnable parameter set to True to enable or False to disable statistics ).
RTMT Alert Tab
The Alert tab in RTMT has only one category (Alert) with one subcategory (Alert Central). Alert Central provides two tables:
Figure 6-12 shows the Alert Central window in RTMT.
Figure 6-12. Alert Central in RTMT
The alert status table shows the name of each alert, whether it is enabled, whether the alert is currently active, the action to take if the alert is triggered, and the date and time the alert was last triggered.
If the current value for the
You can configure various alert actions in RTMT, which will send an e-mail of the alert to one or more configured e-mail addresses. You can set different actions for each alert if you want a different set of people to be notified based on the type of alert.
In addition to the preconfigured alerts, you can create an alert based on any performance counter. For example, if you want to be notified any time a call is rejected because a location has no more available bandwidth to allow that call, you can set an alert for the LocationOutOfResources counter in the Cisco CallManager performance object. To do this, add the counter to either a graph or table view in the Performance category of RTMT, and then right-click the counter and choose Alert/Threshold... . You can then configure the level of the alert, a note that you want added to the e-mail any time the alert is sent, and the criteria under which the alert should trigger. The trigger criteria can be any of the following:
You can also choose to trigger the alert immediately or only when the threshold is exceeded for a certain number of seconds. You can also define during what times of the day the alert can be triggered.
Learn More About Real-Time Monitoring Tool
Detailed information about Real-Time Monitoring Tool is available at the following location or search Cisco.com for "Real-Time Monitoring Tool":
Microsoft Performance is a Windows 2000 administrative tool that monitors and logs resource counters from CallManager nodes in the network. Performance shows CallManager-specific status information and Windows 2000 system information in real time. The CallManager system gathers statistical information about the Cisco IP Telephony deployment and feeds it into Performance by way of objects and counters. For example, you can monitor the number of calls in progress at any time, or the number of calls currently passing through a specific Cisco gateway. Data for non-CallManager-specific objects and counters are gathered by the respective services or the operating system itself. For example, the World Wide Web Publishing service feeds information to Performance about the various web pages
Performance can collect data from multiple CallManager servers at once and compile it into a single log file. You can view the logged information in Performance and then export it into tab-separated value (TSV) or CSV file format that can you can view with most spreadsheet applications. Performance enables you to view statistical data in graphical, histogram, and report form. You can access Performance by clicking Start > Programs > Administrative Tools > Performance .
Customizing Microsoft Performance
Like the Real-Time Monitoring Tool, you can use Performance to monitor various real-time conditions in a CallManager system. For example, you can discover the number of calls in progress on a particular CallManager node at any time, or the number of calls currently being attempted in a CallManager cluster. This information is useful for capacity planning, network planning and design, load balancing, and troubleshooting, among other uses.
Each object includes counters that keep track of statistics such as the number of registered MGCP gateways or the number of registered hardware phones. These counters define current conditions within groups of related information. Each
Just as with RTMT, ensure that statistics are enabled in CallManager Administration for Performance to collect data. Statistics are enabled by default. You can stop CallManager from sending data to Performance and the Real-Time Monitoring Tool by setting the Statistics Enabled parameter to False in the Service Parameters Configuration page in CallManager Administration ( System > Service Parameters > select a server > Cisco CallManager ).
Learn More About Microsoft Performance
Detailed information about Performance is available in Microsoft Windows 2000 documentation.
Trace Collection Tool
The Trace Collection Tool (TCT) is a client plug-in ( Application > Install Plugins ) that you can install on a Windows PC to facilitate collecting trace data from a CallManager cluster. Before you can use TCT you must enable traces as described in the section "Cisco CallManager Serviceability."
You can collect traces for one or more services and one or more nodes of the cluster.
TCT collects traces for the following services:
TCT also enables you to collect traces for the following CallManager applications:
Finally, TCT enables you to collect the following system traces:
Figure 6-13 shows the trace selection screen. TCT enables you to collect all traces for the services selected or only those that fall within a specific timeframe. TCT can also
Figure 6-13. Trace Collection Tool
Time-based collection can prove useful for diagnosing a problem where you know approximately the time of the problem. For example, if a user complains that a call was cut off, you can collect traces for the time period in which the dropped call occurred (say,
Learn More About the Trace Collection Tool
Detailed information about the Trace Collection Tool is available at the following location or search Cisco.com for "Trace Collection Tool":
Microsoft Event Viewer helps identify problems at the system level. For example, a group of users cannot make calls in a system with one gateway. You can use Event Viewer to look for events about the gateway (such as registration or unregistration events) to pinpoint the problem. Event Viewer starts automatically when Windows 2000 is started and records events in three kinds of logs:
Event Viewer displays the following types of events:
By default, CallManager alarms are sent to the Event Viewer at the Error level. You can change the alarm event level in the Alarm Configuration screen in CallManager Serviceability ( Alarm > Configuration > select a server > select a service ). Figure 6-14 shows the details of an information message that indicates that the gateway has registered with CallManager.
Figure 6-14. Event Viewer Information Message
You can access Event Viewer by clicking Start > Programs > Administrative Tools > Event Viewer .
Learn More About Event Viewer
Detailed information about Event Viewer is available in Microsoft Windows 2000 documentation.
Terminal Services Client
Terminal Services Client, part of Windows 2000 Server, enables you to export a remote desktop to your PC. It is particularly useful because it allows you access to the remote PC as though you were local to the machine. You must know an
Installing and Accessing Terminal Services Client
Installing the Terminal Services client is possible using a variety of
Windows XP comes bundled with the RDC client and is available from the Start menu ( Programs > Accessories > Communications ). If you are not running Windows XP, you can obtain the RDC client from Microsoft.com for Windows 2000 or Mac OS X.
If you would like to install the older Terminal Services client instead of the RDC client or do not have Internet access, you must first create floppy disks containing the application. From the CallManager server, click Start > Programs > Administrative Tools > Terminal Services Client Creator . Use the Terminal Services Client Creator to create floppy disks; then install Terminal Services on any PC using the floppy disks. When installed on the PC, you can use Terminal Services by clicking Start > Programs > Terminal Services Client and designating the IP address or host name of the remote server you want to access.
You must have the Terminal Services service started on the CallManager node to which you want to connect.
Cisco installs Terminal Services so that administrators and the Cisco Technical Assistance Center (TAC) can perform remote administration and troubleshooting tasks. Cisco does not support upgrades through Terminal Services. You can use VNC Viewer to perform remote upgrades.
Virtual Computer Networking (VNC) Viewer
Virtual Network Computing (VNC) Viewer is a remote display system that enables you to view a remote desktop environment, similar to Terminal Services. VNC allows you to use one computer to drive actions on a target computer, but
You can access the VNC application and documentation files on the operating system (OS) version 2000.2.2 and later installation disc or download. If you're running an older version of the OS, run the OS upgrade for version 2000.2.2 or later to gain access to the VNC files. OS upgrades are available at the following link (requires Cisco.com login):
Using VNC can expose you to a security risk. Review the "Security Best Practices" section in the Cisco-produced document for installing VNC, which is available on the OS 2000 version 2.2 and later installation disc or at the download link previously shown. Use a complex
CiscoWorks IP Telephony Environment Monitor
CiscoWorks is the network management system (NMS) of choice for all Cisco devices, including the CallManager system. CiscoWorks IP Telephony Environment Monitor (ITEM) adds additional functionality to the base CiscoWorks package, which provides the capability to continuously evaluate and report the operational health of your Cisco IP Telephony implementation and provides some diagnostic tools to help troubleshoot problems that occur in the IP Telephony network. ITEM provides specialized operations and security tools beneficial to large and small IP telephony
Using CiscoWorks, you can configure and produce reports on log messages collected from CallManager nodes and other IP telephony devices. CiscoWorks provides a common system log for applications in the multiple-host and multiple-platform Cisco IP Communications environment. ITEM uses SNMP and CallManager's AXL SOAP interface to provide additional information on each device from which the log messages originate.
Each time a device is added to the ITEM device inventory, a new database entry is created. After the device is added to the list, ITEM gathers some device information over SNMP. You can read and use this information for system maintenance and problem solving.
ITEM includes five components.
In addition to ITEM, the CiscoWorks family of web-based products supports maintenance of Cisco enterprise networks and devices. The products include Resource Management Essentials and Campus Manager, which provide syslog analysis, topology services, path analysis, user tracking, fault management, and other network management services.
System Log Management
The syslog analysis tools are Syslog Collector and Syslog Analyzer. They are
Using the reporting and managing capabilities of these tools, you can monitor and manage a wide range of events and error messages concurrently on each CallManager node and other Cisco devices.
Cisco Syslog Collector
Syslog Collector gathers log messages from a CallManager cluster or node at any network installation. The service collects a wide range of significant event messages that reflect system status. After validating the events or error messages collected, Syslog Collector
Cisco Syslog Analyzer
Syslog Analyzer, which resides on a CiscoWorks server, receives the messages collected from multiple applications by the Syslog Collector. When a collection of data is received, the Syslog Analyzer parses and stores the results in the CiscoWorks database. This interface enables you to access and manage whatever data is collected from the system's managed devices.
With Syslog Analyzer, you can examine the event log reports from each Cisco CallManager system, including the description and recommended actions for each log message. In addition to a cluster of Cisco CallManagers, a network installation can also have some voice equipment, routers, gateways, and other devices generating log messages. After you have set up your system, you can access all of this information through one server.
Learn More About CiscoWorks and ITEM
You can find detailed information about CiscoWorks at the following locations:
Management Information Base
is a structured set of data
SNMP allows CallManager to be managed with standard network management applications, such as CiscoWorks or HP OpenView. Windows 2000 Server provides an extensible SNMP agent that can be installed and run as a service. Cisco extension
HP Insight Agent
For CallManager nodes that run on HP servers, the network management application can get system information from the HP Insight Agent MIB. No trap is provided in this MIB, so the management application needs to poll the information that it is interested in periodically and generate its own trap when it reaches a certain threshold value. You can obtain platform-specific information such as hard disk array and
IBM Director Agent
For CallManager nodes that run on IBM servers, the network management application can get system information from the IBM Director Agents. The IBM Director Agents are similar to the HP Insight Agent in that they provide platform-specific information. The IBM Director Agent can be used by ITEM to monitor the IP telephony server platform or can be used with IBM Director Server and Console to monitor the platform.
CCM MIB Extension Agent
SNMP objects and traps are defined in CISCO-CCM-MIB. The CCM MIB extension agent implements the Cisco CallManager MIB. This MIB exports the data in the CallManager database and in other data sources. A
CDP MIB Extension Agent
The CDP MIB extension agent implements all of the variables related to the tell side of Cisco Discovery Protocol (CDP). The MIB variables implemented are cdpInterfaceTable, cdpGlobalRun, cdpGlobalMessageInterval, and cdpGlobalDeviceId. This is the minimum CDP SNMP support that network management applications such as CiscoWorks need to discover the CallManager server. The variable cdpGlobalDeviceId is of type DisplayString (meaning an alphanumeric string) and will return the same value that CDP reports in its CDP advertisement messages.
Updating the CISCO-CCM-MIB Information
The Cisco RIS Data Collector is primarily responsible for updating the information used by the SNMP agents, and it buffers the CISCO-CCM-MIB information that the agents process. At startup, the Cisco RIS Data Collector updates all of the relevant information by periodically fetching data from CallManager or the CallManager database. This updated information is based on interaction with CallManager and other CallManager-associated services.
Updating the CISCO-CDP-MIB Information
At startup, the CDP SNMP extension agent
Downloading the Latest MIBs
New features and bug fixes cause the CISCO-CCM-MIB to be routinely updated. You can download the latest MIB from the following location:
This site provides detailed information about hundreds of MIBs, including CISCO-CDP-MIB and other MIBs that may be of interest to you.
Cisco Discovery Protocol (CDP)
CDP allows CallManager to advertise itself to other Cisco devices on the network by sending periodic messages to a well-known Multicast address monitored by other neighbor devices. Network operators and analysts use this information for configuration monitoring, topology discovery, and fault diagnosis purposes.
With CDP support, CallManager periodically sends out CDP messages or protocol data units (PDU) on the active physical interfaces. These messages contain CallManager information such as the device ID, interface name, system capabilities, and so on. Any Cisco devices with CDP support can discover CallManager by listening to these periodic messages.
Voice Log Translator (VLT)
CallManager provides a large amount of trace data to help you diagnose problems; however, analyzing trace logs can prove difficult for those unfamiliar with the formatting of the messages. In addition, much of the data in the traces appears in a numeric or hexadecimal format that you must decode to understand what is in the trace. For example, if a call going through a PRI gateway is disconnected, a cause code is sent indicating the reason for the disconnect, but the value in the CCM trace appears in hex, such as 0x8090. You need a way to convert this value into
VLT enables you to open a CCM (SDI) trace file and decode its contents. Figure 6-15 shows the output of the VLT tool. You can see that it is decoding the Q.931 Disconnect cause code of 0x8090 to "Normal call clearing."
Figure 6-15. Voice Log Translator Tool
VLT supports the decoding of the following protocols/interfaces:
In addition to decoding these protocols/interfaces, VLT also enables you to perform advanced searching and filtering of trace data to help you find the root cause of a problem. For example, if you know the called party number for a call that failed, you can do a search for that phone number and VLT will display only those messages that contain that phone number. You can then tell VLT to filter by the call reference for that call to see all the messages related to the call.
VLT will work only with text-based CCM (SDI) trace files and will not work with XML traces, which is another reason why Cisco recommends using the default text-based traces. VLT is available at the following URL or search Cisco.com for "Cisco Voice Tool":
Monitoring Tools. (cisco call manager software)