Monitoring Tools. (cisco call manager software)


Monitoring Tools

This section describes the tools that you can use to monitor CallManager and your network:

  • Cisco CallManager Serviceability

  • Real-Time Monitoring Tool (RTMT)

  • Microsoft Performance

  • Trace Collection Tool (TCT)

  • Event Viewer

  • Terminal Services Client

  • Virtual Network Computing (VNC) Viewer

  • CiscoWorks IP Telephony Environment Monitor

  • SNMP MIBs

  • Cisco Discovery Protocol (CDP)

  • Voice Log Translator (VLT)

Cisco CallManager Serviceability

CallManager Serviceability provides alarms, traces, component version information, service activation and control, Quality Reporting Tool reports, and performance reports of Cisco IP Telephony components in a CallManager cluster. CallManager Serviceability is a web-based application installed with CallManager and accessible from CallManager Administration by clicking Application > Cisco CallManager Serviceability . You can access CallManager Serviceability by browsing to the CallManager server by IP address or Domain Name System (DNS) host name , if applicable .

CallManager Serviceability, shown in Figure 6-5, provides the following features:

  • Alarm configuration and detailed alarm definitions

  • Trace configuration, analysis, and collection

  • Component version information

  • Service Activation for adding and removing services

  • Control center for starting and stopping services

  • QRT Viewer for viewing IP phone problem reports

  • Serviceability Reports Archive for viewing archived serviceability data

Figure 6-5. Example of a Screen in CallManager Serviceability


Alarm Configuration

The CallManager system generates alarms to notify you of various events. The alarms are categorized into different levels, as described in Table 6-1. As of CallManager release 4.1, only error levels Debug and Error trigger alarms. You can configure the system to have alarms forwarded to selected monitors, and for each monitor, you can also choose to capture alarms of certain levels. These settings are configured in the Alarm Configuration screen in CallManager Serviceability for each service (such as Cisco CallManager, Cisco TFTP, and so on). Alarm Configuration also enables you to apply a given configuration to all nodes in the CallManager cluster. The monitors to which alarms can be forwarded include the following:

  • SDI trace log (also known as CCM trace) - Display the alarms in the CCM/SDI trace files.

  • SDL trace log (Cisco CallManager alarms only) - Display the alarms in the SDL trace files.

  • Event Viewer - View the alarms in Event Viewer; see the section "Event Viewer" later in this chapter for more information about viewing CallManager alarms in Event Viewer.

  • Syslog - Specify the IP address of a syslog server to receive the messages.

Table 6-1. Alarm Event Levels

Level

Description

Emergency

The system is unusable.

Alert

Indicates a condition that warrants immediate action.

Critical

Indicates a critical condition.

Error

Indicates an error such as an unregistered device or a CallManager failure.

Warning

Indicates a warning condition.

Notice

Indicates a normal but significant condition.

Informational

Provides information messages only.

Debug

Provides detailed messages for use in debugging by Cisco engineers .


Alarm Definitions

You can view detailed alarm descriptions in CallManager Serviceability. Click Alarm > Definitions to view these definitions. A simple search dialog box is provided that allows you to search for and view alarms, their descriptions, and recommended actions. After a search has returned a list of alarms, you can click any part of the alarm description to see detailed information about that alarm. The detailed information also includes an explanation of any reason codes that are given as part of the alarm. For example, an alarm generated by a transient device will have a reason code that explains why the device is considered a transient device, which can help in diagnosing and solving problems in the CallManager system.

Figure 6-6 shows the Alarm Detail screen after a specific alarm (in this case, SDLLinkOOS) has been clicked from the Alarm Message Definitions screen. You can view detailed alarm information for any alarm by clicking it from the Alarm Message Definitions screen.

Figure 6-6. Alarm Details Window


Tracing

Traces are diagnostic tools that can help you determine the cause of a problem. Running a trace can generate information you might need to identify or isolate the source or symptoms of a problem. The path to problem resolution becomes much simpler after you can point to the cause of the problem.

There are two kinds of traces:

  • SDI trace, which is more commonly known as CCM trace

  • SDL trace

Note

To generate good, usable trace information, all clocks should match on all CallManager- related devices. CallManager automatically installs the Network Time Protocol Daemon (xNTPD) for time synchronization. xNTPD provides a consistent time for all devices that poll it. This results in trace data that accurately reflects a single time across the system.


SDI Traces

SDI traces (also known as CCM traces) are useful for diagnosing most problems you might encounter with the CallManager system. SDI traces log different types of runtime events related to CallManager -including device names , their IP addresses, alarms, and other general information -to help you determine the origin of the problem. You can specify a set of devices for tracing so that the trace log contains only events that originate from the selected devices. The Cluster ID and nodeID appear in the trace files to help you determine which trace files belong to which node of the cluster.

For example, a user reports that a dialed number results in reorder tone. You can turn on SDI trace and trace only the phone and gateway involved to learn why CallManager cannot connect the user 's call. This could show a problem with a conflicting route pattern that might be preventing the dialed number from being routed. In another example, a user reports that calls are being dropped. You can turn on SDI trace on the gateways to help identify the reason that the call is dropping and determine whether the problem lies within the CallManager system or in the Public Switched Telephone Network (PSTN).

You can choose to log SDI traces in XML format or in standard text-based format. When XML trace formatting is selected, the resulting trace logs can be used with Trace Analysis feature; however, Cisco recommends leaving trace levels set to text-based format for readability reasons.

Note

SDI tracing is enabled by default; however, the default logging level is Error, which provides limited debugging information beyond critical system failures.


SDI Trace Output

SDI traces generate files (for example, ccm000000000.txt) that contain traces of CallManager activities. These traces provide information about the CallManager initialization process, registration process, call flow, digit analysis, and related devices, such as Cisco IP Phones, gateways, gatekeepers, and more. This information can help you isolate problems when troubleshooting CallManager.

The trace files are stored in the following default location:

C:\Program Files\Cisco\Trace

Note

Cisco recommends changing the default file location for trace files to the F: drive on dual-processor CallManager servers equipped with extra hard drives for trace collection.


If a trace is enabled, a new trace file is started each time CallManager restarts or when the designated number of lines or minutes has been reached. Although SDI traces do consume some system resources, leaving SDI traces enabled during routine system operation does not typically impact performance unless the system is heavily loaded.

SDL Traces

The Cisco Technical Assistance Center (TAC) and Cisco engineers use SDL traces to diagnose difficult problems. The only time you should change the SDL trace configuration is when directed to do so by TAC. SDL trace logs state transitions only for CallManager and Cisco CTI Manager. Cisco engineers use SDL traces to find the cause of an error. You are not expected to understand the information contained in an SDL trace. However, while working with TAC, you might be asked to change the SDL trace configuration and provide the resulting trace files to TAC.

Note

Before gathering any SDL traces in CallManager Serviceability, you must enable SDL tracing in CallManager Administration. The Cisco TAC representative requesting the trace can advise you on how to enable SDL tracing.


Trace Configuration

Trace configuration allows you to specify the criteria for SDI tracing. Click Trace > Configuration to select the server you want to trace, and then select the service on that server, such as Cisco CallManager, the Database Layer, CTI Manager, and so on. Figure 6-7 shows the Trace Configuration screen with the default values selected. You can custom configure the trace by selecting the level of trace, the trace fields, and device names (if applicable). Table 6-2 provides a list of the debug trace level settings.

Figure 6-7. SDI Trace Configuration Page


Table 6-2. Debug Trace Levels

Level

Description

Error

Use this level for all traces generated in abnormal paths, such as coding errors or other errors that normally should not occur.

Special

Use this level for all informational, nonrepetitive messages, such as process startup messages, registration messages, and so on. All system and device initialization traces are at this level.

State Transition

Use this level to trace call processing events or normal events traced for the subsystem (traces for signaling layers ).

Significant

Use this level to trace media layer events.

Arbitrary

Use this level to generate low-level debug traces. This level is best suited for a testing setup or for debugging difficult problems in a subsystem.

Detailed

This level is used for detailed debug information or highly repetitive messages that are primarily used for debugging, including KeepAlives and responses.


CallManager Serviceability provides the option of device-based filtering, which proves useful when a certain problem is known to be occurring on a specific device and you want to only view trace information related to that device. You select device name-based tracing in the Trace Configuration screen. Selecting the devices and then running the trace returns results for any events involving the selected devices.

By default, trace results are saved in a file. You can choose to save traces in TXT format, where up to 10,000 lines can be saved and the file can be viewed in any text editor, or you can choose to save the results in XML format; however, XML-formatted trace files are limited to fewer than 2000 lines.

After a trace starts, it continues until you turn it off. When tracing reaches its limit in the trace files, it begins to overwrite trace data, starting with the earliest trace files/lines. To view an XML trace, you can use Trace Analysis (see the following section, "Trace Analysis"). To view a text-based trace, open the trace in a text editor. Figure 6-7 shows the SDI trace configuration page in CallManager Serviceability.

Troubleshooting Trace Settings

The Troubleshooting Trace Settings page enables you to easily configure trace settings for troubleshooting most services without having to configure each service one at a time. The Troubleshooting Trace Settings page uses trace levels based on Cisco TAC recommendations. Figure 6-8 shows the Troubleshooting Trace Settings page in CallManager Serviceability.

Figure 6-8. Troubleshooting Trace Settings Page


To apply the TAC-recommended trace settings for a service, select the check box corresponding to the service for which you want to enable traces. You can choose Select all Nodes for a Service to enable traces for a particular service on all nodes in the cluster or use Check all Services for a Node to select all the services on a particular node. When Troubleshooting Trace Settings are enabled, you cannot manually change trace settings until you disable Troubleshooting Trace Settings.

Trace Analysis

Trace Analysis allows you to perform post-filtering on an XML trace file so that only the relevant information displays. Because trace files can be large and encompass so much information, the ability to reduce and sort that information can be useful. You must know which trace file you want to analyze to use Trace Analysis. Click Trace > Analysis to choose a trace file and specify the selection criteria and the fields you want displayed in the Trace Analysis results.

With Trace Analysis, you can specify collection criteria such as the following:

  • Cisco CallManager host

  • Device name

  • IP address

  • Trace type

  • MGCP endpoint

You can filter the trace so that only the pertinent information displays. The following information can be displayed or filtered out of the trace:

  • Cluster

  • Date and time

  • CallManager node

  • Trace type

  • IP address

  • Correlation tag

  • Application name

  • Information

  • Device name

In practice, administrators who are proficient in reading CallManager trace files find that reading and filtering through a raw text-based trace file is much quicker and easier than using Trace Analysis. Also, Trace Analysis is very slow for large amounts of trace data. Future CallManager releases might see XML tracing and the Trace Analysis features removed.

Performance Impact

Trace Analysis runs as a low-priority task so that it does not disrupt higher-priority CallManager functions. However, you should us it judiciously because it can be resource- intensive . Memory impact on the CallManager system is minimal, as long as only a few concurrent users run analysis at any given time. If possible, run Trace Analysis only when the CallManager system is not busy. Because traces are being collected and merged into an output file, the tool continuously accesses the disk. Also, be certain the CallManager system has enough disk space for the temporary output files. Currently, no user interface is available to clean up these temporary files, but the tool does automatically recycle them.

Service Activation

Before a service can be used on a CallManager node in a cluster, it must be activated, which you can do on the Service Activation page in CallManager Serviceability ( Tools > Service Activation ). Activating or deactivating a service notifies members of a cluster that the service state has changed by adding a record into the database, and also modifies the startup type of the Windows service from Disabled to Automatic or vice versa.

Caution

You should never change the service startup type for any Cisco service directly from the Windows Services Administrative Tool. Always use Service Activation to change the startup type.


Control Center

CallManager Serviceability enables you to start and stop services by clicking Tools > Control Center . The Control Center also indicates whether the services on the CallManager nodes are currently running or stopped .

QRT Viewer

The Quality Reporting Tool (QRT) service enables you to add a softkey that allows end users to report problems with their IP phone. To view reports submitted by QRT, you must use the QRT Viewer in CallManager Serviceability ( Tools > QRT Viewer ).

QRT Viewer enables you to select a server in the cluster and a date/time range. After clicking Get Logs , a page showing all QRT reports submitted during that timeframe displays, including details such as the date and time of the report, the problem being reported, the device from which the report was submitted, and voice quality statistics (packet loss, jitter, and so on) if the problem was reported during a call.

Figure 6-9 shows a sample QRT report indicating that a user was unable to place a call and received a fast busy (reorder) tone. The report shows the date and time the problem occurred, device name, phone number, and IP address.

Figure 6-9. QRT Report


Serviceability Reports Archive

CallManager Serviceability monitors various aspects of a CallManager cluster and generates nightly reports in PDF format. You can access these reports from the Serviceability Reports Archive in CallManager Serviceability ( Tools > Serviceability Reports Archive ).

Each day, five reports are generated:

  • Alert report - Shows the number of alerts by severity, number of alerts per server in the cluster, and the top 10 alerts.

  • Call activities report - Shows an hour-by- hour graph of calls attempted and calls completed for the day and a breakdown of calls per gateway type. This information is helpful for determining your peak traffic periods.

  • Device statistics report - Provides hourly graphs showing the number of registered phones, gateways, and trunks.

  • Server statistics report - Shows CPU, memory, and hard disk usage statistics in an hourly graph.

  • Service statistics report - Shows the number of CTI devices registered and the number of TFTP requests and errors in an hourly graph.

The Serviceability Reports Archive data is presented in a graphical format that enables you to view trends or any sudden events that might have occurred on a given day. For example, you might look at a report and see that CPU utilization is always high between 3 and 4 a.m., but there is no significant call activity during this time. You can then use this information to investigate what might be causing the CPU to be utilized during that time period.

By default, the Serviceability Reports Archive stores the last seven days of data. You can increase this in CallManager Administration ( Service > Service Parameters > select a server > Cisco Serviceability Reporter > RTMT Report Deletion Age ). You can also configure the time that the reports are generated. By default they are generated at 12:30 a.m.; you can modify the time in CallManager Administration ( Service > Service Parameters > select a server > Cisco Serviceability Reporter > RTMT Report Generation Time ).

In addition to these PDF reports, the raw performance data used to generate the reports is available in C:\Program Files\Common Files\Cisco\Longs\RTMTLogger on the Publisher. You can open these files in Microsoft Performance or any other software that supports the CSV file format (for example, Microsoft Excel).

Component Version Information

You can check the versions of installed components by clicking Help > Component Versions . Version information proves useful when you are trying to verify whether you have the latest version of an installed component.

Learn More About Cisco CallManager Serviceability

Detailed information about CallManager Serviceability is available at the following location:

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/4_1/service/serv412/index.htm

Real-Time Monitoring Tool (RTMT)

The Real-Time Monitoring Tool (RTMT) is a client plug in that enables you to view real-time serviceability information for a CallManager cluster from a remote PC. RTMT provides real-time clusterwide system monitoring, performance monitoring, and device monitoring. You can monitor CallManager and Windows 2000 performance objects and counters in chart view or table view using the Performance tab in RTMT. You can also set thresholds and generate alerts that can be sent to you as e-mail or pop-up messages. Alerts can be configured for CallManager on a per-node basis, or on phones, gateways, ports, Cisco TFTP, and much more.

RTMT includes several charts and tables that provide an overview of the CallManager cluster. Figure 6-10 shows the summary page in RTMT, which displays the memory utilization, CPU utilization, number of registered IP phones, calls in progress, and active gateway ports on a single screen. Note that each server in the cluster is represented by its own line on the chart.

Figure 6-10. RTMT Summary Screen


RTMT has two main tabs on the bottom left:

  • View - Enables you to view real-time data

  • Alert - Lists all configured alerts and their status

RTMT View Tab

The View tab has seven categories:

  • Summary

  • Server

  • CallProcess

  • Service

  • Device

  • CTI

  • Performance (called "PerfMon" in some versions)

Each category includes one or more subcategories , each of which has graphs and tables similar to the Summary category shown in Figure 6-10. The sections that follow provide additional information about the seven View tab categories.

Summary

The Summary category only has one subcategory , also named Summary, which shows memory utilization, CPU utilization, number of registered IP phones, calls in progress, and active gateway ports and channels.

Server

The Server category has three subcategories that provide information about server resources such as CPU and memory. The three subcategories are as follows :

  • CPU&Memory - Shows memory and CPU utilization graphs similar to the Summary category. Also lists all processes on each node of the CallManager cluster and process-specific information such as CPU utilization, private memory bytes in use, and virtual memory bytes in use.

  • Disk Usage - Shows total and available hard disk space on each drive of each server in the cluster.

  • Critical Services - Lists all services critical to the operation of CallManager for each node in the cluster along with a status of up (working), down (not working), or not activated. This table also shows how long each of the services has been up, enabling you to determine whether a service has restarted unexpectedly.

CallProcess

The CallProcess category has four subcategories that provide call processing statistics such as number of calls active on the system. The four subcategories are as follows:

  • Call Activity - Shows the number of calls completed and calls attempted in the last sampling interval. Also shows the number of calls in progress.

  • Gateway Activity - Allows you to view statistics for various gateway types: MGCP PRI, MGCP FXS, MGCP FXO, MGCP T1, and H.323. This page provides statistics such as Channels Active and Calls Completed for each gateway type.

  • Trunk Activity - Shows the number of calls in progress and calls completed for H.323 and SIP trunks.

  • SDL Queue - Displays the number of signals in each of the four SDL queues: High, Normal, Low, and Lowest. Also displays the number of signals processed during the last sampling interval. If signals are being processed at a rate slower than they are put into the queue, you will see the queue size increase. This can cause performance degradation on the CallManager system and is usually due to some kind of system resource limitation such as CPU, memory, or disk I/O.

Service

The Service category has three subcategories that provide information about TFTP and directory services as well as heartbeat information for various services. The three subcategories are as follows:

  • Cisco TFTP - Monitors the operation of all TFTP services in the cluster by displaying the total number of TFTP requests, the number of requests where the file was not found, and the number of TFTP requests that were aborted.

  • Directory Server - Shows the operational and replication status of the directory services on all nodes in the CallManager cluster.

  • Heartbeat - The CallManager, TFTP, and Telephony Call Dispatcher (TCD) services all have a heartbeat counter that increments periodically to indicate the service is still processing calls. This page shows the heartbeat status for CallManager, TFTP, and TCD. If the heartbeat of a service stops, this category indicates that the service is in a state where it is unable to process calls.

Device

The Device category has two subcategories that provide information on registered devices and enable you to search for specific devices on the cluster. The two subcategories are as follows:

  • Device Summary - Shows a graph of the number of registered phones, gateways, and media resources and provides a breakdown by gateway type and media resource type in table format.

  • Device Search - Provides monitoring information about devices in the CallManager cluster, their IP addresses, their real-time status (such as whether they are successfully registered to CallManager and to which CallManager node), and other useful device information. For devices that support an HTTP server, such as phones and some gateways, you can right-click the specific device information to open an HTTP connection to the selected device. You can then browse information stored locally on the device, such as network configuration and statistics information. You can also launch related screens in CallManager Administration by right-clicking a device type (phone, gateway, voice mail, and so on).

CTI

The CTI category has two subcategories that provide information on CTI devices such as number of open devices and a CTI search similar to the device search on the Device category. The two subcategories are as follows:

  • CtiManager - Shows the number open CTI devices, the number of open CTI lines, and the number of CTI connections to each node in the CallManager cluster.

  • CTI Search - Enables you to search for CTI devices and applications registered to the CallManager cluster. This can prove useful for monitoring whether a CTI application is registered properly with CallManager. The CTI Search subcategory also tells you what user ID an application is using to authenticate with CallManager. If the login is failing, the CTI Search subcategory shows the reason for the failure.

Performance

The Performance category has only a single subcategory, called Performance. This category provides access to the same counters available in Microsoft Performance and enables you to configure alerts based on those counters.

Figure 6-11 shows various counters in table view in the RTMT window. RTMT provides the current, minimum, maximum, and average value for each counter. You can see descriptions of every counter by right-clicking the counter and selecting Counter Description .

Figure 6-11. Monitoring Performance Counters in RTMT


With RTMT, you can select and monitor real-time performance counters that you specify in your CallManager cluster. You can configure multiple tabs under the Performance category. You can configure the view for each tab to be either graph or table format. The graph format provides 6 panes per category, accommodating up to 18 counters. You can add counters to the panes by double-clicking them from the list of counters. You can also drag and drop counters to layer up to three counters in a single pane. You can add new categories to monitor additional counters. For example, you might have eight T1 lines and are capacity-planning to determine whether to acquire additional T1 lines. You can monitor the counters associated with T1 gateways to determine when usage is approaching the limit. Another example is for load balancing. You can monitor the same counters from two different CallManager nodes in the cluster and put them on the same chart to see whether one CallManager is more heavily loaded than the other. You might decide to change the configuration to balance the load better.

Note that CallManager-related statistics must be enabled for RTMT to collect data. These statistics are enabled by default in CallManager and can be turned on or off in the Service Parameters Configuration screen in CallManager Administration ( System > Service Parameters > select a server > Cisco CallManager > StatisticsEnable parameter set to True to enable or False to disable statistics ).

RTMT Alert Tab

The Alert tab in RTMT has only one category (Alert) with one subcategory (Alert Central). Alert Central provides two tables:

  • A list showing the status of all configured alerts

  • An alert history showing any alerts that have been previously triggered

Figure 6-12 shows the Alert Central window in RTMT.

Figure 6-12. Alert Central in RTMT


The alert status table shows the name of each alert, whether it is enabled, whether the alert is currently active, the action to take if the alert is triggered, and the date and time the alert was last triggered.

If the current value for the monitored counter is outside the configured threshold, the InSafeRange column will show No. Being outside the safe range might trigger an alert if the counter remains outside the configured threshold for a specified amount of time (the duration is user-configurable). You can get additional details about the alert by selecting Alert Event Details from the Alert menu. Figure 6-12 shows the alert details for the MgcpDChannelOutOfService alert. The alert details shows the list of gateways that have the D-channel out of service.

You can configure various alert actions in RTMT, which will send an e-mail of the alert to one or more configured e-mail addresses. You can set different actions for each alert if you want a different set of people to be notified based on the type of alert.

In addition to the preconfigured alerts, you can create an alert based on any performance counter. For example, if you want to be notified any time a call is rejected because a location has no more available bandwidth to allow that call, you can set an alert for the LocationOutOfResources counter in the Cisco CallManager performance object. To do this, add the counter to either a graph or table view in the Performance category of RTMT, and then right-click the counter and choose Alert/Threshold... . You can then configure the level of the alert, a note that you want added to the e-mail any time the alert is sent, and the criteria under which the alert should trigger. The trigger criteria can be any of the following:

  • The value is over or under a specified limit

  • The value has changed by a certain amount or percentage since the last sampling interval

You can also choose to trigger the alert immediately or only when the threshold is exceeded for a certain number of seconds. You can also define during what times of the day the alert can be triggered.

Learn More About Real-Time Monitoring Tool

Detailed information about Real-Time Monitoring Tool is available at the following location or search Cisco.com for "Real-Time Monitoring Tool":

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/4_1/service/serv413/ccmsrva/sartmt.htm

Microsoft Performance

Microsoft Performance is a Windows 2000 administrative tool that monitors and logs resource counters from CallManager nodes in the network. Performance shows CallManager-specific status information and Windows 2000 system information in real time. The CallManager system gathers statistical information about the Cisco IP Telephony deployment and feeds it into Performance by way of objects and counters. For example, you can monitor the number of calls in progress at any time, or the number of calls currently passing through a specific Cisco gateway. Data for non-CallManager-specific objects and counters are gathered by the respective services or the operating system itself. For example, the World Wide Web Publishing service feeds information to Performance about the various web pages served on a CallManager server.

Performance can collect data from multiple CallManager servers at once and compile it into a single log file. You can view the logged information in Performance and then export it into tab-separated value (TSV) or CSV file format that can you can view with most spreadsheet applications. Performance enables you to view statistical data in graphical, histogram, and report form. You can access Performance by clicking Start > Programs > Administrative Tools > Performance .

Customizing Microsoft Performance

Like the Real-Time Monitoring Tool, you can use Performance to monitor various real-time conditions in a CallManager system. For example, you can discover the number of calls in progress on a particular CallManager node at any time, or the number of calls currently being attempted in a CallManager cluster. This information is useful for capacity planning, network planning and design, load balancing, and troubleshooting, among other uses.

Each object includes counters that keep track of statistics such as the number of registered MGCP gateways or the number of registered hardware phones. These counters define current conditions within groups of related information. Each group of related information is called an object; each object contains one or more counters, such as Cisco CallManager or Cisco Software Conference Bridge, and each of these objects can have more than one instance of the object. Objects and counters are automatically added when CallManager or the related component (such as Conference Bridge) is installed. Using objects and counters, you can retrieve detailed, relevant, and timely system information. Performance can also be customized to track Cisco applications such as the IP IVR or Windows 2000 system objects and counters. This additional information can be useful to correlate system events with CallManager events. For example, if you notice a surge in CPU utilization, you might also notice that there is a surge in call volume which would explain the higher-than-expected CPU utilization.

Just as with RTMT, ensure that statistics are enabled in CallManager Administration for Performance to collect data. Statistics are enabled by default. You can stop CallManager from sending data to Performance and the Real-Time Monitoring Tool by setting the Statistics Enabled parameter to False in the Service Parameters Configuration page in CallManager Administration ( System > Service Parameters > select a server > Cisco CallManager ).

Learn More About Microsoft Performance

Detailed information about Performance is available in Microsoft Windows 2000 documentation.

Trace Collection Tool

The Trace Collection Tool (TCT) is a client plug-in ( Application > Install Plugins ) that you can install on a Windows PC to facilitate collecting trace data from a CallManager cluster. Before you can use TCT you must enable traces as described in the section "Cisco CallManager Serviceability."

You can collect traces for one or more services and one or more nodes of the cluster.

TCT collects traces for the following services:

  • Cisco CallManager

  • Cisco CDR Insert

  • Cisco Certificate Authority Proxy Function

  • Cisco CTI Manager

  • Cisco CTL Provider

  • Cisco Database Layer Monitor

  • Cisco Extended Functions

  • Cisco Extension Mobility

  • Cisco IP Manager Assistant

  • Cisco IP Voice Media Streaming App

  • Cisco Messaging Interface

  • Cisco MOH Audio Translator

  • Cisco RIS Data Collector

  • Cisco Telephony Call Dispatcher

  • Cisco TFTP

  • Cisco WebDialer

TCT also enables you to collect traces for the following CallManager applications:

  • BAT

  • CAR

  • Cisco Serviceability Reporter

  • Cisco Tomcat

  • Installation Log Files

  • Multilevel Administration (MLA)

  • Quality Reporting Tool (QRT)

  • Tool for Auto-Registered Phone Support

Finally, TCT enables you to collect the following system traces:

  • Event Viewer logs (application, security, and system)

  • Dr. Watson logs (crash dumps created by the Windows operating system)

  • Internet Information Server (IIS) logs

  • Microsoft SQL Server 2000 logs

  • Directory logs

  • System performance logs

  • ProgLogs (includes system information from CallManager startup)

Figure 6-13 shows the trace selection screen. TCT enables you to collect all traces for the services selected or only those that fall within a specific timeframe. TCT can also optionally compress the collected files into a Zip archive to save hard drive space on the PC where you are collecting the traces.

Figure 6-13. Trace Collection Tool


Time-based collection can prove useful for diagnosing a problem where you know approximately the time of the problem. For example, if a user complains that a call was cut off, you can collect traces for the time period in which the dropped call occurred (say, yesterday morning between 7:45 a.m. and 8:15 a.m.).

Learn More About the Trace Collection Tool

Detailed information about the Trace Collection Tool is available at the following location or search Cisco.com for "Trace Collection Tool":

http://www.cisco.com/univercd/cc/td/doc/product/voice/c_callmg/4_1/service/serv413/ccmsrva/satracec.htm

Event Viewer

Microsoft Event Viewer helps identify problems at the system level. For example, a group of users cannot make calls in a system with one gateway. You can use Event Viewer to look for events about the gateway (such as registration or unregistration events) to pinpoint the problem. Event Viewer starts automatically when Windows 2000 is started and records events in three kinds of logs:

  • Application log - Contains events logged by applications or programs, such as CallManager.

  • System log - Contains events logged by the Windows 2000 system components, such as the failure of a system component.

  • Security log - Contains records of security events such as login failures.

Event Viewer displays the following types of events:

  • Error - A significant problem, such as loss of data or loss of functionality. For example, if a problem occurs with a device that is registered with CallManager, an error event provides device information and error details to help you isolate the problem.

  • Warning - An event that is not necessarily significant but might indicate a possible future problem. For example, warning events can include CallManager services that have stopped or started.

  • Information - An event that describes system information, such as host name or IP address and the version of the database layer in use.

By default, CallManager alarms are sent to the Event Viewer at the Error level. You can change the alarm event level in the Alarm Configuration screen in CallManager Serviceability ( Alarm > Configuration > select a server > select a service ). Figure 6-14 shows the details of an information message that indicates that the gateway has registered with CallManager.

Figure 6-14. Event Viewer Information Message


You can access Event Viewer by clicking Start > Programs > Administrative Tools > Event Viewer .

Learn More About Event Viewer

Detailed information about Event Viewer is available in Microsoft Windows 2000 documentation.

Terminal Services Client

Terminal Services Client, part of Windows 2000 Server, enables you to export a remote desktop to your PC. It is particularly useful because it allows you access to the remote PC as though you were local to the machine. You must know an administrator-level password on the remote server to access it via Terminal Services. After you have accessed a server using the Terminal Services client or Microsoft Remote Desktop client, you can perform tasks such as system administration, viewing log files or event logs, and so on.

Installing and Accessing Terminal Services Client

Installing the Terminal Services client is possible using a variety of methods . Cisco recommends using the latest Microsoft Remote Desktop Connection (RDC) client available from Microsoft.com.

Windows XP comes bundled with the RDC client and is available from the Start menu ( Programs > Accessories > Communications ). If you are not running Windows XP, you can obtain the RDC client from Microsoft.com for Windows 2000 or Mac OS X.

If you would like to install the older Terminal Services client instead of the RDC client or do not have Internet access, you must first create floppy disks containing the application. From the CallManager server, click Start > Programs > Administrative Tools > Terminal Services Client Creator . Use the Terminal Services Client Creator to create floppy disks; then install Terminal Services on any PC using the floppy disks. When installed on the PC, you can use Terminal Services by clicking Start > Programs > Terminal Services Client and designating the IP address or host name of the remote server you want to access.

Note

You must have the Terminal Services service started on the CallManager node to which you want to connect.


Note

Cisco installs Terminal Services so that administrators and the Cisco Technical Assistance Center (TAC) can perform remote administration and troubleshooting tasks. Cisco does not support upgrades through Terminal Services. You can use VNC Viewer to perform remote upgrades.


Virtual Computer Networking (VNC) Viewer

Virtual Network Computing (VNC) Viewer is a remote display system that enables you to view a remote desktop environment, similar to Terminal Services. VNC allows you to use one computer to drive actions on a target computer, but differs from Terminal Services because with VNC, any actions performed by you that occur on the target computer can be seen equally by the local user. Unlike Terminal Services, you can use VNC to install, upgrade, or apply patches to CallManager.

You can access the VNC application and documentation files on the operating system (OS) version 2000.2.2 and later installation disc or download. If you're running an older version of the OS, run the OS upgrade for version 2000.2.2 or later to gain access to the VNC files. OS upgrades are available at the following link (requires Cisco.com login):

http://www.cisco.com/cgi-bin/tablebuild.pl/cmva-3des

Caution

Using VNC can expose you to a security risk. Review the "Security Best Practices" section in the Cisco-produced document for installing VNC, which is available on the OS 2000 version 2.2 and later installation disc or at the download link previously shown. Use a complex alphanumeric password for VNC. VNC does not have a username/password structure; it uses only a single password, and VNC limits the password to eight characters , so make sure the password you choose is difficult to crack. A good password includes numbers , upper- and lowercase letters , and special characters, and does not use any known word. For example: 123eye67 is not as good a password choice as 4hW9Lv#g.


CiscoWorks IP Telephony Environment Monitor

CiscoWorks is the network management system (NMS) of choice for all Cisco devices, including the CallManager system. CiscoWorks IP Telephony Environment Monitor (ITEM) adds additional functionality to the base CiscoWorks package, which provides the capability to continuously evaluate and report the operational health of your Cisco IP Telephony implementation and provides some diagnostic tools to help troubleshoot problems that occur in the IP Telephony network. ITEM provides specialized operations and security tools beneficial to large and small IP telephony implementations . ITEM is not bundled with CallManager and must be purchased separately.

Using CiscoWorks, you can configure and produce reports on log messages collected from CallManager nodes and other IP telephony devices. CiscoWorks provides a common system log for applications in the multiple-host and multiple-platform Cisco IP Communications environment. ITEM uses SNMP and CallManager's AXL SOAP interface to provide additional information on each device from which the log messages originate.

Each time a device is added to the ITEM device inventory, a new database entry is created. After the device is added to the list, ITEM gathers some device information over SNMP. You can read and use this information for system maintenance and problem solving.

ITEM includes five components.

  • CiscoWorks IP Telephony Monitor (ITM) - Monitors Cisco voice elements in the network to alert operations personnel to potential problems and to help minimize downtime. This component also includes CiscoWorks Common Services, a common foundation for data storage, login, access privileges, and navigation and launch management for all CiscoWorks applications.

  • CiscoWorks IP Phone Information Utility (IPIU) - Provides operational status and implementation details about an individual IP phone. This component also provides security reports that document IP phone "moves, adds, and changes" as well as information about the physical and logical connections of every Cisco IP Phone installed in a given network.

  • CiscoWorks IP Phone Help Desk Utility (IPHDU) - Reports operational status and implementation details about individual IP phones. This component works in conjunction with the IPIU to make read-only access to Cisco IP Phone installation details available to help desk personnel.

  • CiscoWorks ITEM Gateway Statistics Utility (GSU) - Collects performance and behavior statistics about CallManager-controlled and Cisco IOS Software-based IP telephony gateways, which can be processed by third-party software to produce utilization and capacity management reports.

  • CiscoWorks WAN Performance Utility (WPU) - Measures the performance, latency, and availability of multiprotocol IP networks on an end-to-end and hop-by-hop (router-to-router) basis.

In addition to ITEM, the CiscoWorks family of web-based products supports maintenance of Cisco enterprise networks and devices. The products include Resource Management Essentials and Campus Manager, which provide syslog analysis, topology services, path analysis, user tracking, fault management, and other network management services.

System Log Management

The syslog analysis tools are Syslog Collector and Syslog Analyzer. They are offered with CiscoWorks as part of the Resource Management Essentials package. Syslog output from CallManager can alternatively be adapted for use with other NMSs that support the standard syslog format:

  • The Syslog Collector keeps common system logs that record messages reported to the CallManager system.

  • The Syslog Analyzer controls and displays all events so they can easily be read, interpreted, and used for system maintenance and problem solving.

Using the reporting and managing capabilities of these tools, you can monitor and manage a wide range of events and error messages concurrently on each CallManager node and other Cisco devices.

Cisco Syslog Collector

Syslog Collector gathers log messages from a CallManager cluster or node at any network installation. The service collects a wide range of significant event messages that reflect system status. After validating the events or error messages collected, Syslog Collector passes them to the Syslog Analyzer. When this process is complete, you can use Syslog Analyzer to analyze the log messages.

Cisco Syslog Analyzer

Syslog Analyzer, which resides on a CiscoWorks server, receives the messages collected from multiple applications by the Syslog Collector. When a collection of data is received, the Syslog Analyzer parses and stores the results in the CiscoWorks database. This interface enables you to access and manage whatever data is collected from the system's managed devices.

With Syslog Analyzer, you can examine the event log reports from each Cisco CallManager system, including the description and recommended actions for each log message. In addition to a cluster of Cisco CallManagers, a network installation can also have some voice equipment, routers, gateways, and other devices generating log messages. After you have set up your system, you can access all of this information through one server.

Learn More About CiscoWorks and ITEM

You can find detailed information about CiscoWorks at the following locations:

http://www.cisco.com/en/US/products/sw/cscowork/ps2433/index.html

http://www.cisco.com/en/US/customer/products/sw/cscowork/ps2425/index.html

SNMP MIBs

A Management Information Base (MIB) is a structured set of data variables , called objects, in which each variable represents some resource to be managed. Simple Network Management Protocol (SNMP) MIB conceptual tables organize and distribute the information gathered from your IP telephony system.

SNMP allows CallManager to be managed with standard network management applications, such as CiscoWorks or HP OpenView. Windows 2000 Server provides an extensible SNMP agent that can be installed and run as a service. Cisco extension agents (dynamic link library, or DLL) support Cisco MIBs, which provides support for CallManager-specific data.

HP Insight Agent

For CallManager nodes that run on HP servers, the network management application can get system information from the HP Insight Agent MIB. No trap is provided in this MIB, so the management application needs to poll the information that it is interested in periodically and generate its own trap when it reaches a certain threshold value. You can obtain platform-specific information such as hard disk array and power-supply status from the HP Insight Agent. ITEM uses the HP Insight Agent to provide platform-specific alerts for your IP telephony servers.

IBM Director Agent

For CallManager nodes that run on IBM servers, the network management application can get system information from the IBM Director Agents. The IBM Director Agents are similar to the HP Insight Agent in that they provide platform-specific information. The IBM Director Agent can be used by ITEM to monitor the IP telephony server platform or can be used with IBM Director Server and Console to monitor the platform.

CCM MIB Extension Agent

SNMP objects and traps are defined in CISCO-CCM-MIB. The CCM MIB extension agent implements the Cisco CallManager MIB. This MIB exports the data in the CallManager database and in other data sources. A trap is an unsolicited message sent by an agent to a management station in an asynchronous manner. The purpose is to notify the management station of some unusual event. The traps are sent to trap-receiving hosts configured in the Windows 2000 SNMP service. Network management applications such as ITEM can gather data that can be used for fault management and analysis purposes. Although the design of the MIB and trap is tied to CiscoWorks applications, it does not limit you from developing network management applications by using third-party software. The SNMP extension agent for the CISCO-CCM-MIB is packaged as a DLL file and bundled in the CallManager installation.

CDP MIB Extension Agent

The CDP MIB extension agent implements all of the variables related to the tell side of Cisco Discovery Protocol (CDP). The MIB variables implemented are cdpInterfaceTable, cdpGlobalRun, cdpGlobalMessageInterval, and cdpGlobalDeviceId. This is the minimum CDP SNMP support that network management applications such as CiscoWorks need to discover the CallManager server. The variable cdpGlobalDeviceId is of type DisplayString (meaning an alphanumeric string) and will return the same value that CDP reports in its CDP advertisement messages.

Updating the CISCO-CCM-MIB Information

The Cisco RIS Data Collector is primarily responsible for updating the information used by the SNMP agents, and it buffers the CISCO-CCM-MIB information that the agents process. At startup, the Cisco RIS Data Collector updates all of the relevant information by periodically fetching data from CallManager or the CallManager database. This updated information is based on interaction with CallManager and other CallManager-associated services.

Updating the CISCO-CDP-MIB Information

At startup, the CDP SNMP extension agent interacts with the CDP driver, fetching and buffering CDP-related information.

Downloading the Latest MIBs

New features and bug fixes cause the CISCO-CCM-MIB to be routinely updated. You can download the latest MIB from the following location:

ftp.cisco.com/pub/mibs

This site provides detailed information about hundreds of MIBs, including CISCO-CDP-MIB and other MIBs that may be of interest to you.

Cisco Discovery Protocol (CDP)

CDP allows CallManager to advertise itself to other Cisco devices on the network by sending periodic messages to a well-known Multicast address monitored by other neighbor devices. Network operators and analysts use this information for configuration monitoring, topology discovery, and fault diagnosis purposes.

With CDP support, CallManager periodically sends out CDP messages or protocol data units (PDU) on the active physical interfaces. These messages contain CallManager information such as the device ID, interface name, system capabilities, and so on. Any Cisco devices with CDP support can discover CallManager by listening to these periodic messages.

Voice Log Translator (VLT)

CallManager provides a large amount of trace data to help you diagnose problems; however, analyzing trace logs can prove difficult for those unfamiliar with the formatting of the messages. In addition, much of the data in the traces appears in a numeric or hexadecimal format that you must decode to understand what is in the trace. For example, if a call going through a PRI gateway is disconnected, a cause code is sent indicating the reason for the disconnect, but the value in the CCM trace appears in hex, such as 0x8090. You need a way to convert this value into human-readable format -in this case "Normal call clearing." To facilitate this type of trace decoding, use the Voice Log Translator (VLT) tool.

VLT enables you to open a CCM (SDI) trace file and decode its contents. Figure 6-15 shows the output of the VLT tool. You can see that it is decoding the Q.931 Disconnect cause code of 0x8090 to "Normal call clearing."

Figure 6-15. Voice Log Translator Tool


VLT supports the decoding of the following protocols/interfaces:

  • SCCP

  • Q.931

  • H.225

  • H.245

  • MGCP

  • JTAPI

In addition to decoding these protocols/interfaces, VLT also enables you to perform advanced searching and filtering of trace data to help you find the root cause of a problem. For example, if you know the called party number for a call that failed, you can do a search for that phone number and VLT will display only those messages that contain that phone number. You can then tell VLT to filter by the call reference for that call to see all the messages related to the call.

VLT will work only with text-based CCM (SDI) trace files and will not work with XML traces, which is another reason why Cisco recommends using the default text-based traces. VLT is available at the following URL or search Cisco.com for "Cisco Voice Tool":

http://www.cisco.com/cgi-bin/tablebuild.pl/voice-tool