General Troubleshooting Techniques

Cisco UE offers several types of tools to help you isolate problems on the system:

• Viewing system information and status (using a variety of show commands)
• Event logging to log files including information, warning, error, and fatal messages
• A tracing facility showing detailed information on system events, information, and decisions (similar to Cisco IOS debug information)

Viewing Generic System Information

This book provides you with information about Cisco UE so that you can troubleshoot and fix most of the common problems you might encounter. In case you still have to contact the Cisco Technical Assistance Center (TAC) for additional support, however, it is helpful to provide the Cisco TAC with some basic information about your system. This basic information includes the software version, the software license, the current configuration of the Cisco CME and Cisco UE components, and some information about the current usage of the system. This section describes how to view that basic information.

Example 19-1 uses the show version command to provide information about the hardware form factor of the Cisco UE module, the model type of the router that is hosting the Cisco UE, the speed with which CPU is running, and additional hardware-specific information. One thing to note is that the AIM-CUE runs at 150 MHz if it is hosted on 2600XM or 2691 series access routers. In other routers, it runs at 300 MHz.

Example 19-1. Displaying the System Version

CUE#show version
CPU Model: Celeron (Coppermine)
CPU Speed (MHz): 154.926
CPU Cache (KByte): 128
Chassis Type: C2600
Chassis Serial: FSJC0400673
Module Type: AIM
Module Serial: FHH07330001

Example 19-2 uses the show software version command to display the versions of the Cisco UE software components installed on the system. Identifying the installed software versions is important when troubleshooting interworking issues. For example, some Cisco UE features might be supported only with a certain version of Cisco CME or Cisco UE.

Example 19-2. Displaying the Software Version

CUE#show software version
Installed Packages:
 - Bootloader (Primary) 1.0.17
 - Global 2.0.0.12
 - Voice Mail 2.0.0.12
 - Bootloader (Secondary) 2.0.0
 - Core 2.0.0.12
 - Auto Attendant 2.0.0.12

Installed Languages:
 - US English 2.0.0.0

Example 19-3 shows the license installed on the system by using the show software licenses command.

Example 19-3. Displaying Licenses

CUE#show software licenses
Core:
 - application mode: CCME
 - total usable system ports: 4
Voicemail/Auto Attendant:
 - max system mailbox capacity time: 480
 - max general delivery mailboxes: 15

 - max personal mailboxes: 50
Languages:
 - max installed languages: 1
 - max enabled languages: 1

Example 19-4 uses show voicemail usage to show the current usage of the different resources in the system, including the number of personal mailboxes, general delivery mailboxes (GDMs), and the voice mail capacity allocation.

Example 19-4. Displaying the System's Current Usage

CUE#show voicemail usage
personal mailboxes: 50
general delivery mailboxes: 0
orphaned mailboxes: 0
capacity of voicemail (minutes): 480
allocated capacity (minutes): 350.0
message time used (seconds): 15227
message count: 856
average message length (seconds): 17.78855140186916
greeting time used (seconds): 0
greeting count: 0
average greeting length (seconds): 0.0
total time used (seconds): 15227
total time used (minutes): 253.78334045410156
percentage used time (%): 53

The show voicemail limits command, shown in Example 19-5, shows the limits currently enforced on the voice mail system (based on the system's license). These limits include the default mailbox size, default recording limit for callers and subscribers, and message expiration duration.

Example 19-5. Displaying System Voice Mail Limits

CUE#show voicemail limits
Default Mailbox Size (seconds): 420
Default Caller Message Size (seconds): 60
Maximum Recording Size (seconds): 900
Default Message Age (days): 30
System Capacity (minutes): 480
Default Prompt Language: en_US
Operator Telephone: 0

 

Logging in Cisco UE

Cisco UE provides a full-fledged logging facility. Messages are logged to the console, to a Syslog server, or to the messages.log file on the disk or flash. Cisco UE has four levels of logging. Table 19-1 shows the Cisco UE logging levels mapped to Syslog logging levels.

Table 19-1. Cisco UE Logging Levels

Cisco UE Logging Level	Syslog Logging Level
INFO	Debug, Info, and Notice
WARNING	Warning
ERROR	Error
FATAL	Critical, Alert, and Emergency

If you are monitoring straightforward issues, such as whether the Cisco UE system reloaded, you might want to look at the Syslog messages before you start troubleshooting by turning on traces. These messages provide a brief problem description, a possible reason for the event, and other relevant information about a problem. These messages might be displayed on the console, written to a Syslog server, or stored in the Cisco UE messages.log file, depending on the system's configuration. Example 19-6 shows different ways of turning on logging in Cisco UE. Logging to the console displays the log messages on the console when they occur.

Example 19-6. Logging Levels and Configuration

cue(config)#log console ?
 errors Error messages
 filter Filter events from syslog
 info Information messages
 warning Warning messages
cue(config)#log console info

Logging to a server
cue(config)#log server 1.4.53.1 

If you are using a Network Module Cisco UE (NM-CUE), the log messages are also written to a file called messages.log on the hard disk. If you want offline access to the file, you can use command-line interface (CLI) commands to copy the messages.log file to an FTP server. Example 19-7 shows the log files on the system.

Example 19-7. Viewing Log Files on Cisco UE

cue#show logs
linux_session.log
dmesg
syslog.log
shutdown.log
atrace.log
debug_server.log
memmon.log
klog.log
messages.log 

The following steps show you how to copy the log file to an external server:

Step 1.	Have an FTP server ready that the Cisco UE system can reach. Ensure that the FTP server is active.
Step 2.	Set up a session to the Cisco UE CLI from the router using the service-module service-engine <slot/port> session command. Enter the show logs command, as shown in Example 19-7, to make sure that the desired log files are available.
Step 3.	Enter the following command for each file you need to transfer: CUE#copy log log_file url ftp://ftp_ip_address/   where log_file is the filename of the desired log file and ftp_ip_address is the IP address of the FTP server. An example of a command to copy messages.log to an FTP server is cue#copy log messages.log url ftp://user:password@1.3.61.16/messages.log

Tracing Techniques

Cisco UE provides an extensive CLI-based tracing and debugging facility. It provides tracing commands for most of the system's important components, such as voice mail, the Customer Response Solutions (CRS) component, the web GUI interface, the Session Initiation Protocol (SIP) call processing component, and voice mail networking. Tracing uses the concepts of module, entity, and activity. For example, consider the following command:

cue#trace ccn managerappl dbug

In this example, trace is the command, ccn is the module that represents the CRS component, and managerappl (which stands for application manager) is the entity within the ccn module. The dbug parameter is the activity to be traced.

By default, trace output is stored in a memory buffer. You can use commands to view the trace buffer contents on the console. You also have the option of configuring the trace output to write to a file called atrace.log. If you are using an NM-CUE, writing trace output to the disk is enabled by default. On the Advanced Integration Module (AIM-CUE) hardware, tracing is disabled by default. You should enable it with care, because the AIM-CUE flash card has space limitations and a finite lifetime in terms of the number of write cycles before you must replace it. Doing indiscriminate tracing to your flash card shortens its life unnecessarily. Instead, if you are using an AIM-CUE, it is recommended that you use an external FTP server as the destination device for writing trace files.

trace Command Summary

Table 19-2 lists the most important trace commands. You can find the complete list of commands for tracing, modules, entities, and activities in the "Cisco UE System Administration and CLI Guides" available on Cisco.com.

Table 19-2. Cisco UE trace Commands

Command	Description
trace module entity activity	Turns on the trace for a desired module, entity, and activity.
show trace	Displays the current trace settings.
show trace buffer	Shows the current contents of the trace buffer. This output can be voluminous, depending on what traces were turned on, so use it carefully. This command displays the complete contents of the trace buffer from top to bottom.
show trace buffer containing regexp pattern	Displays the lines in the trace buffer that contain a particular pattern.
show trace buffer tail N	Shows the latest trace entries to the trace buffer in real time as and when they happen. If N is given, it shows the last n most recent trace entries and then continues displaying subsequent trace entries as and when they happen. If N is omitted, only new events are shown as they occur.
show trace buffer tail N long	Prints the traces in long format. The long format expands on the internal codes for different events and messages so that the trace output is more readable. For example, when you use trace voicemail vxml all, the output prints the text equivalent of the prompt file instead of just giving the prompt names.
clear trace	Clears the trace buffer. Usually used when troubleshooting problems in real time and when you don't need the previous contents of the trace buffer.

 

Turning on Traces

Depending on the problem area you are facing, you turn on traces for specific modules and entities. You might often have to turn on traces for several different modules and entities simultaneously to see how each module handles a particular data item or event. You might have to clear the trace buffer between different trace activities to make the troubleshooting output more readable and manageable, especially because the console connection speed is hard-coded to 9600 baud.

Each trace message in the output contains a time stamp of when the trace was generated. It also contains the module and entity that issued the trace output line so that when you read the trace, it is easy to understand which trace line was originated by which module. For example, in the following trace line, ACCN is the module (and stands for the CCN/CRS component), and VBRO is the entity (and stands for the voice browser).

 2699 10/10 12:23:30.115 ACCN VBRO 0 Task:18000000079 Invoke:
 http://localhost/voicemail/vxmlscripts/login.vxml

For most problems, turning on trace for a couple of modules and entities is sufficient, and the trace output is a manageable volume. More complex problems might require tracing to be turned on for many modules and entities, and the trace output can become cumbersome.

If a trace is turned on for only a couple of modules, it is easy to view the trace in real time. Thus, you turn on the trace, clear the trace buffer, issue the show trace buffer tail command, and make the call to trigger the system events you want to trace. As the call enters the Cisco UE application, you see trace output printed on the console. You can stop the trace output by pressing Ctrl-C, as shown in Example 19-8. The trace is cut off the moment you press Ctrl-C.

Example 19-8. Viewing a Trace in Real Time

cue#trace ccn VbrowserCore dbug
cue#clear trace
cue#show trace buffer tail
Press  to exit...
DefaultCompilationConfig is now set to JVM en_US
2043 09/25 15:56:22.305 ACCN VBRW 0 Task:20000000089 enterLevel: SESSION_LEVEL
2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getSessionID() = 6000000059
2043 09/25 15:56:22.306 ACCN VBRW 0 Task:20000000089 VoiceBrowser sessionid:
 6000000059, taskid: 20000000089
2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCallingNumber() = 5702
2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCallingNumber() = 5702
2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getDNIS() = null
2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCalledNumber() = 5800
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getDNIS() = null
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getCalledNumber() = 5800
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getANIIIDigits() = null
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getUserToUserInfo() = null
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getRDNIS() = null
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getLastRedirectedNumber() = null
2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getSessionID() = 6000000059
2043 09/25 15:56:22
cue#

If you run into a situation where you have to turn on tracing for many modules and entities, or even trace all, it is better not to attempt to watch the trace output in real time, because it becomes very difficult to follow. All the trace output is written to a file called atrace.log. On the NM-CUE, this file is written by default. On an AIM-CUE, use the log trace local enable command to allow the trace buffer to be written to flash. You can also use log trace server so that trace output is written to an FTP server instead of a flash on your Cisco UE, as shown in Example 19-9.

Example 19-9. Writing Trace Output to a File

cue(config)#log trace server url ftp://userid:password@192.168.0.1/trace
cue(config)#log trace server enable

After you make the preceding configuration, the trace file is created on the specified server.

Copying a Trace File to an FTP Server

After you make the test call to trigger the events you want to capture to a trace file, you must copy the atrace.log file to an FTP server, as shown in Example 19-10.

Example 19-10. Displaying and Copying a Trace File

cue#show logs
linux_session.log
dmesg
syslog.log
shutdown.log
atrace.log 
debug_server.log
memmon.log
klog.log
messages.log
cue#copy log atrace.log url ftp://user:password@1.3.61.16/atrace.log

The atrace.log file is a binary file that you must decode to view in a text editor. You can get the utility to decode the trace file from the Cisco TAC. The utility consists of three files:

• atrace_decode (an executable)
• trace.tcmd
• trace.tmsg

Decode the atrace.log file as shown in Example 19-11. This creates an output file (dtmf_issue.log, in this example) that you can now view in a text editor. Currently, atrace_decode runs only on a Linux platform.

Example 19-11. Decoding an atrace.log Trace File

[test@CUE-core ftp]$ ./atrace_decode
Usage: ./atrace_decode   
 
format = 0 for short, 1 for long
[test@CUE-core ftp]$./atrace_decode atrace.log 1 dtmf_issue.log ./trace.tcmd

 ./trace.tmsg

 

Troubleshooting System Time Issues and NTP

Many of the Cisco UE features depend on a reliable clock. The voice message envelope information is one important example. As discussed in Chapter 13, "Cisco IPC Express General Administration and Initial System Setup," Cisco UE's clock is managed entirely by Network Time Protocol (NTP) and cannot be set manually.

Another Cisco UE feature that requires clock information is the trace and log messages that carry time stamps. If the clock on the Cisco UE system is not set correctly, troubleshooting might become very difficult, because log and trace messages cannot be correlated with real time.

You should configure an NTP source for Cisco UE that is the clock's source. An NTP source can be the host Cisco CME router (although this is not recommended because the low-end routers do not have a very reliable onboard clock source) or any other clock source in your network. Sometimes, for many reasons, the clock on Cisco UE might not be synchronized to its configured source. You can verify the status of NTP and the location of the clock's source with CLI commands. Example 19-12 shows these commands.

Example 19-12. NTP-Related Commands

NTP not working due to wrong configuration:
CUE#show ntp status
NTP reference server : 1.3.6.190
Status: reject
Time difference (secs): 0.0
Time jitter (secs): 4.0

NTP in Sync:
CUE#show ntp status
NTP reference server : 1.4.13.190
Status: sys.peer
Time difference (secs): 0.02110639284364879
Time jitter (secs): 0.019472182961180806

Trace commands available to debug NTP synchronization issues:
CUE#trace ntp ?
 all Every entity and activity
 ntp Entity
CUE#trace ntp ntp ?
 all Every activity
 clkadj Activity
 clkselect Activity
 clkvalidity Activity
 clockstats Activity
 event Activity
 loopfilter Activity
 loopstats Activity
 packets Activity
 peerstats Activity

Troubleshooting Installation Problems