Common and Cisco-Specific Tools

A variety of tools are available under different platforms. It is worthwhile to explore each of them, and to create a set of pros and cons for each one. This type of research is worth a separate book. One possible approach to classify all tools is the following:

• Network monitoring tools such as local-area network (LAN) and wide-area network (WAN) analyzers and sniffers, such as Sniffer Pro from Network Associates or EtherPeek.

• Remote control software, such as pcAnywhere from www. symantec .com, or Virtual Network Computing (VNC) software, which is used widely throughout the industry.

• Simulative equipment and software. Usually new solutions require lab environment testing before testing in production. The same logic applies to some unique problems when you need to recreate the problem and test the resolution prior to implementing any solution. A convenient tool for this purpose is a hardware simulator. ADTRAN Inc. (www.adtran.com) offers a comprehensive line of WAN test equipment that enables quick and economical deployment of fiber, T3, DSL, T1, E1, Frame Relay, VPN, ISDN, Digital Data Service (DDS), and wireless digital networks.

• Some of the best troubleshooting tools, such as Output Interpreter (www.cisco.com/support/OutputInterpreter/parser.html), are available and updated at Cisco.com.

Network analyzers and sniffers are extremely useful when troubleshooting data link layer and network layer issues such as frame formats, segmentations, and access control lists. (For more information, see Chapter 13, "Troubleshooting Scenarios for ISDN BRI.") Remote control software can be particularly useful for troubleshooting some applications when basic connectivity is available so that you can see what remote users see on their screen. Simulators and software that recreates the problem are recommended tools for the preproduction environment or for the pilot stage of the deployment of new products or services.

Each of the tools listed here requires some investment and it is up to you to define the purpose for each tool and to learn to use it effectively. A common set of tools is widely available for UNIX, Microsoft, and Cisco platforms that can be used for basic troubleshooting. Some of the more common tools are listed in the following sections.

Ping and Privileged (Extended) Ping Commands

These commands are available for Cisco, Microsoft, and UNIX platforms.

The ping utility can determine basic connectivity. It is based on the specifications for RFC 1256 and RFC 791. Ping is assigned a low priority. It is not an indicator of performance, and it is not used for other more sophisticated tests. During a ping test, the host sends an ICMP echo packet and receives a reply ICMP message if basic connectivity exists.

NOTE

An ICMPv6 is available for the IPv6 protocol (see RFC 1885).

The IOS-based ping can be used in two modes : user mode and ping EXEC mode (or Privileged Ping). The user mode ping command is for users who do not have privileged mode access to the device. The command syntax is as follows :

 Router>  ping  [  protocol  ] [  host     address  ] 

The protocol option allows you to define several options such as IP, source-route bridging (SRB), and tag. Only the non-verbose form is available, and if the host address can be resolved by the Domain Name System (DNS) server, the ping command returns !!!!!. If it cannot be resolved by DNS, the host returns ...... If it fails, it is always a good idea to try to ping by IP address and bypass possible DNS issues. The command returns success rate in percent and RTT in minimum, average, and maximum. For example, if you see a RTT value equal to 50 ms, 500 ms, and 5 seconds, your next questions is: Is this what I should expect, or is this far from the baseline? These questions can be answered by using ping and knowing the topology of the network, the baseline, the expected performance characteristics, and the way that the traffic is carried (local exchange carriers [LECs], inter-exchange carriers [IXCs], local loop, and if it is dial, ISDN, Frame Relay, or any other technology). As an example, for satellite-based VPN connections, 0.7 seconds RTT is what you should expect; however, this RTT is unacceptable for Frame Relay.

The extended ping is much more useful and interesting for troubleshooting. The command provides a variety of options for protocols such as IP, Internetwork Packet Exchange (IPX), AppleTalk, Connectionless Network Service (CLNS), Digital Equipment Corporation net (DECnet), Virtual Integrated Network Service (VINES) , and Xerox Network Systems (XNS). The results can be interpreted differently (see Example 4-3).

Example 4-3. Example of Extended Ping to 10.68.10.70

 Router#  ping  Protocol [ip]: Target IP address:  10.68.10.70  Repeat count [5]:  100  Datagram size [100]:  3000  Timeout in seconds [2]:  3  Extended commands [n]:  y  Source address or interface: Type of service [0]: Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]:  0x0000  Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: Type escape sequence to abort. Sending 100, 3000-byte ICMP Echoes to 10.68.10.70, timeout is 3 seconds: Packet has data pattern 0x0000 

The first important consideration is the ability to modify the repeat count (the default is 5), datagram size, and the timeout period (the default is 2). A higher number of ICMP echoes creates traffic for a longer period of time, and combined with the increased size of the packet (3000 bytes), allows you to test Point-to-Point Protocol (PPP), Multilink PPP (MPPP), and Multichassis Multilink PPP (MMP) (see Part II, "Dial," and Part III, "ISDN," for more detail on PPP, MPPP, and MMP) for dialup or ISDN connections, and for thresholds, reliability, txload, and rxload in some interfaces. It is rare to receive a timeout greater than two seconds in wired networks; however, the option is available and can be used accordingly in a wireless environment. One indicator for successfully running compression over a WAN connection is a non-proportional increase in RTT, when you increase the packet size following every test. This is based on the fact that the compression ratio is not a linear function of the packet size.

RFC 791 defines different types of reports that can be used to analyze connectivity issues. The reports are Loose, Strict, Record, Timestamp, and Verbose, and are based on the content of the IP packet. The same is true for TOS parameters and for setting the fragmentation bit. The latter plays a significant role in some connectivity issues and MTU issues in VPN solutions. This, combined with the sweep range of sizes, automatically modifies the packet size and determines the minimum MTU in the path .

NOTE

More about MTU is discussed in Chapter 21, "Remote Access VPN Troubleshooting."

Regarding data patterns, in some instances, it is not appropriate to use the Cisco standard 0xABCD sequence because some connectivity tests are much more productive based on certain specifics of the technology. In T1 and Primary Rate Interface (PRI) connectivity issues, the ones density requirement requires a certain number of 1s to be available. Their absence is referred to as a loss of signal (LOS) (see Chapter 3, "The Cloud"). Running 0x0000 or 0xffff provides more information than a standard Cisco sample. Another example for the same purpose is 0xAAAA, 0xA5A5, and 0x4040. Because of the specifics of T1 lines, it is recommended that you use the 0x0000 pattern for testing. Long series of 0s are indistinguishable from LOS (see Chapter 3 and Chapter 7, "Dial Troubleshooting"), so if the T1 is functional, a zero-insertion procedure is in effect (see Part IV, "Frame Relay"). 0xFFFF helps to localize repeater power problems, and 0x4040 is recommended for timing problems. It is also recommended to use different patterns when testing the compression ratio in a lab environment because of the deficiencies in the standard 0xABCD pattern.

In general, ping returns the following replies:

• . Timed out; waiting for reply

• ! Reply received

• U Destination unreachable

• N Network unreachable

• P Protocol unreachable

• Q Source quench

• M Could not fragment

• ? Unknown packet type

ICMP type messages are listed in Table 4-3.

When the ICMP type is 3 (destination unreachable), the ICMP codes identify the failure. The failures are defined in Table 4-4.

Table 4-3. ICMP Type Messages and Descriptions

ICMP Type Message	Description of ICMP Types
	Echo Reply
3	Destination Unreachable
4	Source Quench
5	Redirect Message
8	Echo Request
11	Time Exceeded
12	Parameter Problem
13	Timestamp Request
14	Timestamp Reply
15	Information Request (No Longer Used)
16	Information Reply (No Longer Used)
17	Address Mask Request
18	Address Mask Reply

Table 4-4. ICMP Codes When the ICMP TYPE Is 3Destination Unreachable

ICMP Codes for the ICMP Message Type = 3Destination Unreachable	Description
	Network unreachable
1	Host unreachable
2	Protocol unreachable
3	Port unreachable
4	Fragment needed and Don't Fragment (DF) bit set
5	Source route failed
6	Network unknown
7	Host unknown
8	Source Host isolated
9	Communication with destination network is administratively prohibited
10	Communication with destination is administratively prohibited
11	Bad type of service for destination network
12	Bad type of service for destination host
13	Administratively blocked by filter

Ping tests the end-to-end connectivity, but if the ping fails, it does not provide you with information on where the possible problem exists. The hop-by-hop information is available in the traceroute command, which in a sense extends ping's functionality.

The Traceroute and Privileged (Extended) Traceroute Commands

Available as traceroute in Cisco and UNIX based-platforms, and tracert in Microsoft-based platforms. The traceroute (Cisco/UNIX) and tracert (Microsoft) commands differ slightly. Cisco/UNIX uses UDP packets, and Microsoft uses ICMP packets, which are more likely to be filtered.

The traceroute command tests the connectivity of the path between two devices (hops). It narrows down connectivity issues, helps to discover routing loops , and determines baseline network layer performance on a hop-by-hop basis.

NOTE

The traceroute command is considered a baseline tool. This suggestion is not exactly correct because every router contains links, switching-forwarding engines, and queues. The mathematical model for traceroute does not consider the queues that are typical for every hop when there is competing traffic. This is why it is more precise to refer to traceroute as a hop-discovery tool.

The traceroute mechanism is based on the TTL field of an IP packet. This field is initialized by the sender. If the field is decremented to 0, the packet is discarded and an error indication packet (an ICMP time exceeded) is sent back to the sender. The source address of the ICMP error packet identifies the hop that discarded the data packet. This is the mechanism that identifies every hop. The sending host sets the TTL initially to 1. Every hop in the path decreases the TTL by 1. So for the first hop in the path, the TTL is decreased by 1, and because TTL now equals 0, the error message containing the hop information packet is sent back to the sender. The source then increases the TTL to 2, and after two hops it is sent back. The sender now tries using 3, and so on. The traceroute repeats itself until the command terminates when either the host is reached, the escape sequence ( Ctrl-Shift-^ ) is triggered, the TTL reaches 30 (default TTL for Cisco), or the TTL exceeds the set amount. If the timer goes off before the response from the probe packet is back, the symbol * is displayed. The traceroute command can be used with the following protocols:

 clns        ISO CLNS Trace ip          IP Trace ipx         IPX Trace oldvines    Vines Trace (Cisco) vines       Vines Trace (Banyan) 

The privileged traceroute command offers some additional features, as shown in Example 4-4.

Example 4-4. The Privileged traceroute Command

 router#  traceroute  Protocol [ip]: Target IP address:  10.68.10.70  Source address: Numeric display [n]: Timeout in seconds [3]: Probe count [3]: Minimum Time to Live [1]: Maximum Time to Live [30]: Port Number [33434]: Loose, Strict, Record, Timestamp, Verbose[none]: Type escape sequence to abort. 

The options are self-explanatory and a detailed description of expected output from this command is listed in Table 4-5.

Table 4-5. Traceroute Output Description

Output	Description of the Output
nn msec	RTT in ms for each number of probes
*	Probe TTL expired
?	Unknown packet type
Q	Source quench
P, N, U, H	Protocol, network, port, host unreachable

The Netcat Utility

The Netcat utility takes advantage of the fact that telnet cannot test UDP connections, but Netcat can. It offers better functionality than telnet and can be used as a replacement because it does not inject too much data into the network. It can also test application connectivity and it is available in UNIX and Microsoft platforms.

The following are some of the options of the Netcat utility for the Microsoft NT platform (see Table 4-6):

 C:\cisco\pnedeltc\tools\netcat>  nc -h  [v1.10 NT] connect to somewhere:   nc [-options] hostname port[s] [ports] ... listen for inbound:     nc -l -p port [options] [hostname] [port] options: 

Table 4-6. Netcat Utility Options for Windows NT 4.0 and Windows 2000

Netcat Option	Description
-d	Detach from console, stealth mode.
-e	Execute inbound program.
-g	Gateway; source-routing hop point(s), up to 8.
-G	Number for source-routing pointer: 4, 8, 12, ....
-i	Delay interval in seconds for lines sent, ports scanned.
-I	Causes Netcat to listen to a given port (as specified with the p Flag); this option is useful for creating mock services to test throughput or connectivity; use with the u flag to create a UDP server.
-l	Listen mode, for inbound connects.
-L	Listen harder; relisten on socket close.
-n	Numeric IP address.
-o	Hex dump of the traffic.
-p	Local port number; when Netcat is run with the -I flag, use the specified port and IP address respectively.
-r	Randomize local and remote ports.
-s	Local source address; when Netcat is run with the -I flag, use the specified port and IP address respectively.
-t	Answer TELNET negotiation.
-u	UDP mode; tell Netcat to use UDP instead of TCP; Netcat simulates a UDP connection.
-v	Verbose (use twice to be more verbose).
-w	Timeout for connects and final net reads in seconds; changes the network inactivity timeout; changing this to at least 3 is useful when checking web or gopher services.
-z	ZeroInput/Output (I/O) mode (used for scanning).

Port numbers can be individual or a range: m-n (inclusive).

A simple example of using Netcat is to pull down a web page from a web server. With Netcat, you see the full HTTP header to see what web server a particular site is running. The command line is as follows:

 C:\cisco\pnedeltc\tools\netcat>  nc -v www.website.com 80 < get.txt  

Netcat makes a connection to port 80, sends the text contained in the file get.txt, and provides the web server's response to stdout . The -v stands for verbose.

Another combination is the -l or listen option and the -e or execute option that instructs Netcat to listen on a particular port for a connection. When a connection is made, Netcat executes the program of your choice and connects the stdin and stdout of the program to the network connection.

 C:\cisco\pnedeltc\tools\netcat>  nc -l -p 23 -t -e cmd.exe  

The telnet equivalent here is as follows:

 C:\cisco\pnedeltc\tools\netcat>  nc 110.71.87.84 23  

You can listen on any port that gives you an opportunity, even to determine if the firewall you might be behind lets port 53 through. To run Netcat listening behind the firewall on port 53, use the following:

 C:\cisco\pnedeltc\tools\netcat>  nc -L -p 53 -e cmd.exe  

Then from outside the firewall, connect to the listening machine by using the following command:

 C:\outside\pnedeltc\tools>  nc -v 10.71.87.84 53  

The extended set of features for different platforms is widely available over the Internet. This particular version was written by Weld Pond ([e-mail]).

Service Assurance Agent

A sophisticated Cisco IOS-based tool is available in IOS versions 12.2 or higher. The Service Assurance Agent (SAA) feature replaces the Response Time Reporter (RTR) in earlier IOS versions and provides extra capabilities such as VoIP metrics, response time, availability of HTTP (port 80) applications, and QoS based on the precedence field of the IP packet. The feature allows you to monitor network performance between a Cisco router and a remote device, which can be another Cisco router, an IP host, or a mainframe host, by measuring key Service Level Agreement (SLA) metrics such as response time, network resources, availability, jitter, connect time, packet loss, and application performance. SAA enables you to perform troubleshooting, problem analysis, and notification based on the statistics collected by the SAA. Example 4-5 shows a sample using IOS 12.2(6) that is configuring the HTTP traffic monitoring to www.cisco.com, and scheduling the data collection.

Example 4-5. SAA Sample

 Router#  conf t  Enter configuration commands, one per line.  End with CNTL/Z. Router(config)#  rtr 1  Router(config-rtr)#  type http operation get url http://www.cisco.com  Router(config-rtr)#  exit  Router(config)#  rtr schedule 1 start-time now  Router(config)#  ^Z  Router#  wr  Router#  show rtr collection-statistics  Collected Statistics Entry Number: 1 HTTP URL: http://www.cisco.com Start Time: *00:15:04.000 UTC Mon NOV 1 2001         Comps: 0             RTTMin: 0         OvrTh: 0             RTTMax: 0         DNSTimeOut: 0        RTTSum: 0         TCPTimeOut: 0        RTTSum2: 0         TraTimeOut: 0        DNSRTT: 0         DNSError: 0          TCPConRTT: 0         HTTPError: 0         TransRTT: 0         IntError: 8          MesgSize: 0 

It is recommended to place the SAA in strategic areas of your network (in the core or distribution layer) to ensure monitoring and to receive performance metrics for the network functionality. The SAA feature provides an extended set of # show and # debug features.

The IOS Commands show and debug

The # show and # debug features of the IOS are the main Cisco-specific tools that have to be continually learned by troubleshooting engineers to obtain the maximum available information from the devices. Cisco defined seven severity levels of report of certain events in Cisco routers. One important point here is to understand that by default the # debug command provides information for a certain severity level and higher. The default severity level can be changed with the command in Example 4-6.

Example 4-6. Changing the Default Severity Level

 804-isdn(config)#  logging console ?  <0-7>           Logging                            severity level   emergencies     System is unusable                 (severity=0)   alerts          Immediate action needed            (severity=1)   critical        Critical conditions                (severity=2)   errors          Error conditions                   (severity=3)   warnings        Warning conditions                 (severity=4)   notifications   Normal but significant conditions  (severity=5)   informational   Informational messages             (severity=6)   debugging       Debugging messages                 (severity=7)   guaranteed      Guarantee console messages 

The # show and # debug commands are widely used in the troubleshooting process. Another important preliminary rule for the # debug command is that when debugging, some of the debug commands have chatty output and if mishandled, can crash the router. You can even cancel the terminal monitoring with Router# terminal no monitor, but this does not cancel the debug process. It is recommended that you open a secondary session (Telnet, Secure Shell [SSH]) to the same router and prepare to cancel the debugging with the shortest command you know. Remember that some debug commands can be used only if the utilization is low (use #show processes cpu [history] command), or in a non-production environment.

These commands are widely described in the following chapters to demonstrate some of the most common and recommended Cisco tools for a remote access environment.