1-9 Some Troubleshooting Tools

  • The ping (packet Internet groper) command can be used to test end-to-end connectivity from a router to a remote host or router. For IP, ping uses ICMP type 8 requests and ICMP type 0 replies.

  • The traceroute command can be used to discover the routers along the path that packets are taking to a destination. For IP, traceroute uses UDP probe packets on port 33434.

  • The telnet command can be used to open Telnet connections to other routers or hosts .

  • Many debug commands and options are available to display information about activity within the router.

  • If a network analyzer is unavailable, a router can be configured to present basic information about network traffic. This is called the "poor man's sniffer."

  • If a router crashes, information can be gathered to assist in troubleshooting the cause.

  • You can view information about the activity of the router CPU, memory, interfaces, and protocols.

  • The Cisco TAC offers a number of troubleshooting tools and technical support resources.

NOTE

This section does not present a complete list of troubleshooting commands. However, you should find a set of useful commands and tools here to use for a variety of situations.


IP Connectivity Tools: ping

 (exec)  ping  [  protocol  ] {  host  } 

Echo requests are sent, and echo replies are expected in return. Ping can be used with a variety of protocols. The protocol value can be appletalk, clns, ip (the default), novell, apollo, vines, decnet, or xns. Some protocols require another Cisco router at the remote end to answer ping packets. The target, host, can be either an address or a host name .

The IP ping sends ICMP type 8 (echo request) packets to the target. The following characters are displayed each time a ping response is expected or seen:

  • ! A successful reply packet was received.

  • . No reply was seen within the timeout period, 2 seconds.

  • U A destination-unreachable error was received.

  • M A could-not-fragment message was received.

  • C A congestion- experienced packet was received.

  • I The ping test was interrupted on the router.

  • ? An unknown packet type was received.

  • & The packet lifetime or time-to-live was exceeded.

As soon as the test completes, the success rate is reported , along with a summary of the round-trip minimum, average, and maximum in milliseconds .

NOTE

For the regular ping command, only the destination address may be given. The source address used in the ping packets comes from the router interface that is closest to the destination. This might or might not be helpful in determining connectivity.


IP Connectivity Tools: Extended ping

 (exec)  ping  

The extended ping is similar to the regular ping, except that it is typed with no options. You are prompted for all available ping options, including the source address to be used. The following options can be specified:

  • Protocol (the default is IP) This can also be appletalk, clns, novell, apollo, vines, decnet, or xns.

  • Target IP address

  • Repeat count (the default is 5 packets) The number of echo packets to send.

  • Datagram size (the default is 100 bytes) The size of the echo packet. Choose a size larger than the MTU to test packet fragmentation.

  • Timeout (the default is 2 seconds) The amount of time to wait for a reply to each request packet.

  • Extended commands

    • Source address or interface Any source address can be given. The address must be the address of any active interface on the router if the reply packets are to be seen.

    • Type of service (the default is 0).

    • Set the DF bit in the IP header (the default is no) If this is set, the packet is not fragmented for a path with a smaller MTU. This can be used to detect the smallest MTU in the path.

    • Validate reply data (the default is no) The data sent in the echo request packet is compared to the data echoed in the reply packet.

    • Data pattern (the default is 0xABCD) The data pattern is a 16-bit field that is repeated throughout the data portion of the packet. This can be useful for testing data integrity with CSU/DSUs and cabling.

    • Loose, Strict, Record, Timestamp, Verbose (the default is none) loose (loose source route with hop addresses), strict (strict source route with hop addresses), record (record the route with a specified number of hops), timestamp (record time stamps at each router hop), or verbose (toggle verbose reporting). The record option can be useful to see a record of the router addresses traversed over the round-trip path.

  • Sweep range of sizes Sends echo requests with a variety of packet sizes:

    • Sweep min size (the default is 36)

    • Sweep max size (the default is 18024)

    • Sweep interval (the default is 1)

IP Connectivity Tools: traceroute

 (exec)  traceroute  [  protocol  ] [  destination  ] 

The traceroute command sends successive probe packets to destination (either a network address or a host name). The protocol field can be appletalk, clns, ip, or vines.

For IP, the first set of packets (the default is 3) is sent with a Time-to-Live (TTL) of 1. The first router along the path decrements the TTL, detects that it is 0, and returns ICMP TTL-exceeded error packets. Successive sets of packets are then sent out, each one with a TTL value incremented by 1. In this fashion, each router along the path responds with an error, allowing the local router to detect successive hops.

The following fields are output as a result of traceroute probes:

  • Probe sequence number The current hop count.

  • Host name of the current router.

  • IP address of the current router.

  • Round-trip times (in milliseconds) of each of the probes in the set.

  • * The probe timed out.

  • U The port unreachable message was received.

  • H The host unreachable message was received.

  • P The protocol unreachable message was received.

  • N The network unreachable message was received.

  • ? An unknown packet type was received.

  • Q The source quench was received.

The traceroute probes continue to be sent until the maximum TTL value (30 by default for IP) is exceeded or until you interrupt the router with the escape sequence ( Ctrl-Shift-6 ).

Traceroute can also be invoked with no options. This allows the router to prompt for the parameters described in the following list:

  • Protocol (the default is IP) Can also be appletalk, clns, or vines.

  • Target IP address

  • Source address An IP address of a router interface. If this isn't specified, the interface closest to the destination is used.

  • Numeric display (the default is no) By default, both the host name and IP address of each hop are displayed. If this is set to yes, only the IP addresses are displayed. This is handy if DNS is unavailable.

  • Timeout in seconds (the default is 3) The amount of time to wait for a response to a probe.

  • Probe count (the default is 3) The number of probes to send to each TTL (or hop) level.

  • Minimum Time-to-Live (the default is 1) The default of one hop can be overridden to begin past the known router hops.

  • Maximum Time-to-Live (the default is 30) The maximum number of hops to trace. Traceroute ends when this number of hops or the destination is reached.

  • Port number (the default is 33434) The UDP destination port for probes.

  • Loose, Strict, Record, Timestamp, Verbose (the default is none) loose (loose source route with hop addresses), strict (strict source route with hop addresses), record (record the route with a specified number of hops), timestamp (record time stamps at each router hop), or verbose (toggle verbose reporting). The record option can be useful to see a record of the router addresses traversed over the round-trip path.

NOTE

Some routers do not respond to traceroute probes correctly. In this case, some or all of the probes sent are reported with asterisks (*) in the display.


IP Connectivity Tools: Telnet

 (exec)  telnet  [  host  ] 

A Telnet session is opened to the target host (either an IP address or a host name). After it is opened, the session can be suspended using the escape sequence ( Ctrl-Shift-6 x ) so that another session can be initiated. See Section 1-1 for more information about controlling and using sessions.

NOTE

The router initiates a Telnet connection using a source IP address taken from the interface that is "closest" to the destination. To force Telnet to use a specific active interface and IP address as a source address, use the ip telnet source-interface interface command in global configuration mode.


Debugging Output from the Router

  1. Choose a method to collect debug output.

    (See Section 1-5 for more information about logging router messages to a syslog server.)

    1. Save debugging output in a router buffer:

       (global)  logging buffered  [  size  ] 

      All debugging output is stored in a circular buffer on the router itself. The size of the buffer can be given as size, ranging from 4096 to 4294967295 bytes. The default size is hardware-dependent. It can be verified using the show logging command.

    2. Send debugging output to a syslog server:

       (global)  logging   host  

      The target, host (either an IP address or a host name), receives debug output as syslog messages from the router. By default, the syslog messages are at the debugging level.

    3. Send debugging output to a nonconsole session:

       (exec)  terminal monitor  

      The Telnet session that this command is executed from receives system messages and debug output. If debugging output is not seen, use the logging monitor debugging command to enable output at the debugging level.

    4. Send debugging output to the router console:

       (global)  logging console  [  level  ] 

      All debugging output is sent to the router console. By default, logging is performed at the debugging level. If this is changed in the router configuration, you can override it by specifying the level option as debugging.

      NOTE

      Use caution when sending debugging output to various destinations. Debugging commands can output a large volume of message data in a short amount of time, resulting in very sluggish router performance. Therefore, choose a destination that can collect and present the data efficiently . The logging buffered command is the most efficient method, because the messages are stored directly in a router buffer. The debug logging methods , in increasing order of system overhead, are logging buffered (the least overhead), logging host, terminal monitor, and logging console (the most overhead). In fact, debugging to the console can actually crash a router because of the overhead involved in large outputs. You should use extreme caution when using any debug command and use debug commands that focus on the information you want to look at (that is, debug events or use access lists to control the debugging actions).

  2. Enable time stamps on debugging output:

     (global)  service timestamp debug datetime  [  msec  ] [  localtime  ]   [  show-timezone  ] 

    Debugging output messages are recorded with the date and time. Time is shown with millisecond resolution if the msec keyword is used. By default, time is displayed in the UTC format. To record time stamps using the local time zone, use the localtime keyword. In addition, the show-timezone keyword causes the local time zone name to be displayed.

    NOTE

    Before time stamps are enabled, be sure that the router clock (and hardware calendar) have been set correctly. Refer to Section 1-4 for more information.

  3. Enable appropriate debugging:

     (exec)  debug  

    There are many debug commands and options! Use the context-sensitive help ( debug ? ) to determine which debug options are available, or refer to the online Debug Command Reference at http://www.cisco.com/univercd/cc/td/doc/product/software/ios121/121sup/121debug/index.htm.

    Use extreme caution and common sense before enabling any debug command. Debug messages are processed at a higher priority than other network traffic. Consider the amount and types of traffic on your network, realizing that every packet or condition matching the debug command causes a message to be generated. For example, enabling debug all generates debug output for every known debug condition. In a large IPX network, debug ipx sap generates messages for each SAP advertisement. Although debug ip packet can be very useful for determining routing problems, it also generates messages for every packet passing through the router.

    To reduce the amount of debug information and processing, either choose a time when the traffic load is low, or fine-tune the debug parameters to further limit the activity that is being observed .

    In the event that you begin to notice large amounts of debug output piling up, be ready to quickly disable the debug command. You can either type the no debug options form of the command, or you can use no debug all, undebug all, or u all to disable all possible forms of debugging. In any event, you should always disable debugging when you are finished testing so that the router doesn't continue reporting debugging output without your knowledge.

Poor Man's Sniffer

  1. Use IP accounting.

    1. Enable IP accounting on an interface:

       (interface)  ip accounting  [  access-violations  ] 

      The router keeps records of the outbound traffic through the interface. The database consists of the source and destination addresses, the number of packets, and the number of bytes (both IP header and payload) switched for the conversation. The traffic data is gathered, regardless of the switching path through the router. This information can be useful to give you an idea of the pairs of hosts talking and the volume of data passing between them.

      If the access-violations keyword is included, information is gathered about outbound traffic that failed to pass an access list. This information can be used to determine whether the failed traffic is due to an attempt to breach security.

    2. Display the IP accounting information:

       (exec)  show ip accounting  [  checkpoint  ] [  output-packets   access-violations  ] 

      The checkpoint keyword can be used to display the checkpointed database; otherwise , the active database is shown. The output-packets keyword causes the total number of outbound packets to be displayed (the default). The access-violations keyword shows the total traffic that has failed to pass access lists, along with the access list number of the last packet failure.

  2. Use an extended access list to determine something about a traffic flow.

    1. Define an extended IP access list (see Section 14-1).

      A named extended IP access list is the most useful for this purpose, because you can edit it without removing and reentering the entire access list. Make permit conditions that break down the unknown traffic into possible categories. For example, if you need information about what protocol is being used between two hosts, use permit conditions that specify individual protocols. The port operators can also be useful to see into what range of port numbers a certain traffic flow falls . This can be done with the eq, gt, lt, neq, and range operators. Use as many permit statements as is practical, covering all possibilities for a parameter. Be sure to add a permit all condition at the end so that existing traffic flows are not affected by this access list. The goal here is to get the router to flag access list statements to identify traffic, not alter the traffic flow.

      The following example defines an access list that helps determine whether two hosts are using TCP or UDP (or some other IP protocol) for a data connection:

        ip access-list extended sniffer   permit tcp host 192.168.7.15 host 12.1.6.4   permit udp host 192.168.7.15 host 12.1.6.4   permit icmp host 192.168.7.15 host 12.1.6.4   permit ip any any  
    2. Apply the access list to an interface:

       (interface)  ip access-group   acc-list  {  in   out  } 

      The access list acc-list is used to process either inbound ( in ) or outbound ( out ) traffic on an interface.

    3. Look at the access list activity:

       (exec)  show access-lists  [  acc-list  ] 

      For an extended IP access list, a count is displayed for each packet that matches a condition in the list. To see the results for a single access list, you can give an optional access list number or name as acc-list. If you defined the access list breakdown wisely, you should begin to see activity for the matching traffic displayed as (xx matches) at the end of some access list conditions.

    4. (Optional) Refine the access list to uncover more detail.

      If you need more information about the type of traffic, you can refine the access list. For example, you might want to know which UDP port number, TCP port number, ICMP type code, IP precedence, or ToS is in use. You can add tests for these options by editing lines in the named access list. Remember to provide access list conditions for all or groups of all possibilities so that you can locate positive results.

    5. Remove the access list from the interface:

       (interface)  no ip access-group   acc-list  {  in   out  } (global)  no ip access-list extended   acc-list  

      Don't forget to unbind the access list from the interface and to remove the access list from the configuration when you are finished with your analysis.

Troubleshooting Router Crashes

  1. Collect a stack trace:

     (exec)  show stacks  

    If the router has a system failure and restarts itself, the system stack trace is saved. You can view the stack trace, along with the reason for the last router restart, using this command. The stack trace output can be copied and pasted into Cisco's Stack Decoder at http://www.cisco.com/stack/stackdecoder.shtml to get further diagnostic information (a CCO login is required). Problem isolation and software bug IDs can be determined through the stack decoding process.

  2. Collect crashinfo data.

    When a router crashes due to data or stack corruption, a collection of useful information is saved as a file in the bootflash partition on the router. Crashinfo data is saved on the Cisco 7000, 7200, 7500, and 12000 series routers.

    1. Find the crashinfo file:

       (exec)  dir /all bootflash:  

      The files in the bootflash are listed. Crashinfo files are named.

    2. Look at the contents of the crashinfo file:

       (exec)  show file information bootflash:crashinfo  
    3. Collect a core dump.

      • Choose a method to write a core dump:

         (global)  exception protocol  {  ftp   rcp   tftp  } 

        The core dump file can be written to a server using FTP, RCP, or TFTP (the default).

      • Choose a server to store the core dump:

         (global)  exception dump   ip-address  

        When the router crashes, the core dump file is written to the server at ip-address. The file is named hostname -core, where hostname is the name of the router. If TFTP is used, only the first 16 MB of the core dump is written. If the router memory is greater than 16 MB, FTP or RCP should be used.

Monitoring Router Activity

  1. Watch IP packets as they are routed.

    1. Create an extended IP access list to identify traffic to watch.

      The access list must permit only the packets that you are interested in seeing. Define the permit conditions as narrowly as you can so that only a small amount of traffic is selected to display. Any traffic that is denied will not be seen in the debugging output. (See Section 14-1 for further information.)

    2. Enable IP packet debugging:

       (global)  debug ip packet detail  [  acc-list  ] 

      The router displays information about IP packets as they are processed. Obviously, this command has the potential to generate great volumes of information. In all cases, you should use the optional access list field, acc-list, to reduce the number of packets being reported. Packets that are permitted by the extended IP access list are displayed in the debugging output. Only packets that are not fast-switched can be examined by the debug command. For this reason, you should use the (interface) no ip route-cache command to first disable fast switching on specific interfaces.

      If the debug output shows "encapsulation failed," this indicates that the packet could not be encapsulated in a lower-layer protocol. In the following example, a ping request packet was queued for the Ethernet 0 interface. Because the router could not find an ARP entry for the target address (10.5.1.5), the ping packet could not be encapsulated in a Layer 2 frame with a destination MAC address. Therefore, debug reported that the encapsulation failed.

       00:20:41:     ICMP type=8, code=0. 00:20:43: IP: s=10.5.1.1 (local), d=10.5.1.5 (Ethernet0), len 53, sending 00:20:43:     ICMP type=8, code=0  00:20:43: IP: s=10.5.1.1 (local), d=10.5.1.5 (Ethernet0), len 53,   encapsulation failed  
  2. Watch the router CPU activity:

     (exec)  show processes cpu  

    The average router CPU utilization is displayed for the last 5 seconds, 1 minute, and 5 minutes. The various running processes are also displayed, along with information that tells you how long each process has been running and its contribution to CPU utilization. Cisco recommends that a router CPU should stay below 70% utilization.

  3. Watch the router memory:

     (exec)  show memory free  

    Statistics regarding free router memory are displayed. This information can be useful in determining whether there is enough available memory after a router initializes, loads its IOS image, and is actively routing traffic.

  4. View statistics about an interface:

     (exec)  show interface  [  interface  ] 

    If the optional interface is not specified, information is displayed about all interfaces on the router.

  5. View information about interfaces by protocol:

     (exec)  show interface accounting  

    Each router interface is listed, along with a breakdown of the bytes and packets inbound and outbound by protocol. This information can be useful to help you see the protocols and traffic volumes that are passing through an interface.

  6. View information about a specific protocol on an interface:

     (exec)  show   protocol   interface   type num  

    For the protocol specified (ip, ipx, appletalk) on the interface, information is presented regarding how the protocol is processed. This includes any access lists that are applied, ICMP behavior, switching paths, and many more configuration parameters.

  7. View summary statistics about a protocol:

     (exec)  show   protocol   traffic  

    Statistics are displayed about the protocol and its major components . Summary counts of the protocol traffic and router activity are shown as totals through all interfaces.

Getting Assistance from Cisco

You can find information about the router using the following command:

 (global)  show version  

Information about the router hardware, bootstrap code, IOS software image version, router uptime, available interfaces, and router memory is displayed. This output is very useful to help you identify the version of code running on the router, as well as the amount of memory and Flash available. You can also view the contents of the router configuration register.

Information for the Cisco Technical Assistance Center (TAC)

  1. Gather the information:

     (global)  show tech-support  

    Output from a large predetermined set of commands is generated. You should capture this data with a terminal emulator so that it can be sent to a TAC engineer.

  2. Open a TAC case.

    If you have a service contract with Cisco for your router, you can open a TAC case either by phone or by Web browser. You can contact the Cisco TAC by phone using a number listed for your location at http://www.cisco.com/warp/customer/687/Directory/DirTAC.shtml.

    To open a case using a browser, go to http://www.cisco.com/kobayashi/support/case_open.shtml and fill in the required information. This also requires a CCO login with a profile that has been updated with your service contract number.

  3. Cisco IOS Software bugs .

    Cisco offers information about Cisco IOS Software versions and bug reports on its CCO Web site. The Software Bug Toolkit consists of Bug Navigator II (an interactive tool that reports bug information for IOS versions and feature sets), Bug Watcher (a tool that allows you to monitor bug information and receive alerts as new bugs are reported), and a tool that allows the bug database to be searched by BugID. These tools are available at http://www.cisco.com/support/bugtools/.



Cisco Field Manual[c] Router Configuration
Cisco Field Manual[c] Router Configuration
ISBN: 1587050242
EAN: N/A
Year: 2005
Pages: 185

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net