At one point or another, every administrator must troubleshoot a user 's complaint of not being able to reach some destination on the network. Lack of connectivity can be the result of network outages caused by WAN service failures, misconfiguration of routers and other devices in the network, access list controls (whether intentional or otherwise ), and myriad other possibilities. Although there is no substitute for network test equipment, such as protocol analyzers, the router does provide several very useful tools for verifying IP connectivity and investigating potential problems. Let's examine a few of those tools.
As mentioned earlier, a router should have a specific route or some kind of default or summary route to every destination that an IP station might try to reach. One of the best tools for troubleshooting is the show ip route command, examined earlier in this chapter. When a station is having trouble reaching other stations ”either within or outside the intranet ”one of the first troubleshooting steps is to verify that the router closest to the user has a route to the destination IP address. If a specific route is not found, or if an expected default or summary route is not present, you probably need to investigate the dynamic routing protocols to determine why the route is not present. The reason could be obvious, such as a network segment failure (for example, a WAN service failure), or more subtle, such as a minor misconfiguration on another router in the network.
If you establish that a route to the desired destination exists, you should test to determine whether the router can reach the destination. UNIX users are familiar with the ping command, which is an acronym for the Packet Internet Groper. The ping command, which is implemented on the router, makes use of the IP Control Message Protocol (ICMP) to send echo requests to a destination IP address. The station receiving the ICMP echo request sends back an ICMP echo-response. In this way, a source station can determine whether a destination station is reachable and approximately how long the echo request and reply take to reach and return from the destination station. The following is an example of the IOS EXEC command ping being used on the ZIP SF-Core-1 to test the reachability of the San Jose router:
SF-Core-1# ping 126.96.36.199 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 188.8.131.52, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 25/25/25 ms SF-Core-1#
The router sends five ICMP echo requests and reports via the exclamation point (!) that all the replies are received. It also reports the number of echo request attempts and the number of echo replies received, and it calculates the percentage of successful ping s. Minimum, maximum, and average response times are also calculated.
When a router ping s an IP address for the first time or again after a long period of time, the router typically does not receive the first echo reply, resulting in four out of five ping responses. This is because the router must wait for the ARP resolution of the IP address before sending the echo request. Normally, the ARP reply does not arrive in time for the first echo request to be sent and the reply to be received before the request times out.
Table 4-6 shows the different response characters that can be received as a result of a ping .
Table 4-6. ping Command Response Characters
The ping command has both a privileged and an unprivileged version. In user EXEC mode, the unprivileged version allows the user to only specify an IP address. The privileged version, available in enable EXEC mode, enables the user to modify parameters of the echo request, including the number of requests, the size of the packets sent, the timeout value, the source IP address of the request, the pattern of data in the echo request, and numerous other values. The following is an example of the privileged version of the ping command executed on the SF-Core-1 router. In this example, the source address has been specified as the IP address on the Fast Ethernet interface, the destination is the address 184.108.40.206 on the San-Jose router, and the packet size is 1500 bytes:
SF-Core-1# ping Protocol [ip]: Target IP address: 220.127.116.11 Repeat count : Datagram size : 1500 Timeout in seconds : Extended commands [n]: y Source address or interface: 18.104.22.168 Type of service : Set DF bit in IP header? [no]: Validate reply data? [no]: Data pattern [0xABCD]: Loose, Strict, Record, Timestamp, Verbose[none]: Sweep range of sizes [n]: Type escape sequence to abort. Sending 5, 1500-byte ICMP Echos to 22.214.171.124, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 29/29/29 ms SF-Core-1#
If you suspect that lack of connectivity is because of a missing route on a downstream router or an incorrect path that a packet is taking, the router has a command called trace that enables you to verify the path that a packet travels to reach a destination IP address. The trace function is similar to the UNIX traceroute utility. As with the ping command, the IOS EXEC command trace has both a privileged and an unprivileged version. The unprivileged version allows the user to only supply an IP destination address, while the privileged version enables the user to modify parameters, just as with the ping command.
The trace function makes use of the ICMP "TTL- Expired " (Time To Live) message to identify routers in the path to a destination IP address. The source router sends a UDP packet with a TTL of 1 toward the destination. The first router in the path receives the packet and decrements the TTL field by 1. As a result, the TTL expires (goes to 0), and the router does not forward the packet. Instead, this first router in the path sends an ICMP"TTL-Expired" message back to the source of the packet so that the source now knows the first router hop in the path.
The source router now sends another UDP packet, but it sets the TTL to 2. The first router in the path receives the packet, decrements the TTL to 1, and forwards the packet to the second router in the path. The second router receives the packet, decrements the TTL to 0, and does not forward the packet because the TTL has expired. The second router sends an ICMP "TTL-Expired" message back to the originating station, and now the source router knows the second router in the path. This process is repeated until the packet reaches the ultimate destination IP address. The packet is addressed to a high-number UDP port, usually above 33434, which the destination device does not support. Therefore, the destination IP address responds with an ICMP "Port Unreachable" message, which alerts the source router that the final destination has been reached.
The following is an example of the trace command on the ZIP SF-Core-1 router requesting the path to a station off of Seoul-1 router:
SF-Core-1# trace 126.96.36.199 Type escape sequence to abort. Tracing the route to testy.zipnet.com (188.8.131.52) 1 s0/0-SanJose-sj.zipnet.com (184.108.40.206) 25 msec 25 msec 25 msec 2 s1-Seoul1-kr.zipnet.com (220.127.116.11) 176 msec * 176 msec 3 testy.zipnet.com (18.104.22.168) 178 msec 178 msec 178 msec SF-Core-1#
In the preceding example, time values are displayed after the name and IP addresses of the routers in the network path. These values are an approximation of the round-trip time from the source address to the router in the path. As many as three time values ”one for each of the three packets (probes) ”are displayed for each destination IP address. Some devices have limitations on the rate at which they can respond with ICMP messages. For such a device, fewer than three time values may appear. For each probe that the device does not respond to because of rate limitations, an asterisk is shown in place of the time value. You can see an example of this in the preceding output. The second-hop router was not capable of responding to the second probe, as indicated by the asterisk. Cisco IOS-based devices rate-limit ICMP responses to one per second.
In addition to rate-limiting ICMP messages, some routers in the path may not respond with an ICMP "TTL-Expired" message. Some may reuse the TTL of the incoming packet, which results in the expiration of the ICMP message's TTL before the message can return to the sender. And in some cases, packet filtering may prevent the ICMP response packets from reaching the source router. In all these cases, a line of asterisks is seen in an output line instead of address information. In the following trace output, the second router in the path has failed to respond to the trace queries:
SF-Core-1# trace 22.214.171.124 Type escape sequence to abort. Tracing the route to testy.zipnet.com (126.96.36.199) 1 s0/0-SanJose-sj.zipnet.com (188.8.131.52) 25 msec 25 msec 25 msec 2 * * * 3 testy.zipnet.com (184.108.40.206) 178 msec 178 msec 178 msec SF-Core-1#
The privileged version of the trace command allows for adjusting the command's parameters, including whether IP addresses are reverse resolved to host names , the number of probes sent for each TTL step, a minimum and maximum TTL value, and so on. The following is a previous trace example repeated in privileged trace mode in which only the numeric responses are displayed:
SF-Core-1# trace Protocol [ip]: Target IP address: 220.127.116.11 Source address: Numeric display [n]: y Timeout in seconds : Probe count : Minimum Time to Live : Maximum Time to Live : Port Number : Loose, Strict, Record, Timestamp, Verbose[none]: Type escape sequence to abort. Tracing the route to 18.104.22.168 1 22.214.171.124 25 msec 25 msec 25 msec 2 126.96.36.199 176 msec * 176 msec 3 188.8.131.52 178 msec 178 msec 178 msec SF-Core-1#
If a station that is reachable via a directly connected LAN interface is not responding, the reason may be that the router is not capable of resolving the IP address to the MAC address. To verify the MAC addresses that the router has been capable of resolving, use the Cisco IOS EXEC command show ip arp . This command takes as a parameter either a specific IP address, a specific interface, or a specific 48-bit MAC address. It displays only the ARP entries for that parameter. When no parameter is supplied, all IP ARP entries are displayed. The output of the command includes the IP-to-ARP mapping, the age of the entry in the table, and the interface that the ARP entry is associated with. (The router times out an ARP entry from the ARP table after four hours by default.) The following is an example of the show ip arp command on the ZIP SF-Core-1 router:
SF-Core-1# show ip arp Protocol Address Age (min) Hardware Addr Type Interface Internet 131.108.20. - 0000.0c07.b627 ARPA FastEthernet0/0 Internet 184.108.40.206 4 0000.0c67.b62c ARPA FastEthernet0/0 Internet 220.127.116.11 2 0000.0cf1.a9c1 ARPA FastEthernet0/0 Internet 18.104.22.168 12 0000.0cb8.02bc ARPA FastEthernet0/0 Internet 22.214.171.124 0 Incomplete ARPA SF-Core-1#
In the preceding example, the ARP table entry for 126.96.36.199 shows the word incomplete instead of an actual MAC hardware address, which indicates that the router sent an ARP request but that no reply was received to complete the ARP table entry. In this case, we can assume either that no station exists with this address or that the station is incapable of responding, perhaps because it is powered off.
Overall statistics about the operation of the IP protocol on the router can be obtained from the show ip traffic command. It includes counters for such information as the total number of packets received and sent by the router, the number of broadcasts received and sent, ICMP/UDP/TCP protocol statistics, and much more. These statistics can aid in determining whether the router has sent or received an ICMP echo, whether an IP address fails to resolve to a MAC address (known as an encapsulation failure), and where certain routing protocol packets are being received or sent. The counters in show ip traffic are cumulative, and they are reset only when the router is reloaded or power-cycled. The following is an example of the output of the show ip traffic command on the ZIP SF-Core-1 router:
SF-Core-1# show ip traffic IP statistics: Rcvd: 4686565 total, 2623438 local destination 0 format errors, 0 checksum errors, 77 bad hop count 0 unknown protocol, 1 not a gateway 0 security failures, 0 bad options, 0 with options Opts: 0 end, 0 nop, 0 basic security, 0 loose source route 0 timestamp, 0 extended security, 0 record route 0 stream ID, 0 strict source route, 0 alert, 0 other Frags: 0 reassembled, 0 timeouts, 0 couldn't reassemble 0 fragmented, 0 couldn't fragment Bcast: 5981 received, 0 sent Mcast: 2482184 received, 3581861 sent Sent: 3893477 generated, 2062048 forwarded 954 encapsulation failed, 208 no route ICMP statistics: Rcvd: 0 format errors, 0 checksum errors, 5 redirects, 5070 unreachable 3 echo, 16 echo reply, 0 mask requests, 0 mask replies, 0 quench 0 parameter, 0 timestamp, 0 info request, 0 other 0 irdp solicitations, 0 irdp advertisements Sent: 0 redirects, 18050 unreachable, 66 echo, 3 echo reply 0 mask requests, 0 mask replies, 0 quench, 0 timestamp 0 info reply, 7 time exceeded, 0 parameter problem 0 irdp solicitations, 0 irdp advertisements UDP statistics: Rcvd: 52836 total, 4 checksum errors, 18085 no port Sent: 50699 total, 5949 forwarded broadcasts TCP statistics: Rcvd: 47895 total, 0 checksum errors, 1 no port Sent: 46883 total Probe statistics: Rcvd: 0 address requests, 0 address replies 0 proxy name requests, 0 where-is requests, 0 other Sent: 0 address requests, 0 address replies (0 proxy) 0 proxy name replies, 0 where-is replies EGP statistics: Rcvd: 0 total, 0 format errors, 0 checksum errors, 0 no listener Sent: 0 total IGRP statistics: Rcvd: 0 total, 0 checksum errors Sent: 0 total OSPF statistics: Rcvd: 0 total, 0 checksum errors 0 hello, 0 database desc, 0 link state req 0 link state updates, 0 link state acks Sent: 0 total IP-IGRP2 statistics: Rcvd: 2105381 total Sent: 3140121 total PIMv2 statistics: Sent/Received Total: 0/0, 0 checksum errors, 0 format errors Registers: 0/0, Register Stops: 0/0 IGMP statistics: Sent/Received Total: 0/0, Format errors: 0/0, Checksum errors: 0/0 Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 00 DVMRP: 0/0, PIM: 0/0 ARP statistics: Rcvd: 8540 requests, 4 replies, 0 reverse, 0 other Sent: 89 requests, 9018 replies (0 proxy), 0 reverse SF-Core-1#
The counters in the output of show ip traffic count both events that have occurred and types of packets that have been sent and received. If the encapsulation failed counter were increasing, it would indicate that the router did not receive ARP replies to its ARP requests for packets that were attempting to be switched to destination interfaces and that the packets were discarded. The ICMP echo count indicates how many pings the router is generating, and the echo reply count indicates the number of pings to which it is responding.
In addition to the troubleshooting and verification commands presented in this section, numerous IOS EXEC debug commands exist to aid in determining the operation of the IP on the router. These debug commands provide both general and detailed diagnostic output that can aid in troubleshooting problems and in verifying the operation of the router, routing protocols, and other functions. Some of the more common debug commands used for TCP/IP are summarized in Table 4-7.
Table 4-7. Debug Commands for IP
Debugging commands for the various dynamic routing protocols include debug ip rip, debug ip eigrp, debug ip igrp, debug ip ospf , and debug ip bgp . Each of these debugging commands has optional parameters that control what debugging information about the routing protocol is displayed to the user. Care should be exercised when using some versions of these commands, because they can be CPU- intensive . For a complete description of all debugging commands and sample output, refer to the Cisco Connection Documentation CD-ROM or to the online version found at http://www.cisco.com/univercd/home/home. htm.
When using debug commands that are known to increase CPU load, do not execute them on the console port. Instead, disable console logging via the IOS global configuration command no logging console , and enable buffered logging via the IOS global configuration command logging buffered . Then execute the command from a virtual terminal session and view the output in that session. If the session becomes unresponsive , the console can be used to disable the debug because the console has higher priority than the virtual terminal session. The debug output can then be reviewed in the log buffer via the IOS EXEC command show log . If syslogging is enabled, the output can also be viewed in the log file on the syslog server.