Troubleshooting Static Routes

Followers of the assorted American political scandals of the past 30 or so years will have heard a congressional investigator ask the question, "What did he know and when did he know it?" The same question serves an internetworking investigator well. When troubleshooting routing problems, the first step should almost always be to examine the route table. What does the router know? Does the router know how to reach the destination in question? Is the information in the route table accurate? Knowing how to trace a route is essential to successfully troubleshooting an internetwork.

Case Study: Tracing a Failed Route

Figure 3.13 shows a previously configured internetwork, with each router's associated static routes. A problem has been discovered . Devices on subnet 192.168.1.0/27, connected to Pooh's Ethernet interface, can communicate with devices on subnet 10.1.0.0/16 just fine. However, when a ping is sent from Pooh itself to subnet 10.1.0.0/16, the ping fails (Figure 3.14). This seems strange . If packets being routed by Pooh successfully reach their destinations, why do packets originated by the same router fail?

Figure 3.13. Packets from subnet 192.168.1.0/27 to subnet 10.1.0.0/16 are routed correctly, but Pooh itself cannot ping any device on 10.1.0.0/16.

Figure 3.14. A device on subnet 192.168.1.0/27 successfully pings Piglet's Ethernet interface, but pings from Pooh fail.

Addressing this problem requires tracing the route of the ping. First, Pooh's route table is examined (Figure 3.15). The destination address of 10.1.5.1 matches the route entry for 10.0.0.0/8, which (according to the table) is reached via the next -hop address 192.168.1.34 ”one of Eeyore's interfaces.

Figure 3.15. A packet with a destination address of 10.1.5.1 matches the route entry for 10.0.0.0/8 and will be forwarded to the next-hop router at 192.168.1.34.

Next the route table for Eeyore must be examined (Figure 3.16). The destination address 10.1.5.1 matches the entry 10.1.0.0/16, with a next-hop address of 10.4.6.1. This address is one of Tigger's interfaces.

Figure 3.16. 10.1.5.1 matches the entry for 10.1.0.0/16 and will be forwarded to 10.4.6.1.

Figure 3.17 shows Tigger's route table. The destination address matches the entry for 10.1.0.0/16 and will be forwarded to 192.168.1.194, which is at Piglet.

Figure 3.17. 10.1.5.1 matches the entry for 10.1.0.0/16 and will be forwarded to 192.168.1.194.

Piglet's route table (Figure 3.18) reveals that the target network, 10.1.0.0, is directly connected. In other words, the packet has arrived. The target address 10.1.5.1 is Piglet's own interface to that network. Because the path to the address has just been verified as good, we can assume that the ICMP echo packets from Pooh are reaching the destination.

Figure 3.18. The destination network, 10.1.0.0, is directly connected to Piglet.

The next step is to trace the path of the responding ICMP echo reply packets. To trace this path, you need to know the source address of the echo packet ”that address will be the destination address of the echo reply packet. The source address of a packet that is originated from a router is the address of the interface from which the packet was transmitted. ^[6] In this example, Pooh originally forwarded the echo packet to 192.168.1.34. Figure 3.13 shows that the source address of that packet is 192.168.1.33. So this address is the destination address to which Piglet will send the echo reply.

^[6] Unless the Extended Ping utility is used to set the source address to something different.

Referring again to Piglet's route table in Figure 3.18, 192.168.1.33 will match the entry for 192.168.1.0/24 and be forwarded to 192.168.1.193, which is another of Tigger's interfaces. A reexamination of Tigger's route table in Figure 3.17 at first suggests that there is an entry for 192.168.1.0. However, care must be taken to correctly interpret the information that is actually there.

Compare the entries in Tigger's route table for the 10.0.0.0 subnets to those of the 192.168.1.0 subnets. The heading for the former says that 10.0.0.0 is variably subnetted ; in other words, Tigger's static route to subnet 10.4.7.0 uses a 24-bit mask, and the static route to subnet 10.1.0.0 uses a 16-bit mask. The table records the correct mask at each subnet.

The heading for 192.168.1.0 is different. This heading states that Tigger knows of three subnets of 192.168.1.0 and that they all have a mask of 255.255.255.224. This mask will be used on the destination address of 192.168.1.33 to derive a destination network of 192.168.1.32/27. The route table has entries for 192.168.1.64/27, 192.168.1.0/27, and 192.168.1.192/27. There is no entry for 192.168.1.32/27, so the router does not know how to reach this subnet.

The problem, then, is that the ICMP echo reply packet is being dropped at Tigger. One solution is to create another static route entry for network 192.168.1.32, with a mask of 255.255.255.224 and pointing to a next hop of either 192.168.1.65 or 10.4.6.2. Another solution would be to change the mask in the existing static route entry for 192.168.1.0 from 255.255.255.224 to 255.255.255.0.

The moral of this story is that when you are tracing a route, you must consider the complete communication process. Verify not only that the path to a destination is good but also that the return path is good.

Note

Check both the destination and return path when a route fails.

Case Study: A Protocol Conflict

Figure 3.19 shows two routers connected by two Ethernet networks, one of which includes a bridge. This bridge handles traffic for several other links not shown and occasionally becomes congested . The host Milne is a mission-critical server; the network administrator is worried about traffic to Milne being delayed by the bridge, so a static host route has been added in Roo that will direct packets destined for Milne across the top Ethernet, avoiding the bridge.

Figure 3.19. A host route directs packets from Roo to Milne across the top Ethernet, avoiding the occasionally congested bridge.

This solution seems to be logical, but it isn't working. After the static route was added, packets routed through Roo no longer reach the server. Not only that, but packets routed through Kanga no longer reach the server, although no changes were made to that router.

The first step, as always, is to check the route table. Roo's route table (Figure 3.20) indicates that packets with a destination address of 172.16.20.75 will in fact be forwarded to Kanga's E1 interface, as desired. Kanga is directly connected to the destination network, so no more routing should be occurring; a quick check verifies that the Ethernet interfaces are functioning at both Kanga and at Milne.

Figure 3.20. Roo's route table, showing the static host route to Milne via Kanga's E1 interface.

In Figure 3.21, a trace is performed from Roo to Milne, and a symptom is found. Instead of delivering the packets to Milne, Kanga is forwarding them to Roo's E0 interface. Roo forwards the packets to Kanga's E1 interface, and Kanga sends the packets right back to Roo. A routing loop seems to have occurred, but why?

Figure 3.21. A trace from Roo to Milne reveals that Kanga is forwarding the packets back to Roo instead of delivering them to the correct destination.

The suspicious aspect of all of this is that Kanga should not be routing the packet, which appears to be the case. Kanga should recognize that the destination address of the packet is for its directly connected network 172.16.20.0 and should be using the data link to deliver the packet to the host. Therefore, suspicion should fall on the data link. Just as the route table should be examined to see whether the router has the correct information for reaching a network across a logical path, the ARP cache should be examined to see whether the router has the correct information for reaching a host across a physical path.

Note

Check a router's ARP cached when packets do not go to the correct destination on a data link.

Figure 3.22 shows the ARP cache for Kanga. The IP address for Milne is in Kanga's cache as expected, with a MAC identifier of 00e0.1e58.dc39. When Milne's interface is checked, though, it shows that it has a MAC identifier of 0002.6779.0f4c; Kanga has somehow acquired incorrect information.

Figure 3.22. Kanga's ARP cache has an entry for Milne, but the associated data link identifier is wrong.

Another look at Kanga's ARP table reveals that the MAC identifier associated with Milne is suspiciously similar to the MAC identifier of Kanga's own Cisco interfaces (the MAC addresses with no ages associated with them are for the router's interfaces). Because Milne is not a Cisco product, the first three octets of its MAC identifier should be different from the first three octets of Kanga's MAC identifier. The only other Cisco product in the internetwork is Roo, so its ARP cache is examined (Figure 3.23); 00e0.1e58.dc39 is the MAC identifier of Roo's E0 interface.

Figure 3.23. Roo's ARP cache shows that the MAC identifier that Kanga has for Milne is actually Roo's E0 interface.

So Kanga mistakenly believes that the E0 interface of Roo is Milne. It frames packets destined for Milne with a destination identifier of 00e0.1e58.dc39; Roo accepts the frame, reads the destination address of the enclosed packet, and routes the packet back to Kanga.

But how did Kanga get the wrong information? The answer is proxy ARP. When Kanga first receives a packet for Milne, it will ARP for that device's data link identifier. Milne responds, but Roo also hears the ARP Request on its E0 interface. Because Roo has a route to Milne on a different network from the one on which it received the ARP Request, it issues a proxy ARP in reply. Kanga receives Milne's ARP reply and enters it into the ARP cache. The proxy ARP reply from Roo arrives later because of the delay of the bridge. The original ARP cache entry is overwritten by what Kanga thinks is new information.

There are two solutions to the problem. The first is to turn off proxy ARP at Roo's E0 with the command:

 
 Roo(config-if)#  interface e0  
 
 Roo(config)#  no ip proxy-arp  
 

The second solution is to configure a static ARP entry for Milne at Kanga with the command:

 
 Kanga(config)#  arp 172.16.20.75 0002.6779.0f4c arpa  
 

This entry will not be overwritten by any ARP reply. Figure 3.24 shows the static ARP entry being entered and the resulting ARP cache at Kanga. Note that because the entry is static, no age is associated with it.

Figure 3.24. A static ARP entry corrects the problem caused by proxy ARP.

The circumstances of the network should help determine which of the two solutions is best. A static ARP entry means that if the network interface at Milne is ever replaced , someone must remember to change the ARP entry to reflect the new MAC identifier. On the other hand, turning off proxy ARP is a good solution only if no hosts make use of it.