After speaking with our network design team, we received the correct OSPF authentication key and placed it on Serial0.2 and Serial1.2 on ROUTER A. Almost instantly, the OSPF adjacencies were formed and the states were FULL. The output that follows shows the relevant configuration modification and resulting OSPF events.
ROUTER_A#conf t Enter configuration commands, one per line. End with CNTL/Z. ROUTER_A(config)#int s0.2 ROUTER_A(config-subif)#ip ospf authentication-key secretkey ROUTER_A(config-subif)# OSPF: Receive dbd from 22.214.171.124 seq 0x2503 OSPF: 2 Way Communication to neighbor 126.96.36.199 OSPF: send DBD packet to 188.8.131.52 seq 0x22C3 OSPF: NBR Negotiation Done We are the SLAVE OSPF: send DBD packet to 184.108.40.206 seq 0x2503 OSPF: Receive dbd from 220.127.116.11 seq 0x2504 OSPF: send DBD packet to 18.104.22.168 seq 0x2504 OSPF: Database request to 22.214.171.124 OSPF: sent LS REQ packet to 126.96.36.199, length 864 OSPF: Receive dbd from 188.8.131.52 seq 0x2505 OSPF: send DBD packet to 184.108.40.206 seq 0x2505 OSPF: Database request to 220.127.116.11 OSPF: sent LS REQ packet to 18.104.22.168, length 1080 OSPF: Receive dbd from 22.214.171.124 seq 0x2506 OSPF: Exchange Done with neighbor 126.96.36.199 OSPF: send DBD packet to 188.8.131.52 seq 0x2506 OSPF: Synchronized with neighbor 184.108.40.206, state:FULL OSPF_ROUTER_A(config-subif)# OSPF: Neighbor 220.127.116.11 is dead OSPF: neighbor 18.104.22.168 is dead, state DOWN OSPF: Tried to build Router LSA within MinLSInterval OSPF: Rcv pkt from 22.214.171.124, Serial1.2 : Mismatch Authentication Key - Clear Textint s1.2 OSPF_ROUTER_A(config-subif)#ip ospf authentication-key secretkey OSPF: Rcv pkt from 126.96.36.199, Serial1.2 : Mismatch Authentication Key - Clear Text OSPF_ROUTER_A(config-subif)# OSPF: 2 Way Communication to neighbor 188.8.131.52 OSPF: send DBD packet to 184.108.40.206 seq 0xCC5 OSPF: Receive dbd from 220.127.116.11 seq 0x794 OSPF: NBR Negotiation Done We are the SLAVE OSPF: send DBD packet to 18.104.22.168 seq 0x794 OSPF: Receive dbd from 22.214.171.124 seq 0x795 OSPF: send DBD packet to 126.96.36.199 seq 0x795 OSPF: Receive dbd from 188.8.131.52 seq 0x796 OSPF: send DBD packet to 184.108.40.206 seq 0x796 OSPF: Receive dbd from 220.127.116.11 seq 0x797 OSPF: Exchange Done with neighbor 18.104.22.168 OSPF: Synchronized with neighbor 22.214.171.124, state:FULL
Step 6 Revisited Again: Gather Results
We then confirmed that the OSPF adjacency STATES were now correct because they both were FULL. This meant that the OSPF link-state databases for all routers were 100 percent synchronized with each other, and, most importantly, routing was now working correctly as shown in the following output.
ROUTER_A#show ip ospf neighbor Neighbor ID Pri State Dead Time Address Interface 126.96.36.199 1 FULL/DR 00:00:31 188.8.131.52 Ethernet0 184.108.40.206 1 FULL/ - 00:00:39 220.127.116.11 Serial0.2 18.104.22.168 1 FULL/ - 00:00:30 22.214.171.124 Serial1.2
By executing some trace routes, we confirmed that the primary links (768K Frame Links) were now routing traffic through the proper primary path, namely Router A. The final relevant OSPF router configuration for Router A was as follows:
! router ospf 202 network 126.96.36.199 0.0.0.255 area 0 network 188.8.131.52 0.0.0.255 area 184.108.40.206 area 220.127.116.11 authentication area 18.104.22.168 stub !
This was certainly a solid fix relating to the originally reported routing problem. However, users were still complaining of a slowdown.
Problem #2: Performance Issues
In Step 3, we identified that there were, in fact, two problems being seen in the network. These two problems (routing and performance) were causing the customers slow downs. At this point we have successfully resolved problem #1 and its routing issues. So we must now resolve the second problem: network performance.
Step 1: Define the Problem
At this point, the network was routing according to design, but there still was a high percentage of traffic coming into Router A from the LAN segment Downtown. Users Downtown were still complaining of a slowdown, so we had a clear problem definition. We used the troubleshooting model again to develop and implement an action plan.
Step 2: Gather Facts
We observed that there were still a significant amount of output drops on Router C and Router D. The following output is for Router C:
ROUTER_C#show int s0/0:0 Serial0/0:0 is up, line protocol is up Hardware is DSX1 Description: Frame Relay Circuit to Headquarters MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec, rely 255/255, load 6/255 Encapsulation FRAME-RELAY IETF, loopback not set, keepalive set (10 sec) LMI enq sent 16732, LMI stat recvd 16732, LMI upd recvd 0, DTE LMI up LMI enq recvd 0, LMI stat sent 0, LMI upd sent 0 LMI DLCI 0 LMI type is ANSI Annex D frame relay DTE Broadcast queue 0/64, broadcasts sent/dropped 983815/57443, interface broadcasts 1035677 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters 1d22h Input queue: 0/75/48 (size/max/drops); Total output drops: 1500632 Queueing strategy: weighted fair Output queue: 0/64/19 (size/threshold/drops) Conversations 0/51 (active/max active) Reserved Conversations 0/0 (allocated/max allocated) 5 minute input rate 50000 bits/sec, 19 packets/sec 5 minute output rate 42000 bits/sec, 8 packets/sec 1493015 packets input, 320768751 bytes, 0 no buffer Received 0 broadcasts, 2 runts, 0 giants, 0 throttles 48 input errors, 35 CRC, 13 frame, 0 overrun, 0 ignored, 2 abort 2335606 packets output, 845399484 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets
This output is very useful in troubleshooting suspected utilization issues. The statistics that are presented here will show, for example, if the circuit is taking errors. Additionally, it will describe what kinds of errors the circuit is seeing. In this case, the output showed a large number of output drops.
The vast number of output drops was a signal that there was a lot of unnecessary traffic being routed across the WAN links. Essentially, a vicious cycle of repeated retransmits was formed, further aggravating the problem.