Scenario 1: New Install Issues

To troubleshoot new install issues, you must clearly understand what stage of the installation process you are troubleshooting. The most common scenario is when the provisioning process is complete, and the confirmation of circuit installation is received from the service provider, which indicates that the termination device is installed in the minimum point of entry (MPOE), or the sometimes-called Demarcation Point (D-mark). Also as part of the scenario, the local loop is checked and the technician reports that the line is installed.

The first situation here is when you have an administratively down situation for interfaces. A user might have forgotten to not shut the interface (shutdown is the default condition for a new interface that has not been configured). However, this is a basic step that most engineers won't miss to unshut the interface. If the end user's router configuration is correct, three typical indicators exist for new install issues:

• The line is down, the protocol is down.

• The line is up, the protocol is down.

• The line is up, the protocol is up, the router cannot pass data.

NOTE

In explanations throughout this chapter, you see references to up and down states. When there are two references to these states (such as up, up), the first is referring to the line status and the second is referring to the protocol layer status.

Example 18-1 shows the first case. Example 18-2 shows the second case. Finally, Example 18-3 shows the third case.

In the first case, if the line to the MPOE is reported as operational, you might be dealing with internal wiring problems (IW in the local exchange carrier's [LEC's] terminology) or router cable problems. Review the information pertaining to these problems in Chapter 17.

Example 18-1. The Line Is Reported Down, the Protocol Layer Is Down

 1602-frame#show ip interface brief Interface                  IP-Address      OK? Method Status         Protocol Ethernet0                  10.84.14.169    YES NVRAM  up             up Serial0                    unassigned      YES NVRAM  down           down Serial0.37                 10.84.14.169    YES unset  down           down 1602-frame# 

Example 18-2. The Line Is Reported Up, the Protocol Layer Is Down

 1602-frame#show ip interface brief Interface                    IP-Address      OK? Method  Status      Protocol Ethernet0                    10.84.14.169    YES NVRAM   up          up Serial0                      unassigned      YES NVRAM   up          down Serial0.62                   10.84.14.169    YES unset   down        down 1602-frame# 

Example 18-3. The Line Is Reported Up, the Protocol Layer Is Up, but the Router Cannot Pass Data

 1602-frame#show ip interface brief Interface                  IP-Address      OK? Method  Status        Protocol Ethernet0                  10.84.14.169    YES NVRAM   up            up Serial0                    unassigned      YES NVRAM   up            up Serial0.62                 10.84.14.169    YES unset   up            up 1602-frame# 

A more interesting case for troubleshooting purposes is the second case, where the line is reported up but the protocol layer is reported down. An initial checklist of possible issues include (following the described methodology in Chapter 16, "Basic and Advanced Frame Relay Configurations"):

• Configuration issues:

  - Local router configuration errors

  - Core router configuration errors

  - Existing loopback from the core side

• Service provider issues:

  - The permanent virtual circuit (PVC) is not mapped

  - The switch is in maintenance mode

  - The access rate requirements do not match

  - The Local Management Interface (LMI) type does not match

NOTE

For the purposes of the explanation, both sides are configured to use data-link connection identifier (DLCI) = 62; however, the number does not have to be the same on both ends because DLCI only has a local significance.

Starting with the configuration issues, when analyzing both sides, you see the following.

The remote user's router is configured correctly for 56 kbps, it is defined as Frame Relay Internet Engineering Task Force (IETF) encapsulation, and it is set up for LMI type AnnexD. It also reports the output listed in Example 18-2 with line up and protocol down. The remote user's router is configured as shown in Example 18-4.

Example 18-4. Fragment of the Remote User Configuration

 1602-frame#show running-config <output omitted> interface Serial0  no ip address ! The Frame Relay encapsulation is IETF:  encapsulation frame-relay IETF  service-module 56k clock source line  service-module 56k network-type dds ! The LMI type is ANSI:  frame-relay lmi-type ansi ! interface Serial0.62 point-to-point  bandwidth 56  ip unnumbered Ethernet0 ! The encapsulation of DLCI 62 is IETF because the Serial0 is IETF  frame-relay interface-dlci 62 <output omitted> 

The core router reports the output shown in Example 18-5.

Example 18-5. Core Router Output

 3640-frame#show ip interface brief Interface                IP-Address      OK? Method Status    Protocol FastEthernet0/0          10.84.5.111     YES manual up        up FastEthernet1/0          unassigned      YES NVRAM  up        up Serial3/2:0.65           unassigned      YES unset  deleted   down Serial3/2:0.66           10.84.5.111     YES unset  up        up <output omitted> Serial3/3:0.59           10.84.5.111     YES unset  up        up Serial3/3:0.60           10.84.5.111     YES unset  down      down Serial3/3:0.61           10.84.5.111     YES unset  up        up Serial3/3:0.62           10.84.5.111     YES unset  up        up ! The connection serial3/3:0.62  is the one that requires troubleshooting: Serial3/3:0.70           unassigned      YES unset  deleted   down Serial3/3:0.138          10.84.5.111     YES unset  up        up Serial4/0:0              unassigned      YES NVRAM  up        up Serial4/0:0.17           unassigned      YES unset  deleted   down 

The output in Example 18-6 shows the status of DLCI = 62 on the core router.

Example 18-6. From the Core Router Perspective, DLCI 62 Is Reported Up, Up

 3640-frame#show interfaces serial 3/3:0.62 Serial3/3:0.62 is up, line protocol is up   Hardware is Multichannel T1   Description: frame to 1602-frame: 10.84.14.169   Interface is unnumbered. Using address of FastEthernet0/0 (10.84.5.222)   MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,      reliability 255/255, txload 6/255, rxload 1/255   Encapsulation FRAME-RELAY IETF 3640-frame# 

The service from the core side is configured as point-to-point with unnumbered IP, as shown in Example 18-7.

Example 18-7. How to Check if the Configuration Is Correct

 3640-frame#show running-config interface serial s3/3:0.62 ! The configuration of the serial interface matches the ! Encapsulation Frame Relay IETF. Building configuration... Current configuration: 236 bytes ! ! Matches the remote user's router settings: interface Serial3/3:0.62 point-to-point  description frame-relay service for 1602-frame: 10.84.14.169 ! The bandwidth matches:  bandwidth 56  ip unnumbered FastEthernet0/0  no ip route-cache ! DLCI IETF matches the local switch:  frame-relay interface-dlci 62 IETF end 3640-frame# 

Obviously, the core side is connected to the local switch and the User-Network Interface (UNI) is functioning correctly.

NOTE

The way the loopback is set up can affect the way the router reports it. The core router can report the line and protocol up for remote loopback, or if you set a local loopback, it can report looped instead of up. However, if the service provider's test technician sets loopback on the interface, the core router does not indicate loopback and the switch must be checked.

Start with the remote user's router to identify what it is reporting (see Example 18-8).

Example 18-8. Debugging the LMI

 1602-frame#debug frame-relay lmi Frame Relay LMI debugging is on Displaying all Frame Relay LMI data 1602-frame# ! The DTE sends myseq 55. It does not receive anything from the switch. ! The DTE is down: 3d07h: Serial0(out): StEnq, myseq 55, yourseen 0, DTE down 3d07h: datagramstart = 0x2B9CD48, datagramsize = 14 ! The Frame Relay encapsulation is 0x00010308 (Cisco), (0xFCF10309 IETF): 3d07h: FR encap = 0x00010308 3d07h: 00 75 95 01 01 00 03 02 37 00 3d07h: ! The DTE sends myseq 56. It does not receive anything from the switch. ! The DTE is down: 3d07h: Serial0(out): StEnq, myseq 56, yourseen 0, DTE down 3d07h: datagramstart = 0x2B9CD48, datagramsize = 14 3d07h: FR encap = 0x00010308 3d07h: 00 75 95 01 01 00 03 02 38 00 3d07h: ! The DTE sends myseq 57. It does not receive anything from the switch. ! The DTE is down: <output omitted> 

3d07h: Serial0(out): StEnq, myseq 57, yourseen 0, DTE downChange the encapsulation to high-level data link control (HDLC) and try again (see Example 18-9).

Example 18-9. Debug of the Serial Interface, After Changing the Encapsulation to HDLC

 1602-frame-frame# debug serial interface Serial network interface debugging is on 1602-frame# 3d07h: Serial0: attempting to restart 3d07h: QUICC_SERIAL(0): Reset from 0x200C8FE 3d07h: Serial0(out): StEnq, myseq 62, yourseen 0, DTE down 3d07h: Serial0(out): StEnq, myseq 63, yourseen 0, DTE down 3d07h: Serial0(out): StEnq, myseq 64, yourseen 0, DTE down ! Increase the number of errors; restart the interface: 3d07h: Serial0: attempting to restart 3d07h: QUICC_SERIAL(0): Reset from 0x200C8FE 3d07h: Serial0(out): StEnq, myseq 65, yourseen 0, DTE down 3d07h: Serial0(out): StEnq, myseq 66, yourseen 0, DTE down 3d07h: Serial0(out): StEnq, myseq 67, yourseen 0, DTE down ! Increase the number of errors; restart the interface: 3d07h: Serial0: attempting to restart 3d07h: QUICC_SERIAL(0): Reset from 0x200C8FE 3d07h: Serial0(out): StEnq, myseq 68, yourseen 0, DTE down 1602-frame# 

Change the encapsulation back to Frame Relay because the report is the same. When working with the service provider, the technician explicitly defines the LMI type from none to ANSI to match the remote user's router settings. As a result, the message in Example 18-10 on the screen shows the protocol coming back up.

Example 18-10. Changing the LMI Type to ANSI Brings the Protocol Layer to Up

 1602-frame#show ip interface brief Interface                  IP-Address      OK? Method Status         Protocol Ethernet0                  10.84.14.169    YES NVRAM  up             up Serial0                    unassigned      YES NVRAM  up             down Serial0.62                 10.84.14.169    YES unset  down           down 1602-frame# ! The line protocol changed state to up: 3d07h: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to up 1602-frame# 

Consequently, the output in Example 18-11 is shown.

Example 18-11. Checking the Status of the Interfaces of a Remote User's Router

 1602-frame#show ip interface brief Interface             IP-Address       OK? Method   Status        Protocol Ethernet0             10.84.14.169     YES NVRAM    up            up Serial0               unassigned       YES NVRAM    up            up Serial0.62            10.84.14.169     YES unset    up            up 

The last thing to do is to test the new service, as shown in Example 18-12.

Example 18-12. Check if the Service Is Functioning Properly and Check the LMI Reports

 1602-frame#show interfaces <output omitted> Serial0 is up, line protocol is up   Hardware is QUICC Serial (with onboard CSU/DSU)   MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,      reliability 255/255, txload 1/255, rxload 1/255   Encapsulation FRAME-RELAY IETF, loopback not set   Keepalive set (10 sec)   LMI enq sent  28694, LMI stat recvd 20, LMI upd recvd 0, DTE LMI up   LMI enq recvd 0, LMI stat sent  0, LMI upd sent  0   LMI DLCI 0  LMI type is ANSI Annex D  frame relay DTE   FR SVC disabled, LAPF state down   Broadcast queue 0/64, broadcasts sent/dropped 5/0, interface broadcasts 1   Last input 00:00:00, output 00:00:00, output hang never   Last clearing of "show interfaces" counters 3d07h   Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0   Queuing strategy: weighted fair   Output queue: 0/1000/64/0 (size/max total/threshold/drops)      Conversations  0/23/256 (active/max active/max total)      Reserved Conversations 0/0 (allocated/max allocated)   5 minute input rate 3000 bits/sec, 6 packets/sec   5 minute output rate 4000 bits/sec, 6 packets/sec      68159 packets input, 5690460 bytes, 0 no buffer      Received 0 broadcasts, 0 runts, 0 giants, 0 throttles      2629 input errors, 0 CRC, 2615 frame, 0 overrun, 0 ignored, 14 abort ! These errors were displayed when the line was down. ! Now you can ignore them because they are not increasing; ! or you may want to clear the counters and monitor again.      29578 packets output, 497773 bytes, 0 underruns      0 output errors, 0 collisions, 9558 interface resets      0 output buffer failures, 0 output buffers swapped out      2 carrier transitions      DCD=up  DSR=up  DTR=up  RTS=up  CTS=up Serial0.62 is up, line protocol is up   Hardware is QUICC Serial (with onboard CSU/DSU)   Interface is unnumbered. Using address of Ethernet0 (10.84.14.169)   MTU 1500 bytes, BW 56 Kbit, DLY 20000 usec,      reliability 255/255, txload 1/255, rxload 1/255   Encapsulation FRAME-RELAY IETF 1602-frame# 1602-frame#show frame-relay lmi LMI Statistics for interface Serial0 (Frame Relay DTE) LMI TYPE = ANSI   Invalid Unnumbered info 0             Invalid Prot Disc 0   Invalid dummy Call Ref 0              Invalid Msg Type 0   Invalid Status Message 0              Invalid Lock Shift 0   Invalid Information ID 0              Invalid Report IE Len 0   Invalid Report Request 0              Invalid Keep IE Len 0 ! All invalid message counters are reporting 0s   Num Status Enq. Sent 28699            Num Status msgs Rcvd 28699   Num Update Status Rcvd 0              Num Status Timeouts 0 1602-frame# 

Next, run a ping test to check if the router can pass IP traffic (see Example 18-13).

Example 18-13. The Ping Test Shows that the Router Passes IP Traffic

 3640-frame#ping 10.84.14.169 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 10.84.14.169, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 36/40/48 ms 3640-frame#ping Protocol [ip]: Target IP address: 10.84.14.169 Repeat count [5]: 50 Datagram size [100]: 1000 Timeout in seconds [2]: Extended commands [n]: Sweep range of sizes [n]: Type escape sequence to abort. Sending 50, 1000-byte ICMP Echos to 10.84.14.169, timeout is 2 seconds: !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! 

The third case occurs when the interfaces of the remote user report line up, protocol up, but the remote user cannot pass any traffic (ping, telnet, and so on) between the home network and the corporate network. This might be related to a variety of problems, including the following:

• The router on the other end is not configured or not configured properly.

• The router network addressing scheme is incorrect.

• The routing protocol is not configured properly.

• The Frame Relay network is not mapped correctly.

The first action to take in this case is to view both ends of the connection, focusing on the core router. Typically, the core router serial interface is up but the subinterfaces report down, as shown in Example 18-14.

Example 18-14. This Is How the Troubled Connection Serial 3/0:0.47 Looks from the Core Router Perspective

 7206-frame#show interfaces serial 3/0:0.47 Serial3/0:0.47 is down, line protocol is down   Hardware is Multichannel T1   Description: 1602-frame: 10.10.253.136/29 : 9833502 : 3814500206   Interface is unnumbered. Using address of Loopback2 (171.68.88.1)   MTU 1500 bytes, BW 56 Kbit, DLY 20000 usec,      reliability 255/255, txload 1/255, rxload 1/255   Encapsulation FRAME-RELAY 7206-frame# 

If you check the interfaces from the viewpoint of the remote user, most probably it shows that the interface is up, up, and the subinterface shows up, up.

Checking two endsthe remote side and the core sidepoints to a conclusion that the PVC is probably not mapped by the service provider or that the remote user is looped back instead of mapping the PVC to the other end.

If everything on the physical and data link layers seems to function correctly, it makes sense to move on and look for another set of possible issues. The most typical ones are routed protocol-related mismatches. In the case of IP, there are generally three groups:

• Duplicate IP addresses IP duplication can take place between subnets and within subnets. Large, fast-growing IP spaces can create this situation, and poor IP management, oversight, and merging networks. In the case of duplicates within a subnet, IOS usually provides a warning before assigning the IP address, when the interface is Ethernet or FastEthernet. In wide-area network (WAN) links, there is no warning (possibly because of bridge configurations). The result of the duplicate IP is either sporadic connectivity or total loss of IP connectivity. One possible action is to check the Domain Name System (DNS) server entries to ensure that there are no duplicate IP addresses (Cisco uses a web-based utility to provide access to the DNS tree). Another solution is to use #show cdp neighbor detail (see Chapter 4, "Troubleshooting Approaches, Models, and Tools").

• Duplicate subnets In some cases, when you try to merge two or more subnets or to assign a larger subnet to a remote user, it might result in overlapping or duplicate subnets. This can cause loss of connectivity or intermittent connectivity. Again, the #show cdp neighbor detail, #show ip interface brief, and #show ip route commands, and available DNS utilities, are the recommended tools.

• Misconfigured masks The mask defines what part of the IP address is a network address, and what part is designated for host addresses. The mask plays a significant role in all routing protocols and can be a source of different issues. It is recommended that the troubleshooting of this situation be performed in the context of the chosen routing protocol, such as Open Shortest Path First (OSPF) or Border Gateway Protocol (BGP).

For possible routing issues and overhead, see Chapter 16. DLCI and PVC issues are discussed in the following scenario.