Recommendations for Practical Troubleshooting of ISDN Remote Services

It is important to understand that most troubleshooting processes occur remotely. The following sections are written with this premise in mind: You are located somewhere else, possibly thousands of miles from the remote user's location. The core router is located somewhere on your campus, or possibly both the core and the end user's router are miles away from you.

NOTE

Make sure that you understand the environment in which you are troubleshooting because troubleshooting is an environment-dependant activity.

After you establish the environment with which you are dealing, you need to set up your own environment. This section focuses on how you should set up your troubleshooting process. Of course, the real world can offer you much less, but any part of these recommendations can help you pinpoint the problem faster. These recommendations include the following:

• Using #show isdn status to view service layers

• Preconfiguring the routers on both ends

• Accessing the remote user's router

Using #show isdn status to View Service Layers

If the remote user's ISDN is not functioning correctly, you cannot telnet to the router. Your best chance is to work with the remote user. Take time to help the user (if necessary) by teaching him or her how to use the telnet session or the console connection to the router.

One of the most useful IOS commands for ISDN troubleshooting is show isdn status, which provides a snapshot of all layers of the service. Example 13-1 shows an ISDN line in the activation stage. The first layer of this IOS-based router is active, but the second layer negotiation is still in progress.

Example 13-1. Snapshot of ISDN BRI Layers

 804-isdn#show isdn status Global ISDN Switchtype = basic-5ess ISDN BRI0 interface         dsl 0, interface ISDN Switchtype = basic-ni     Layer 1 Status:         ACTIVE     Layer 2 Status:         TEI = 96, Ces = 1, SAPI = 0, State = TEI_ASSIGNED         TEI = 105, Ces = 2, SAPI = 0, State = TEI_ASSIGNED         TEI 96, ces = 1, state = 3(await establishment)             spid1 configured, spid1 sent, spid1 valid             Endpoint ID Info: epsf = 0, usid = 0, tid = 1         TEI 105, ces = 2, state = 1(terminal down)             spid2 configured, spid2 sent, spid2 valid             Endpoint ID Info: epsf = 0, usid = 1, tid = 1     Layer 3 Status:         0 Active Layer 3 Call(s)     Active dsl 0 CCBs = 0     The Free Channel Mask:  0x80000003     Total Allocated ISDN CCBs = 0 804-isdn# 

NOTE

The dynamics of ISDN events can be seen using the powerful command 804-isdn#debug isdn events [interface bri 0]. This command requires a detailed understanding of the messages on the screen and generates significant output. If you feel confident enough, use it; but when you use it on the core router, limit the duration to less than ten minutes and turn it off.

The #show isdn status output in Example 13-2 is of a properly functioning BRI circuit. In Example 13-2, Layer 1 is active and Layer 2 is in a MULTIPLE_FRAME_ESTABLISHED state. The terminal endpoint identifiers (TEIs) are successfully negotiated, and the ISDN Layer 3 is ready to make or receive calls.

Example 13-2. All Layers of ISDN Are Active

 804-isdn#show isdn status Global ISDN Switchtype = basic-5ess ISDN BRI0 interface         dsl 0, interface ISDN Switchtype = basic-ni     Layer 1 Status:         ACTIVE     Layer 2 Status:         TEI = 96, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED         TEI = 105, Ces = 2, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED         TEI 96, ces = 1, state = 5(init)             spid1 configured, spid1 sent, spid1 valid             Endpoint ID Info: epsf = 0, usid = 0, tid = 1         TEI 105, ces = 2, state = 5(init)             spid2 configured, spid2 sent, spid2 valid             Endpoint ID Info: epsf = 0, usid = 1, tid = 1     Layer 3 Status:         0 Active Layer 3 Call(s)     Active dsl 0 CCBs = 0     The Free Channel Mask:  0x80000003     Total Allocated ISDN CCBs = 0 804-isdn# 

This is an important command. You must understand each line of this output. Table 13-1 provides a detailed description of the output.

Table 13-1. Output from the #show isdn status Command

Line of the Output	Description
Global ISDN Switchtype
The current ISDN Switchtype = basic-5ess	Basic-5ess is the switch-type, configured in the general configuration of the router.
ISDN BRI0 interface dsl 0, interface ISDN Switchtype = basic-ni	Basic-NI1 (National) is configured under interface BRI0. This configuration takes precedence over the previous line. If there are more BRI interfaces without a specific switch-type configuration, they will accept the Basic-5ess settings. See Chapter 10, "ISDN Design Solutions."
Layer 1 status
ACTIVE	Layer 1 Status: Verifies the physical layer connectivity with the LEC's ISDN switch. This can be ACTIVE or DEACTIVATED. See Chapters 11 and 12 for more information.
Layer 2 Status
TEI = 96, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED TEI = 105, Ces = 2, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED	Status of ISDN Layer 2 with the TEI number and multiframe state. If the switch uses dynamic assignments, the valid TEI numbers are from 64 to 124. If they are static assignments, the numbers are from 1 to 63. TEI=127 stands for broadcast-request. The most common Layer 2 states are MULTIPLE_FRAME_ESTABLISHED and TEI_ASSIGNED. A state=MULTIPLE_FRAME_ESTABLISHED indicates that there is data-link connectivity to the LEC's ISDN switch. You should see this state under normal operation. Any other state indicates a problem on the circuit. A Layer 2 Status of down is indicated by Layer 2 NOT Activated. See Chapter 12 for more information.
Spid Status
TEI 96, ces = 1, state = 8(established) TEI 106, ces=2, state = 5(init)	TEI number and state. The most common state values are State = 1(terminal down) State = 3(await establishment) State = 5(init) State = 6(not initialized) State = 8(established) Only states 8 and 5 indicate a working BRI circuit. The other states mean the circuit is not properly established.
spid(1 or 2) configured, spid(1 or 2) sent, spid(1 or 2) valid , no LDN	spid(1 or 2) configuration information for a working BRI. The valid combination is shown on the left column and other possible combinations are spid(1 or 2) configured, no LDN, spid1 sent, spid1 valid spid(1 or 2) NOT configured, spid1 NOT sent, spid1 NOT valid spid(1 or 2) configured, spid1 NOT sent, spid1 NOT valid spid(1 or 2) configured, spid1 sent, spid1 NOT valid The last three states indicate that either the spid was not configured or that it is incorrect. No LDN means that the associated LDN(1 or 2) is not configured.
Endpoint ID Info: epsf = 0, usid = 1, tid = 1 Endpoint ID Info: epsf = 0, usid = 3, tid = 1	Endpoint identifier information can be used by the router to decide which channel will answer the call. The message ENDPOINT ID in the isdn q931 format is associated with the usid and tid.
Usid	User Service Identifier.
tid	Terminal Identifier.
Layer 3 Status
2 Active Layer 3 Call(s)	Number of active calls.
CCB:callid=103, sapi=0, ces=1, B-chan=1, calltype=DATA CCB:callid=819B, sapi=0, ces=2, B-chan=2, calltype=DATA	Information about the active calls. The first line is incoming calls to the user's router, and the second line is outgoing calls. For a connected call, it displays the caller ID information, call reference, and the B channel it is occupying. The first call's callid is a smaller number than the second one. The call type can be DATA or VOICE. Any other type, such as INTERNAL, identifies a problem with the LEC.
Activated dsl 0 CCBs = 2	Number of Digital Signal Link (dsl) activated. Number of Call Control Blocks (CCB) in use.
Total Allocated ISDN CCBs = 2	Number of ISDN CCBs that are allocated.

In the enterprise environmentwhere usually you have deployed limited types of Cisco ISDN routersyou might have more options available, such as ISDN lines and Cisco routers configured for test purposes. The following sections give you details on these.

Preconfiguring the Routers on Both Ends

The first thing you must consider is the router that's immediately available to you. It's a good idea to use any Cisco ISDN router, such as Cisco 776, Cisco 804, Cisco 1604, and so on, to perform initial testing of the remote user's router. If you are a network administrator who is responsible for a relatively big ISDN environment, it's a good idea to have an ISDN line at your desk.

For example, the Cisco 77x router at your desk with a simple configuration can help you perform initial troubleshooting. The following are recommended configuration lines:

 set system ENS set ppp secret client secret secret set ppp secret host secret secret set ppp auth out none 

NOTE

The system name can be anything as long as the system name on both ends (the test side and the remote user side) is the same.

If the remote user has a 77x router, the following configuration needs to be applied to the user's router to allow remote access for troubleshooting:

 set user ENS set ppp client ENS set bridging on set ppp authentication outgoing chap set ppp secret client secret secret set ppp secret host secret secret cd set active ENS 

NOTE

One possible option is to turn off the authentication on both ends of the link to rule it out as a problem.

The benefit of this configuration is that, if the remote user's router includes this user configuration, you can call the LDN using the 77x router at your desk. The user's router simply prompts you for a password.

NOTE

Remember that the Cisco 77x series router has only one level of password authentication. This is unlike IOS routers, which have more sophisticated password systems, such as vty authentication option, enable password, and secrets.

To verify that the user profile exists and it is active on both sides, enter the show user command on both routers, as shown in Example 13-3.

Example 13-3. show user Output

 ENS> show user User                       State      Connection -------------------------------------------- LAN                        Active     LAN Internal                   Active     INTERNAL Standard                   Active     1 gateway                    Active     2 ENS                        Active     3 ENS> 

NOTE

The remote user's router must be powered up for testing purposes. Even when troubleshooting is performed by the LEC, the technician cannot run any tests without the powered-up NT1. If the router has a built-in NT1, the router must be powered on; if not, the power supply for NT1 must be turned on. Most of the line tests by the local providers require looping back the NT1.

Accessing the Remote User's Router

With your router and the user's router setup, you can now make a manual call to the remote user's router from your test router, which will result in a number of typical situations:

• The circuit is not available, and the message from your test router is usually "Cause 34 No Circuit/Channel Available." Typically, after receiving this message, there's nothing to troubleshoot. Either the circuit is disconnected, the service is completely shut down, or you called a non-ISDN number:

   ENS> call 1 14087785444 02/04/1995 20:54:45  L05  0  14087785444  Outgoing Call Initiated ENS> ENS> 02/04/1995 20:54:46  L12  1          Disconnected Remotely Cause 34  No Circuit/Channel Available ENS> 02/04/1995 20:54:46  L27  1               Disconnected ENS> 

  
  

  Could this condition be caused by a misconfiguration? It would be prudent to explore this possibility first, and double-check it with the remote user, the switch type, SPIDs, and LDNs.

  If everything is okay, it's a good idea to open a trouble ticket with the local provider at this point and double-check how the service is provisioned.

• If you are calling a non-ISDN number, the typical response will be bearer capability error, as shown in the following:

   ENS> call 1 14083691136 02/04/1995 21:00:21  L05  0  14083691136  Outgoing Call Initiated ENS> ENS> 02/04/1995 21:00:22  L12  1               Disconnected Remotely Cause 65  Bearer Capability Error ENS> 02/04/1995 21:00:22  L27  1               Disconnected ENS> 

  
  

• The call is rejected. When the channel is already up, you cannot make a call to the remote user's router (unless call waiting is configured on the LEC side). You will receive a Receiver Busy or Call Rejected message. The Call Rejected message is shown here:

   ENS> call 1 14085764740 02/04/1995 21:04:34  L05  0  14085764740  Outgoing Call Initiated ENS> ENS> 02/04/1995 21:04:35  L11  1   4085764740  Call Requested ENS> 02/04/1995 21:04:35  L09  1               Call Rejected Cause 17  Receiver Busy ENS> 02/04/1995 21:04:35  L12  1               Disconnected Remotely Cause 17  Receiver Busy ENS> 02/04/1995 21:04:35  L27  1               Disconnected 

  
  

  For other call rejection messages, see Table 12-1 in Chapter 12.

• The channel that you are calling is idle, so you can make a call. An idle channel and an initiated call is shown here:

   ENS> call 1 14086175555 01/05/1995 21:34:49  L05  0  14086174271  Outgoing Call Initiated ENS> ENS> 01/05/1995 21:34:50  L08  1  14086174271  Call Connected ENS> 01/05/1995 21:34:51  Connection 2 Add     Link 1 Channel 1 ! At this point you have a connection ENS> login remote ENS> Enter Password: 

  
  

After this point, you are in the remote user's router. If the provided password is correct, you see the remote user's router prompt:

 776-isdn> 

Now you have direct access to the user's router without needing the end user to assist you with troubleshooting.

Preliminary Test of Remote IOS-Based ISDN Router with Test 776 Router

If the user's router is an IOS-based ISDN router and you are calling from a 77x router, you can still make a call, but the IOS will not prompt you (by default). To gain access to an IOS router, you usually use a Telnet or Secure Shell (SSH) session. The problem is: How do you use Telnet if the remote user's router cannot pass data? This is the benefit of using a 77x to call an 804 router. If the call is successful, you see the following message:

 ENS> call 1 14087312598 01/01/1995 00:07:36  L05  0  14087312598  Outgoing Call Initiated ENS> ENS> 01/01/1995 00:07:37  L08  1  14087312598  Call Connected ENS> 01/01/1995 00:07:37  L60  1               Password Not Found ENS> 01/01/1995 00:07:37  L12  1               Disconnected Remotely Cause 16  Normal Disconnect ENS> 01/01/1995 00:07:37  L27  1               Disconnected ENS> 

Your 776 router makes a call to 1 408 731-2598, which is configured in Cisco IOS 804 router. If remote users see the channel1 light, and both the TxD and RxD lights blink, the status of the ISDN service is okay (see the show isdn status command later). But the connection drops with the message Password Not Found, which gives you an indication of what the problem is. First of all, this is the expected message and the connection is dropping at the authentication phase of PPP protocol because you are calling from the test router, but not from the core router. This narrows down the probable causes of the problem. Recall from the explanations in Chapter 12 that to reach the PPP authentication phase, the ISDN service goes through all three layers of the D channel, through the LCP stage of PPP, and, just after that, it reaches the authentication phase. At this point, all you have to do is to concentrate on possible PAP/CHAP issues on both the remote user's and core routers.

Preliminary Test of Remote 77x Router with Test IOS-Based ISDN Router

How do you call from an IOS-based router to a 77x router? Example 13-4 shows you how this works. Your test router ENS-isdn is calling 1 831 430-0444 by using your first channel. You are debugging the ppp negotiation with the command #debug ppp negotiation.

Example 13-4. The Output of Calling a 77x Router from an IOS-Based Router

 ENS-isdn#isdn call interface bri0:1 18314300444 *Mar  7 02:28:31.895: BR0:1 PPP: Phase is TERMINATING *Mar  7 02:28:31.899: BR0:1 LCP: O TERMREQ [Open] id 107 len 4 *Mar  7 02:28:31.899: Vi1 CDPCP: State is Closed *Mar  7 02:28:31.903: Vi1 IPCP: State is Closed *Mar  7 02:28:31.907: %LINK-3-UPDOWN: Interface Virtual-Access1,      changed state  to down *Mar  7 02:28:31.907: Vi1 PPP: Phase is TERMINATING *Mar  7 02:28:31.911: Vi1 LCP: State is Closed *Mar  7 02:28:31.911: Vi1 PPP: Phase is DOWN *Mar  7 02:28:31.911: Di1 IPCP: Remove route to 172.30.253.254 *Mar  7 02:28:32.115: BR0:1 LCP: I TERMACK [TERMsent] id 107 len 4 *Mar  7 02:28:32.115: BR0:1 LCP: State is Closed *Mar  7 02:28:32.119: BR0:1 PPP: Phase is DOWN *Mar  7 02:28:32.119: BR0:1 PPP: Phase is ESTABLISHING, Passive Open *Mar  7 02:28:32.119: BR0:1 LCP: State is Listen *Mar  7 02:28:32.127: %ISDN-6-DISCONNECT: Interface BRI0:1  disconnected from   18007735555 access-gw1, call lasted 61 seconds *Mar  7 02:28:32.319: %LINK-3-UPDOWN: Interface BRI0:1, changed state to down *Mar  7 02:28:32.331: BR0:1 LCP: State is Closed *Mar  7 02:28:32.331: BR0:1 PPP: Phase is DOWN8314300444 *Mar  7 02:28:32.895: %LINEPROTO-5-UPDOWN: Line protocol on Interface BRI0:1,      changed state to down *Mar  7 02:28:32.907: %LINEPROTO-5-UPDOWN: Line protocol on Interface   Virtual-Access1, changed state to down *Mar  7 02:28:35.883: %LINK-3-UPDOWN: Interface BRI0:1, changed state to up *Mar  7 02:28:35.899: BR0:1 PPP: Treating connection as a callout *Mar  7 02:28:35.899: BR0:1 PPP: Phase is ESTABLISHING, Active Open *Mar  7 02:28:35.903: BR0:1 LCP: O CONFREQ [Closed] id 108 len 35 *Mar  7 02:28:35.903: BR0:1 LCP:    AuthProto CHAP (0x0305C22305) *Mar  7 02:28:35.907: BR0:1 LCP:    MagicNumber 0xD030C260 (0x0506D030C260) *Mar  7 02:28:35.907: BR0:1 LCP:    MRRU 1524 (0x110405F4) *Mar  7 02:28:35.907: BR0:1 LCP:    EndpointDisc 1 Local     (0x131001706E6564656C74632D6973646E) *Mar  7 02:28:37.539: BR0:1 LCP: I CONFREQ [REQsent] id 89 len 31 *Mar  7 02:28:37.543: BR0:1 LCP:    AuthProto CHAP (0x0305C22305) *Mar  7 02:28:37.543: BR0:1 LCP:    MagicNumber 0xB13FEF1B (0x0506B13FEF1B) *Mar  7 02:28:37.543: BR0:1 LCP:    MRRU 1524 (0x110405F4) *Mar  7 02:28:37.547: BR0:1 LCP:    EndpointDisc 1 Local     (0x130C01627564692D6973646E) *Mar  7 02:28:37.551: BR0:1 LCP: O CONFACK [REQsent] id 89 len 31 *Mar  7 02:28:37.551: BR0:1 LCP:    AuthProto CHAP (0x0305C22305) *Mar  7 02:28:37.551: BR0:1 LCP:    MagicNumber 0xB13FEF1B (0x0506B13FEF1B) *Mar  7 02:28:37.555: BR0:1 LCP:    MRRU 1524 (0x110405F4) *Mar  7 02:28:37.555: BR0:1 LCP:    EndpointDisc 1 Local     (0x130C01627564692D6973646E) *Mar  7 02:28:37.903: BR0:1 LCP: TIMEout: State ACKsent *Mar  7 02:28:37.903: BR0:1 LCP: O CONFREQ [ACKsent] id 109 len 35 *Mar  7 02:28:37.903: BR0:1 LCP:    AuthProto CHAP (0x0305C22305) *Mar  7 02:28:37.907: BR0:1 LCP:    MagicNumber 0xD030C260 (0x0506D030C260) *Mar  7 02:28:37.907: BR0:1 LCP:    MRRU 1524 (0x110405F4) *Mar  7 02:28:37.907: BR0:1 LCP:    EndpointDisc 1 Local     (0x131001706E6564656C74632D6973646E) *Mar  7 02:28:37.947: BR0:1 LCP: I CONFACK [ACKsent] id 109 len 35 *Mar  7 02:28:37.947: BR0:1 LCP:    AuthProto CHAP (0x0305C22305) *Mar  7 02:28:37.947: BR0:1 LCP:    MagicNumber 0xD030C260 (0x0506D030C260) *Mar  7 02:28:37.951: BR0:1 LCP:    MRRU 1524 (0x110405F4) *Mar  7 02:28:37.951: BR0:1 LCP:    EndpointDisc 1 Local     (0x131001706E6564656C74632D6973646E) *Mar  7 02:28:37.955: BR0:1 LCP: State is Open *Mar  7 02:28:37.955: BR0:1 PPP: Phase is AUTHENTICATING, by both *Mar  7 02:28:37.955: BR0:1 CHAP: O CHALLENGE id 176 len 34 from "ENS-isdn" *Mar  7 02:28:37.959: BR0:1 CHAP: I CHALLENGE id 87 len 30 from "804-isdn" *Mar  7 02:28:37.963: BR0:1 CHAP: Username 804-isdn not found ! Username is not found *Mar  7 02:28:37.963: BR0:1 CHAP: Unable to authenticate for peer ! Unable to authenticate *Mar  7 02:28:37.963: BR0:1 PPP: Phase is TERMINATING *Mar  7 02:28:37.967: BR0:1 LCP: O TERMREQ [Open] id 110 len 4 *Mar  7 02:28:37.995: BR0:1 LCP: I TERMACK [TERMsent] id 110 len 4 *Mar  7 02:28:37.995: BR0:1 LCP: State is Closed *Mar  7 02:28:37.995: BR0:1 PPP: Phase is DOWN *Mar  7 02:28:37.995: BR0:1 PPP: Phase is ESTABLISHING, Passive Open *Mar  7 02:28:37.999: BR0:1 LCP: State is Listen *Mar  7 02:28:38.003: %ISDN-6-DISCONNECT: Interface BRI0:1  disconnected     from 18314300475 804-isdn, call lasted 2 seconds *Mar  7 02:28:38.159: %LINK-3-UPDOWN: Interface BRI0:1, changed state to down *Mar  7 02:28:38.167: BR0:1 LCP: State is Closed *Mar  7 02:28:38.167: BR0:1 PPP: Phase is DOWN ENS-isdn# 

The procedure is successful until the authentication phase (see Chapter 12 for more information on the authentication phase), at which point the call is dropped in two seconds. This shows that the ISDN layers are okay, PPP is configured, LCP is okay, CHAP is the chosen authentication option, and CHAP is configured two ways. From this point, you can focus on authentication, NCP protocols, and routing.

NOTE

It is important to understand that, in this example, username failed because, by default, IOS is trying to use the host name for the CHAP username.

With theses initial recommendations, you are now ready to tackle some of the problems you might face. The following scenarios walk you through typical situations that you might encounter when troubleshooting ISDN remote services.