Scenario 3: ISDN Performance Problems

Usually, a user has bandwidth expectations and uses software or a tester to measure the network speed. If the service is provisioned for 2 x 56+D, you can expect a maximum bandwidth of 112 kbps. If the service is provisioned for 2 x 64+D, you can expect a maximum of 128 kbps. The user must be educated about the overall bandwidth, which needs to be distinguished from the user's expectations, based on the size of his test files. Don't forget that, in ISDN, a significant part of the frame (1/4) is used for signaling information due to the format of an ISDN frame in the S/T coding technique (Pseudo-Ternary Line Coding) and one tenth in the U-interface coding technique (two binary, one quaternary) (see Chapter 2, "Telecommunication Basics," for more information). Also, given the fact that overhead is associated with all routing and routed protocols, you should set a user's expectations accordingly.

Performance problems don't result immediately from these situations; instead, there is an unexpected degradation of the line speed over a period of time.

The solution to this type of problem requires systematic research of all possible reasons. The problems can be caused by the following:

• Short-term routing issues

• Line problems

• Configuration setting problems

• LEC switch problems

Each issue is discussed in the following sections.

Short-Term Routing Issues

Problems due to routing are related to different aspects of the corporate network, such as routing, server hosting, server availability, and performance. Usually, problems are due to a malfunction of the intranet. A duplicate IP address can severely degrade a user's performance, especially in statically assigned IP subnets, where different subnets can affect each other if there is an overlapping address space. The easiest way to discover the problem is to check the DNS and DHCP pools to determine if access is available, or ping every IP address in the subnet to see if there is a difference in the response time and the number of successful responses.

Line Problems

After you ensure the service is set up correctly and fully operational (the router can make calls), check the status of the BRI service by using the following set of very useful commands:

 804-isdn#show interface bri0 804-isdn#show interfaces bri0 1 2 

The detailed output for these commands is explained in Table 13-5. Before you look at this output, note some of the troubleshooting steps you can take for this category of problems:

1.	Make sure that the user's router is configured correctlythat it dials out and accepts calls. NOTE Always clear the interfaces and counters before testing to limit results to only your actions.
2.	Clear the data channels of the user's router before testing by using the following command: 804-isdn#clear interface bri0:1 804-isdn#clear interface bri0:2 NOTE Just clearing interface bri0 brings down both channels.
3.	From the core router or any server, run the following: server1.cisco.com:/users/pnedeltc>PING 804-isdn 3000 If the threshold of the user's router is set to 10 percent of the bandwidth, after 1520 seconds, you will see the second channel come up. Run 200 packet pings and check the statistics. If the packet loss is greater than 0 percent, a problem exists.
4.	Check the information on the D channel, and the BRI0:1 and BRI0:2. Compare the results to the output lines in Table 13-5.
5.	Try a 56 K call to determine if the problem still exists. If it does, follow the recommendations in Table 13-5, or work with the LEC to resolve the issue.

Table 13-5. Explanation of Output from #show interface bri0 (D Channel) and #show interface bri0 1 2 (For BRI Data Channels 1 and 2)

Output	Description
BRI0 is up, line protocol is up (spoofing) BRI0:1 is down, line protocol is down	D channel information. It shows that the D channel is spoofing and indicates that the D channel is up and the dial-on-demand is ready. Unlike BRI0:1 and BRI0:2, this interface must be up at all times for DDR to function. BRI0:1 information. The line and protocol are down, which indicates the channel is inactive and does not send or receive data. This does not mean the BRI1 is not functional.
Hardware is BRI with U-interface and POTS	Hardware information that shows the router has a built-in U-interface and supports plain old telephone service (POTS).
MTU 1500 bytes BW 64 Kbit DLY 20000 usec	Maximum transmit unit (MTU) is 1500 bytes. Bandwidth is 64 Kbps. The D channel in 2B+D is 64 Kbps; however, for signaling, it uses only 16 Kbps. A delay of 20,000 microseconds = 20 milliseconds.
reliability 255/255, txload 1/255, rxload 1/255	Reliability, transmit load (txload), and receive load (rxload) are a fraction of 255.
Encapsulation PPP loopback not set	The encapsulation of the interface is Point-to-Point Protocol (PPP). The loopback is not set.
DTR is pulsed for 1 seconds on reset	The Data Terminal Ready (DTR) timer is set to 1 second.
LCP Closed, multilink Closed Closed: CDPCP, IPCP	LCP and multilink can be closed or open. Cisco Discovery Protocol Control Protocol (CDPCP) and IP Control Protocol (IPCP) are part of the NCP suit and can be closed or open as well. These lines exist only for B channels.
Last input 00:00:24, output 00:00:24,	Number of hours, minutes, and seconds since the last packets were received (input) or sent (output) by this interface.
output hang never	This indicates hours, minutes, seconds, or never since the last reset of this interface. If the time is more than 24 hours, you see 2d3h. But if this format is not sufficient, you will see *****, which indicates that more symbols are required than this format can support.
Last clearing of "show interface" counters 6d23h	Often in troubleshooting, you need to start with "fresh" data. This value indicates the last time in hours:minutes:seconds (or never) that the interface counters were cleared using the command: 804-isdn#clear interface bri0:1 or bri0:2 The 804-isdn#clear interface bri0 works as a reset of the D channel and, therefore, works as a reset of the BRI interface.
Input queue: 0/75/0/0 (size/max/ drops/flushes); Total output drops: 0 Output queue: 0/1000/64/0 (size/max total/threshold/drops)	Input queue with self-explanatory parameters and the number of dropped packets because of a lack of queue slots. If the number of dropped packets is high, you might want to check the interface buffers with the following: 804-isdn#show buffers
Queuing strategy: weighted fair	The queuing strategy.
Conversations 0/1/16 (active/max active/max total) Reserved Conversations 0/0 (allocated/max allocated)	This entry displays the number of conversations in regards to weighted fair queuing (WFQ). If FIFO is configured on the interface, the output would just show output queue and input queue statistics.
The Following Output Is Where the Line Problems Are Reported
5 minute input rate 4200 bits/sec, 1 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec	The average rate of data exchange, both in and out in five-minute intervals, measured in bits per second and packets per second.
42554 packets input, 187949 bytes, 0 no buffer	Number of packets and bytes passed with no errors. Number of discarded packets because there's no buffer.
Received 0 broadcasts	Total number of received broadcasts on this interface.
0 runts	Number of discarded packets because their size was smaller than the minimum median packet size.
0 giants	Number of packets that were discarded because their size was greater than the maximum median packet size.
0 throttles	Number of times the receiver on the port was disabled; possibly due to buffer or processor overload.
81 input errors	Total number of input errors since the last #clear interface command.
1 CRC	Cyclic Redundancy Check (CRC) errors usually indicate noise or other transmission problems.
69 frame	Incorrectly received packets, or packets with a non-integer number of octets, which usually indicates noise or other transmission problems.
0 overrun	The number of times the input rate exceeded the receiver's ability to buffer and handle the data.
0 ignored	The number of times the hardware ran low in internal buffers. Broadcast storms and bursts of noise can be the possible cause of the problem.
11 abort	Number of aborts. It shows the illegal number of 1s in the interface. It is an indication of a clocking problem on the interface. (Check the U- or S/T interface cabling.)
42601 packets output, 179159 bytes	Number of output packets and bytes in the interface passed without errors.
0 underruns	Number of times when the transmitter runs faster than the router can handle.
0 output errors	Number of output errors.
0 collisions	Number of collisions on the interface. The collisions can come from the contention failures
9 interface resets	Number of interface resets, which indicate "no signal" in the interface. The router waits ten seconds and then resets the interface trying to trigger the exchange.
0 output buffers failures, 0 output buffers swapped out	Number of buffer failures and swapped output buffers. Use 804-isdn#show buffers failures to check the type and the time of failures.
280 carrier transitions	Number of carrier transitions, which is a number that you need to divide by 2 because there are 140 ups and 140 downs in this D channel. This number in the D channel should be much smaller than the same measurement on the B channels because the B channel goes up and down to exchange data, while the D channel stays up (presumably always).

For a Cisco 77x router, the line quality of the ISDN line can be checked with the show packets command as displayed in Example 13-7.

Example 13-7. show packets Output

 776-isdn:gateway> show 2 packets Packet Statistics for Connection 2 Filtered: 0  Forwarded: 150  Received: 201 Dropped: 59  Lost: 0   Corrupted: 1  Misordered: 0 Compression Ratio: 0.99:1 IP Packet that triggered last call Source Address: 151.70.209.82 Destination Address: 151.68.10.140 Protocol: UDP Source Port: 1242 Destination Port: 53 Date/Time: 01/01/1995 00:00:47 776-isdn:gateway> 

The line quality information is shown and explained in Table 13-6.

Table 13-6. Output and Description of the show packets Output From a 77x Router

Output	Description
Packet statistics for Connection 2	The packet diagnostic statistics for Link=2.
Filtered	Packets received by the bridge engine and not forwarded.
Forwarded	Packets forwarded to Link=2.
Received	Packets received from Link=2.
The Following Are Where the Problems Are Reported
Dropped	Packets received from Link=2 and dropped because the queue of packets to be forwarded was too long.
Lost	Packets received from Link=2, but not successfully transmitted (faulty line).
Corrupted	Packets received from Link=2 with bad CRC and discarded as corrupted.
Misordered	Packets received out of sequence when using the ordered or fragmented protocol.
Compression Ratio	If using compression, it shows the compression ratio.

The line-quality issues might be only part of the problem, but they're very significant and can cause frustrating intermittent problems.

NOTE

One of the most unusual cases that I recall involved a user whose MPOE was actually a wooden cabinet in her backyard. On rainy days, her service was intermittent. But every time the LEC technician was onsite testing, everything would be fine. In the end, it turns out that, when the cabinet door was closed, it put pressure on the cable so that moisture caused intermittent connectivity. When the cabinet door was open (such as when the technician was running tests) connectivity was fine.

The interface reports should be interpreted according to the specifics of the service, which can include many and different devices in the local loop. Some configuration settings in the user's router can cause problems as well; they are discussed in the next section.

Configuration Setting Problems

The buffer shortage is included under interface commands and shows the number of successfully passed packets and bytes, as well as the dropped packets due to no buffer. This setting might cause performance problems.

Some simple recommendations that usually work for issues caused by settings include the following:

• If the number of drops increments after clearing the appropriate interfaces and running a ping test from the opposite site, determine what destination is incrementinginput or output, looking to the following lines:

   Input queue: 0/75/20 (size/max/drops); Total output drops: 30 Output queue: 0/1000/64/0 (size/max total/threshold/drops) 

  
  

  Here, the input queue has dropped 30 packets.

• If any queue drops packets, gradually increase the size of the queue by 25 percent, using either of the commands:

   804-isdn(config-if)#hold-queue length out 804-isdn(config-if)#hold-queue length in 

  
  

  From the previous example where the input queue dropped 30 packets, you should increase the input queue size.

If you see that the number of dropped packets is decreasing after repeated testing, increase the packets for further testing.

LEC Switch Problems

LEC switch problems are related to the quality of service provided by the LEC. Also, there are different departments within a LEC's organization, and the troubleshooting engineer does not always have visibility to the LEC's internal processes. The following is an example.

A user is taking an e-learning course and experiences slow performance of data and video. The user has a /30 bit (255.255.255.252) subnet assigned on one computer at homea new notebook.

Here are the troubleshooting steps for this realistic scenario, and the results for this particular case:

1.	Gather information from the user. In this case, a conversation with the user determines that the connection is four months old, and has been the same from the beginning.
2.	Do a quick check of the configuration, which looks correct for this scenario. The ISDN numbers are isdn spid1 = 56174270270100 LDN1 = 7427027 isdn spid2 = 56174270280100 LDN2 = 7427028
3.	Do a quick check of the IP subnet in the DNS and DHCP server to see if it looks correct and there is no overlapping. In this case, there is no overlapping.
4.	Have the user make two calls outone from the BRI0:2 VOICE and one from BRI0:1 DATA. Assume that the router reports that the BRI0:1 makes a voice call, but the BRI0:2 is making a data call, as shown in Example 13-8. This is just the opposite of what you expect to see. Remember this! Example 13-8. show isdn status Output 804-isdn#show isdn status Global ISDN Switchtype = basic-ni ISDN BRI0 interface dsl 0, interface ISDN Switchtype = basic-ni Layer 1 Status: ACTIVE Layer 2 Status: TEI = 123, Ces = 1, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED TEI = 124, Ces = 2, SAPI = 0, State = MULTIPLE_FRAME_ESTABLISHED Spid Status: TEI 123, ces = 1, state = 5(init) spid1 configured, spid1 sent, spid1 valid Endpoint ID Info: epsf = 0, usid = 1, tid = 0 TEI 124, ces = 2, state = 5(init) spid2 configured, spid2 sent, spid2 valid Endpoint ID Info: epsf = 0, usid = 2, tid = 0 Layer 3 Status: 1 Active Layer 3 Call(s) Activated dsl 0 CCBs = 2 CCB:callid=8021, sapi=0, ces=1, B-chan=2, calltype=DATA ! The router reports that BRI0:1 makes a VOICE call, but the BRI0:2 is ! making a data call -- the opposite to what you would expect. ! Remember the word asymmetric. CCB:callid=8024, sapi=0, ces=1, B-chan=1, calltype=VOICE The Free Channel Mask: 0x80000000 Total Allocated ISDN CCBs = 2
5.	Have the user download data and check the interfaces to see if they report any errors. In this case, they don't. The input line errors are not the problem, as you can see from the following fragment from the commands show interface bri0:1 and show interface bri0:2: 609 packets input, 3440 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
6.	Check the dialer interface for errors. Under the Dialer 1 interface, you see the following. Obviously, no input or output errors are reported and they are not the problem: 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 input packets with dribble condition detected 6464 packets output, 2178130 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collision, 14 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out
7.	Check the error statistics for virtual interface 1 with show interface dialer 1 command. The same command provides you with great details, including the five-minute input and output rates on this interface. Again, recall that the user is downloading data and, for some reason, the router is reporting an asymmetric performance, as shown in the following code fragment: 5 minute input rate 2000 bits/sec, 0 packets/sec 5 minute output rate 62000 bits/sec, 0 packets/sec The most concise commands to check the Ethernet 0, Dialer, and Virtual Access 1 interfaces for certain common values is to use the output modifiers of IOS, as follows: 804-isdn#show interface ethernet0 | include 5 minute 804-isdn#show interface dialer1 | include 5 minute 804-isdn#show interface virtual-access 1 | include 5 minute
8.	Ask the user to upload data. The result is again the same asymmetric performance: 5 minute input rate 62000 bits/sec, 0 packets/sec 5 minute output rate 2000 bits/sec, 0 packets/sec From the last two tests, you can conclude the tested direction is slow.
9.	Do a quick check of the buffers for failures. In this case, it shows that none are present: 804-isdn#shows buffers failures Obviously, the buffer failures are not causing any issue.
10.	Due to the effect the low size of the MTU can have over the performance, check the E0 interface and computer settings and determine if any MTU is set to less than 1500 for the download direction. In this case, the settings were determined by default. The MTU is not a problem.
11.	Check the core router with the following: 7206-isdn#show running-config | include 804-isdn dialer map ip 10.19.236.5 name 804-isdn speed 56 15617427027 From this command, you can determine that the core router is calling SPID1, but the call is reaching SPID2. Again, you can see another instance of asymmetric behavior in this particular ISDN service.
12.	Given the amount of asymmetric instances, the word asymmetric can characterize this ISDN service. The action plan should now be to make the service symmetric by removing all asymmetric odds from the service and seeing if this corrects the problem.
13.	The easiest way to correct the problem is to change the dialer map from the core router from (SPID1), as shown here: 7206-isdn(config-if)#dialer map ip 10.19.236.5 name 804-isdn speed 56 15617427027 The core router is dialing SPID1 (15617427027). Make the core router call SPID2 (15617427028) with the following change: 7206-isdn(config-if)#dialer map ip 10.19.236.5 name 804-isdn speed 56 15617427028
14a.	Ask the user to download the same information from the same server, as he did in an earlier step. You can check the virtual access interface of the user's router with the following command: 804-isdn#show interface virtual-Access 1 | include 5 minute ... 5 minute input rate 40000 bits/sec, 8 packets/sec 5 minute output rate 3000 bits/sec, 3 packets/sec ... Recall that the user is downloading. Your testing direction, looking from the remote user's router, is input. If you compare Step 7 and its output with the new report, you can see that the service has improved and the input rate is increasing. Here is the output from Step 7: 5 minute input rate 2000 bits/sec, 0 packets/sec Here is the new, improved output: 5 minute input rate 40000 bits/sec, 8 packets/sec If the user is not available to participate in testing, you must consider how to test without him.
14b.	Run a ping test with a packet size of 3000 and then increase the size to 4200, as shown here: server1.cisco.com:/users/pnedeltc>PING 804-isdn 3000 Measure the five-minute rate from the user's router with this: 804-isdn#show interface Virtual-Access 1 | incl 5 minu When testing with packet size 3000 bytes, the results are 5 minute input rate 24000 bits/sec, 4 packets/sec 5 minute output rate 25000 bits/sec, 4 packets/sec When testing 4500 bytes, the results are 5 minute input rate 34000 bits/sec, 4 packets/sec 5 minute output rate 35000 bits/sec, 4 packets/sec

For this case, the result of these steps is that now the connection is according with the expected behavior (symmetric), the input rate is increasing, and no packet is lost, as shown in the following fragment from ping 804-isdn 3000 statistics:

 ----804-isdn.cisco.com PING Statistics---- 439 packets transmitted, 438 packets received, 0% packet loss round-trip (ms)  min/avg/max = 654/659/947 

This example is provided to show how the repeated observation and key impressions of an engineer who doesn't necessarily have knowledge of internal processes of the LEC can still result in resolution of problems.

In this particular case, most probably, you were dealing with the way the circuit was designed by the LEC. Although these types of solutions might not have the perfect technical explanation (after all, the engineer is guessing based on his observations), they often solve the problem regardless.