MPLS Troubleshooting

The forthcoming sections cover various MPLS troubleshooting scenarios. In particular, LSR bringup problem cases and LVC starvation scenarios will be categorized and analyzed.

Troubleshooting LSR Bringup Problems

Essentially, two parameters need to match between the VSI master and the VSI slaves: Controller ID and VSI base-vc.

The debug command debug vsi errors in the LSC shown in Example 6-84 helps you troubleshoot LSC switch communication problems.

Example 6-84. Debugging VSI Errors

 LSC#debug vsi errors 02:18:37: VSI Master: parse error (unexpected controller id) in IFC GET CNFG TRAP rcvd on ATM1/0:0/42 (slave 2) 02:19:29: VSI Master: got NAK reason 105 (unknown ctrler id) in GEN ERROR RSP 

This example shows two different manifestations of the same problem.

The first VSI error is a VSI master parser error. The VSI master receives a VSI message from the VSI slave that has an unexpected controller ID. That is, the VSI slave sends a VSI message to the master with a controller ID different from the one the master has configured. This error is generated in the master (LSC) when any VSI message (such as IFC GET CNFG) is received.

The second VSI error is generated at the VSI slave. The VSI master receives a special VSI message generated at the slave: GEN ERROR RSP. This message indicates that the VSI slave (BXM/UXM) had an error, which is shown in the NAK reason (error code). NAK reason 105 indicates an unknown controller ID.

Running Out of LVCs

Running out of LVCs is one of the most common problems in ATM MPLS networks. The immediate symptom is an error message on the LSC that says TCATM-XCONNECT_FAILED. This means that there are insufficient LCNs for LVCs on one of the LSR interface's partitions:

 00:30:32: %TCATM-4-XCONNECT_FAILED: 10.10.10.3/32 Head end terminating on XTagATM3102 00:30:32: %TCATM-4-XCONNECT_FAILED: 10.10.10.5/32 Tail end terminating on XTagATM3102 

The following is a quote from Josh Gahm, an early LSC developer, regarding LVC starvation:

A network that has run out of cross-connects is basically non-functioning. There really isn't any way to control which [LVCs] get created and which don't. This may result in permanent black holes and/or wildly overloaded control processors. The network needs to be designed from the start such that it won't run out of connections.

In the case of LVC starvation, ordinary IP traffic (meaning traffic that uses an MPLS label stack of one entry) is sent on the untagged VC (the control-vc, 0/32 by default). This traffic is processed by the LSCs and is not switched in the ATM switching fabrics, causing high CPU utilization. This traffic has poor performance and can affect LDP and IP routing traffic (because it shares the same VC). For services that use MPLS, such as MPLS VPNs, traffic is discarded (causing black holes) because there is no LVC to the destination eLSR.

We cannot stress enough the importance of a well-designed and dimensioned network. It's important to calculate the worst-case scenario for the number of LVCs in each link and configure the guaranteed LCN field (minLCN) to at least that value.

Identifying the Problem

To see whether the problem is LCN resource starvation in the VSI partition, we can use the command dspvsipartinfo on IGX-8400 or BPX-8600 platforms. See Example 6-85. Zero available LCNs indicates that the partition is starved for channel resources.

Example 6-85. Checking Resource Usage from the Switch

 i8430-2a       TN    Cisco           IGX 8430  9.3.11    Apr. 15 2001 05:09 GMT             VSI Resources Status for trunk  31.2 Partition 3 Minimum Lcns       :     100     Minimum BW   (cps) :  100000 Maximum Lcns       :     300     Maximum BW   (cps) :  100000 Used Lcns          :     300     Used BW      (cps) :       0 Available Lcns     :       0     Available BW (cps) :  100000 Start VPI          :       2     End VPI            :       2 This Command: dspvsipartinfo 31.2 3 Hit DEL key to quit: 

In an AXSM card on an MGX-8850 platform, we can use the command dspload. Refer to Example 6-86. In the dsload command output, the available channel lines are of prime importance.

Example 6-86. Checking Resource Usage from the Switch

 P3-m8850.1.AXSM.a > dspload 1 2         +-------------------------------------------+         |  I N T E R F A C E    L O A D   I N F O   |         +-------------------------------------------+         | Maximum Channels         : 0001024        |         | Guaranteed Channels      : 0000512        |         | Igr Maximum Bandwidth    : 0096000        |         | Igr Guaranteed Bandwidth : 0096000        |         | Egr Maximum Bandwidth    : 0096000        |         | Egr Guaranteed Bandwidth : 0096000        |         | Available Igr Channels   : 0000000        |         | Available Egr Channels   : 0000000        |         | Available Igr Bandwidth  : 0096000        |         | Available Egr Bandwidth  : 0096000        |         +-------------------------------------------+         |          E X C E P T -- V A L U E S       |         +-------------------------------------------+         | SERV-CATEG | VAR-TYPE | INGRESS  | EGRESS   |         +-------------------------------------------+         | TAG0       | Avl Bw   | 0027840  | 0027840   |     ! 96000 * 29 %         | TAG1       | Avl Bw   | 0033600  | 0033600   |     ! 96000 * 35 %         | TAG2       | Avl Bw   | 0033600  | 0033600   |     ! 96000 * 35 %         | TAG3       | Avl Bw   | 0000960  | 0000960   |     ! 96000 * 1 %         +-------------------------------------------+ P3-m8850.1.AXSM.a > 

It is interesting to note that the command dspload in AXSM cards shows except values for bandwidth resources and not for LCN resources. This is because the LSC sends only bandwidth CoS values in the interface policy parameter VSI message. In contrast, the PNNI controller sends both bandwidth and LCN CoS values (as discussed in Chapter 11, "Advanced PNNI Configuration").

From the LSC, we can use the command show controllers vsi descriptor. Refer to Example 6-87. The same information can be gathered using VSI Master MIB.

Example 6-87. Checking Resource Usage from the LSC

 LSC#show controller vsi descriptor Phys desc: 0.31.2.0 Log intf:  0x001F0200 Interface: XTagATM3102 IF status: up                    IFC state: ACTIVE Min VPI:   2                     Maximum cell rate:  100000 Max VPI:   2                     Available channels: 0 Min VCI:   32                    Available cell rate (forward):  100000 Max VCI:   65535                 Available cell rate (backward): 100000 

To totally identify the problem, we can use the commands show mpls atm-ldp summary, discussed earlier, to check for bindings in the Bindwait state, and show mpls atm-ldp bindwait to identify the destination.

LCN Usage

The best way to solve problems is not to have them in the first place. To help design our network, we recommend reviewing the VSI Partitioning section in Chapter 2 titled "Hard, Soft, and Dynamic Resource Partitioning" and using two simple commands.

First, as shown in Example 6-88, we can use dspcd to identify the port groups (PGs) in each card. Even though the output of this command is slightly different on BPX-8600, IGX-8400, and MGX-8850 platforms, the port group information is present in all of them.

Example 6-88. Identifying Port Groups

 P1-b8620       TN    Cisco           BPX 8620  9.3.30    Dec. 9 2001  21:22 GMT Detailed Card Display for BXM-155 in slot 3 Status:          Active Revision:        FBN              Backcard Installed Serial Number:   688728             Type:          LM-BXM Top Asm Number:  28215802           Revision:      BA Queue Size:      228300             Serial Number: 759169 Supp:4 Pts, OC3, FST, VcShp         Top Asm Number: Supp: VT,ChStLv 1,VSI(Lv 3,ITSM)    Supp: 4 Pts,OC3,MMF,RedSlot:NO Supp: APS(FW), F4F5 Support: LMIv 1,ILMIv 1,NbrDisc,XL Supp: OAMLp,TrfcGen,OAM-E #Ch:16320,PG[1]:8160,PG[2]:8160    ! Port Group 1: Intfs 1 and 2. PG[1]:1,2,PG[2]:3,4,               ! Port Group 2: Intfs 3 and 4. #Sched_Ch:16384 #Total_Ch:16320 Last Command: dspcd 3 Next Command: 

In this case, we have a BXM card with two port groups, each of which has a maximum of 8160 LCNs to be shared. PG 1 spans ports 1 and 2, and PG 2 includes ports 3 and 4. For every first-level label allocated to a next-hop destination, an LCN is used. As you will see, enabling multi-vc multiplies the number of LVCs by the number of classes used. LSC redundancy also multiplies the number of LVCs used.

Second, as shown in Example 6-89, we can use the command dspchuse on the BPX-8600 and IGX-8400 platforms to check the current LCN usage.

Example 6-89. Identifying Port Groups and Checking LCN Reservations

 P1-b8620       TN    Cisco           BPX 8620  9.3.30    Dec. 9 2001  21:22 GMT                    Channel Management Summary for Slot 3                 max    used   avail  netw   pvc cnfg  f4-f5  vsi mgmt  vsi cnf card 3:      16320   2582  13738    814    256         0      0      1512 port grp 1:   8160   1798   6362    542    256         0      0      1000 port grp 2:   8160    784   7376    272      0         0      0       512            pvc cnfg  pvc used  nw used   f4-f5   vsi mgmt  vsi min     vsi max phy if 1:     0         0       271         0       0       ---       --- ! -+  phy trk:     0         0                  ---                            !  |   part 1:                                                   500      1000 !  | PG1 phy if 2:   256         0       271         0       0       ---       --- !  |  phy trk:   256         0                  ---                            ! -+ phy if 3:     0         0       272         0       0       ---       --- ! -+  vtrk 1:      0         0                  ---                            !  |   part 1:                                                   256       512 !  |  vtrk 2:      0         0                  ---                            !  | PG2   part 1:                                                   256       512 !  | phy if 4:     0         0         0         0       0       ---       --- ! -+ Last Command: dspchuse 3 Next Command: 

From this output, we can calculate the sum of the vsi min and find the largest vsi max.

Solving the Problem

The solution to the running out of LVCs problem is straightforward: Increase the number of LCN resources for the VSI partition. We do this with cnfrsrc on the IGX-8400 and BPX-8600 platforms and with cnfpart for AXSM cards on MGX-8850 platforms. The drawback is that this procedure affects service.

After that, we will monitor that there are no more bindings in the Bindwait state and that there are available LCN resources in the VSI slave.

Reducing the Number of LVCs

You can reduce the number of LVCs in an ATM MPLS network in several ways. For large networks with several eLSRs and destinations, the only option might be to enable VC-merge, but an expert network design plays a more important role.

By default, ATM LSRs also perform eLSR functions. They request LVCs for the IP destinations they learn (building headend LVCs), and they accept LVC requests from other eLSRs (creating tailend LVCs). Those headend and tailend LVCs terminating in the LSC normally are not needed. From the MPLS architectural model, the objective is to set up shortcut LVCs among all eLSRs, but not to the LSRs.

Disabling Headend LVCs

We can instruct the LSCs not to request bindings for the destinations in the routing table, effectively disabling headend LVCs from the LSC.

This does not mean that those destinations are unreachable from the LSC, but in the absence of LVC to a destination, the untagged VC (control-vc) forwards that traffic. It is important to note that in normal operation, LVCs originating on and terminating in the LSC are not used for customer traffic.

To achieve this, we can use the global configuration command mpls atm disable-headend-vc in all the LSCs.

If we look at a destination (in this case, a remote eLSR), we see that the LSC is setting up a headend LVC. See Example 6-90.

Example 6-90. Identifying Headend VCs

 P1_LSC_c7204#show mpls atm-ldp bindings 172.27.1.133 32  Destination: 172.27.1.133/32     Headend Switch XTagATM1341 (2 hops) 27/37  Active, VCD=75     Transit XTagATM332 10/76 Active -> XTagATM1341 27/43 Active     Transit XTagATM331 9/76 Active -> XTagATM1341 27/49 Active     Transit XTagATM133 2/128 Active -> XTagATM1341 27/55 Active     Transit XTagATM133 2/134 Active -> XTagATM1341 27/61 Active     Transit XTagATM133 2/140 Active -> XTagATM1341 27/67 Active P1_LSC_c7204#conf t 

We then disable headend eLSR functionality in the LSC. Refer to Example 6-91.

Example 6-91. Disabling Headend Edge Functionality in the LSC

 P1_LSC_c7204#conf t Enter configuration commands, one per line.  End with CNTL/Z. P1_LSC_c7204(config)#mpls atm disable-headend-vc P1_LSC_c7204(config)#^Z 

We see in Example 6-92 that the headend LVC is removed.

Example 6-92. Displaying Bindings with Headend Functionality Disabled

 P1_LSC_c7204#show mpls atm-ldp bindings 172.27.1.133 32  Destination: 172.27.1.133/32     Transit XTagATM332 10/76 Active -> XTagATM1341 27/43 Active     Transit XTagATM331 9/76 Active -> XTagATM1341 27/49 Active     Transit XTagATM133 2/128 Active -> XTagATM1341 27/55 Active     Transit XTagATM133 2/134 Active -> XTagATM1341 27/61 Active     Transit XTagATM133 2/140 Active -> XTagATM1341 27/67 Active P1_LSC_c7204# 

We then perform this procedure on all the LSCs in the network.

Disabling Tailend LVCs and LVCs to Numbered Links

At this point, we have disabled half of the eLSR functionality in the LSCs. The other portion consists of the tailend VCs.

We can see in Example 6-93 that these LVCs exist by default, checking for bindings for the local loopback address. There are eight tailend LVCs.

Example 6-93. Checking for Tailend Bindings

 P1_LSC_c7204#show mpls atm-ldp bindings 172.27.1.1 32  Destination: 172.27.1.1/32     Tailend Switch XTagATM331 9/34 Active -> Terminating Active, VCD=19     Tailend Switch XTagATM332 10/34 Active -> Terminating Active, VCD=15     Tailend Switch XTagATM133 2/34 Active -> Terminating Active, VCD=23     Tailend Switch XTagATM133 2/46 Active -> Terminating Active, VCD=24     Tailend Switch XTagATM133 2/58 Active -> Terminating Active, VCD=25     Tailend Switch XTagATM1341 27/34 Active -> Terminating Active, VCD=73     Tailend Switch XTagATM1341 27/58 Active -> Terminating Active, VCD=77     Tailend Switch XTagATM1341 27/64 Active -> Terminating Active, VCD=78 P1_LSC_c7204# 

We can find the headend LVC in an eLSR corresponding to one of the tailend LVCs. Refer to Example 6-94.

Example 6-94. Identifying a Headend LVC

 PE_m8250_RPMB_10#show mpls atm-ldp bindings 172.27.1.1 32  Destination: 172.27.1.1/32     Headend Router Switch1.1 (2 hops) 10/34  Active, VCD=155 PE_m8250_RPMB_10# PE_m8250_RPMB_10#show mpls atm summary Total number of destinations: 9 ATM label bindings summary       interface   total  active   local  remote   Bwait   Rwait  IFwait       Switch1.1      13      13       6       7       0       0       0       Switch1.2       2       2       1       1       0       0       0 PE_m8250_RPMB_10# 

Now we can go ahead and filter the LSC loopback addresses in this eLSR using the global configuration command mpls ldp request-labels for {ACL}>.

NOTE

The command mpls ldp advertise-labels has no effect on LC-ATM interfaces. This is because the label distribution method is Downstream on Demand (upstream-controlled).

We define an access list that denies the LSC loopback address and permits everything else (see Example 6-95). We call it BlockThis.

Example 6-95. Creating an ACL Blocking LSC Loopback Addresses

 PE_m8250_RPMB_10#conf t Enter configuration commands, one per line.  End with CNTL/Z. PE_m8250_RPMB_10(config)#ip access-list standard BlockThis PE_m8250_RPM(config-std-nacl)#deny 172.27.1.0 0.0.0.127 PE_m8250_RPM(config-std-nacl)#permit any PE_m8250_RPM(config-std-nacl)#exit PE_m8250_RPMB_10(config)# 

NOTE

When planning our network, we assigned the LSC and eLSR loopback addresses such that there was a block assigned only to LSCs and a different block of addresses for the eLSRs. That allowed us to simplify this access list.

Finally, we apply the access list to the addresses we do not want to set up LVCs to:

 PE_m8250_RPMB_10(config)#mpls ldp request-labels for BlockThis 

We can check that this eLSR now has three fewer remote bindings (see Example 6-96). These are the three LSC loopback addresses for which we are not requesting LVCs.

Example 6-96. Checking the MPLS ATM Binding Summary

 PE_m8250_RPMB_10#show mpls atm summ Total number of destinations: 6 ATM label bindings summary       interface   total  active   local  remote   Bwait   Rwait  IFwait       Switch1.1      10      10       6       4       0       0       0       Switch1.2       2       2       1       1       0       0       0 PE_m8250_RPMB_10# PE_m8250_RPMB_10#show mpls atm-ldp bindings 172.27.1.1 32 PE_m8250_RPMB_10#show mpls atm-ldp bindings 172.27.1.2 32 PE_m8250_RPMB_10#show mpls atm-ldp bindings 172.27.1.3 32 PE_m8250_RPMB_10# 

In the LSC, we can see that the tailend LVC on VPI 10 (from the eLSR we just configured) is no longer there. Now there are seven tailend LVCs, as shown in Example 6-97.

Example 6-97. Showing Tailend LVCs

 P1_LSC_c7204#show mpls atm-ldp bindings 172.27.1.1 32  Destination: 172.27.1.1/32     Tailend Switch XTagATM331 9/34 Active -> Terminating Active, VCD=19     Tailend Switch XTagATM133 2/34 Active -> Terminating Active, VCD=23     Tailend Switch XTagATM133 2/46 Active -> Terminating Active, VCD=24     Tailend Switch XTagATM133 2/58 Active -> Terminating Active, VCD=25     Tailend Switch XTagATM1341 27/34 Active -> Terminating Active, VCD=73     Tailend Switch XTagATM1341 27/58 Active -> Terminating Active, VCD=77     Tailend Switch XTagATM1341 27/64 Active -> Terminating Active, VCD=78 P1_LSC_c7204# 

We need to perform this procedure in all eLSRs and LSRs (in the LSCs) in the network.

NOTE

It is extremely important that we do not deny any eLSR address in the access list. If we do so, MPLS VPN connectivity will be broken because an LVC will not be bound to an eLSR destination.

This mechanism of applying an access list to the destinations that MPLS should request a label for can also be applied to any numbered link in the network.

Enabling VC-Merge

In very large networks, the best alternative to save LCN resources might be to enable VC-merge, also called multipoint-to-point. VC-merging is enabled by default in the LSC. It can be disabled with the global configuration command no mpls ldp atm vc-merge. However, for VC-merge to be supported, it must also be enabled in the VSI slave.

On BPX-8600 platforms, BXM VC-merge capability is displayed with the command dspcd. The command cnfcdparm enables VC-merge on a per-card basis. The VC-merge state can be displayed with the command dspcdparm.

From the LSC, as shown in Example 6-98, the VC-merge capability negotiated through LDP in the ATM Session Parameters TLV can be displayed using the command show mpls atm-ldp capability.

Example 6-98. Identifying the VC-Merge Capability

 P3_LSC_m8850_RPM-PR_9#show mpls atm-ldp capability                VPI           VCI           Alloc   Odd/Even VC Merge XTagATM1144    Range         Range         Scheme  Scheme   IN   OUT   Negotiated   [27 - 27]     [33 - 65535]  BIDIR            -    -   Local        [27 - 27]     [33 - 65535]  BIDIR   ODD      EN   EN   Peer         [27 - 27]     [33 - 65535]  BIDIR   EVEN     -    - P3_LSC_m8850_RPM-PR_9# 

In a VC-merging environment, VC-merged bindings can be displayed with the command show mpls atm-ldp bindings vc-merged. See Example 6-99.

Example 6-99. Showing VC-Merged Bindings

 P3_LSC_m8850_RPM-PR_9#show mpls atm-ldp bindings vc-merged  Destination: 172.27.1.128/32     Transit XTagATM101 0/121 Active -> XTagATM1114 0/41 Active     Transit XTagATM1111 0/113 Active -> XTagATM1114 0/41 Active 

In this command output, we can see how two different VPI/VCI pairs from two different interfaces are cross-connected to the same interface and VPI/VCI in a multipoint-to-point fashion.

We can use the same command (see Example 6-100) to check VC-merged bindings if multi-vc is enabled in the eLSRs.

Example 6-100. Showing VC-Merged Bindings with Multi-vc Enabled

 P3_LSC_m8850_RPM-PR_9#show mpls atm-ldp bindings vc-merged 172.27.1.128 32  Destination: 172.27.1.128/32     Transit XTagATM101 0/53 Active -> XTagATM1114 0/41 Active, CoS=available     Transit XTagATM1111 0/53 Active -> XTagATM1114 0/41 Active, CoS=available     Transit XTagATM101 0/55 Active -> XTagATM1114 0/45 Active, CoS=standard     Transit XTagATM1111 0/55 Active -> XTagATM1114 0/45 Active, CoS=standard     Transit XTagATM101 0/57 Active -> XTagATM1114 0/47 Active, CoS=premium     Transit XTagATM1111 0/57 Active -> XTagATM1114 0/47 Active, CoS=premium     Transit XTagATM101 0/59 Active -> XTagATM1114 0/49 Active, CoS=control     Transit XTagATM1111 0/59 Active -> XTagATM1114 0/49 Active, CoS=control 

When VC-merge is used in conjunction with multi-vc, only LVCs with the same service type are merged.

Tracing an LVC

Another common task in ATM MPLS networks is to trace the path an LVC is taking. The traceroute command does not work in cell-based MPLS networks because LSRs switch cells. To perform that function, three steps need to be carried out based on router functionality:

• Headend router Displays the FEC binding
  
  
• Transit router Displays the cross-connects
  
  
• Tailend router Displays the ATM VC
  
  

The headend router and tailend router always exist in an LVC. The transit router does not exist if there is a direct connection between eLSRs. Therefore, the first and third steps always need to be performed.

On the headend router (see Example 6-101), we check the binding using the command show mpls atm-ldp bindings for the specific destination and mask.

Example 6-101. Tracing an LVC Headend Router

 PE_Headend#show mpls atm-ldp bindings 10.1.2.0 24  Destination: 10.1.2.0/24     Headend Router ATM1/0/0.1 (2 hops) 6/38  Active, VCD=8, CoS=available PE_Headend# 

From the command show mpls atm-ldp bindings, we gather the output interface and VPI/VCI pair that the LVC is taking. We do this for the FEC defined by the destination, subnet mask, and service class.

The next step (see Example 6-102) is to check the cross-connects in all LSRs that the LVC is traversing as a transit binding. Using the command show mpls ldp neighbor, we verify that the neighbor P_Transit is connected to ATM1/0/0.1 in PE_Headend. To check the cross-connect, we use the command show xtagatm cross-connect using the VPI and VCI from the headend router and the interface connecting to the headend router.

Example 6-102. Tracing an LVC Transit Router

 P_Transit#show xtagatm cross-connect descriptor 0.9.2.0 | incl 6/38 0.9.2.0      6/38        ->     0.1.3.0      6/35        UP P_Transit# 

NOTE

The command show xtagatm cross-connect can be used with the parameter traffic to gather LVC traffic statistics from the LSC. These traffic counters reside in the VSI slave and are retrieved using VSI statistics commands. From AXSM VSI slaves in MGX platforms, the command dspchancnt can be used to get traffic counters.

We follow the LVC through the different LSRs it might traverse until we reach the tailend eLSR router. In this case, there is only one LSR between the eLSRs. At that instance (see Example 6-103), we check the ATM VC at the tailend router using the command show atm vc.

Example 6-103. Tracing an LVC Tailend Router

 PE_Tailend#show atm vc | include 35 1/0.1      5              6    35  TVC    MUX      UBR  155000                UP PE_Tailend# 

This is the last step, and with the information gathered, we know the path taken by this LVC. The path is shown in Figure 6-15.

Figure 6-15. Tracing an LVC