Troubleshooting IS-IS Routing Update Problems

‚  < ‚  Free Open Study ‚  > ‚  

Configuration in IS-IS is fairly simple and straightforward. In the two-stage process discussed in Chapter 10, the routing process is enabled globally, and IS-IS adjacency formation and LSP flooding is enabled on an interface by applying the command ip router isis. This command also puts the IP subnet information of the interface into the router's LSP that is generated and flooded to adjacent neighbors. This section covers IS-IS routing update problems on the premise that there are no adjacency problems. This essentially implies that troubleshooting any routing update problems should start with verifying the appropriate adjacencies. Adjacency problems and related troubleshooting methodology are discussed extensively in the previous sections.

The following routing update problems are covered in this section:

  • Route advertisement problems

  • Route flaps

  • Route redistribution problems

One important method for troubleshooting routing problems in IS-IS is by direct inspection of the contents of link-state packets (LSPs). Depending on its configuration, an IS-IS router generates an LSP for each of the levels of routing that it participates in ‚ Level 1 LSP, Level 2 LSP, or both. Inspection of the LSP contents of the expected source of a route can help diagnose routing advertisement problems in cases when no obvious adjacency problems exist. The show isis database detail command displays the contents of a specific LSP. Example 11-25 shows sample output from this command.

Example 11-25 Displaying the Routing Information in an LSP
 RT1#  show isis database level-1 RT2.00-00 detail  IS-IS Level-2 LSP RT2.00-00 LSPID             LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL RT2.00-00         0x00001C9C   0x5F3E        1015              0/0/0   Area Address: 49.0002   NLPID:        0xCC   Hostname: RT2   IP Address:   11.1.1.2   Metric: 10         IS-Extended RT1.00   Metric: 10         IP 10.1.2.0/24   Metric: 0          IP 11.1.1.2/32   Metric: 10         IP 11.1.1.6/32   Metric: 10         IP 192.168.1.0/30 

RT2.00-00 is the LSP ID of RT2. Detail output of the LSP, with ID RT2.00-00, shows the IP subnets for directly connected links, together with their metric information.

Another interesting command is show isis topology, which displays a list of all known routers. For example, Example 11-26 shows the IS-IS topology for Figure 11-6 as captured on RT1.

Figure 11-6. A Simple IS-IS Network Topology

graphics/11fig06.gif

The backbone (Level 2) is the shaded area. Example 11-26 shows the Level 1 and Level 2 data-bases of RT1. The Level 1 database features RT3 and RT5, confirming that these routers are indeed in the same area as RT1. The Level 2 database describes the backbone, which features RT2 and RT3. RT2 is not in the same area as RT1, but it is connected to the backbone, whereas RT3 is a Level 1 ‚ 2 router in the same area as RT1.

All this information gleaned from Example 11-26 agrees with the layout in Figure 11-6; this makes the show isis topology command an invaluable command for troubleshooting routing-related problems.

Example 11-26 Obtaining IS-IS Topology Information
 RT1#  show isis topology  IS-IS paths to level-1 routers System Id       Metric  Next-Hop        Interface       SNPA RT1             -- RT3             10      RT3             Et0/0           00d0.58eb.f841 RT5             10      RT5             Et0/0           00d0.58eb.ff01 IS-IS paths to level-2 routers System Id       Metric  Next-Hop        Interface       SNPA RT1             -- RT2             10      RT2             Se0/0           *HDLC* RT3             10      RT3             Et0/0           00d0.58eb.f841 

Route Advertisement Problems

Most route advertisement problems can be narrowed down to configuration problems at the source or LSP propagation issues.

Suppose that there is concern that a specific route is missing on RT5 (see Figure 11-6). You can start troubleshooting the problem by obtaining more information on the topology of the network. This should help you determine which router is the original source of the route. Suppose that the route 11.1.1.2/32 turns out to be the loopback address of RT2; the flowchart in Figure 11-7 can be followed to narrow the cause of the problem.

Figure 11-7. Flowchart for Troubleshooting Missing Routes

graphics/11fig07.gif

Using knowledge about the topology, you know that RT2 and RT5 are not in the same area. In this case, if route leaking is not enabled in the network, RT5, which is a Level 1 ‚ only router, should not see any route from RT2. Hence, tracking the problem takes you along the flowchart through boxes 0-1-7-10-5, where you determine that this is normal behavior. As a Level 1 ‚ only router, RT5 should follow a default router to the nearest Level 1 router to get to remote areas.

If you end up in Box 9 or Box 11 along the flowchart, the next case studies and flowcharts might help narrow the problem.

The procedure covered by the flowchart in Figure 11-7 provides a generic method for trouble-shooting missing route problems. The following sections discuss procedures for specific scenarios.

Local Routes Not Being Advertised to Remote

Because IS-IS is a link-state protocol, IS-IS routers depend on LSP flooding to obtain topology and routing information. For example, during stable conditions, each IS-IS router in an area will have the same Level 1 link-state database, which contains the LSPs generated by every router in the area. Dijkstra's algorithm is run over the LS database to obtain the best path to every advertised route. Therefore, if a route is missing at a section of the area, it's because the routers in that section did not receive the original LSP, or the LSP was received corrupted and, there-fore, was purged. An even simpler reason could be that the route was not even put into the LSP at the source. The flowchart in Figure 11-8 provides the troubleshooting methodology for such problems. The output of debug isis update-packets and debug isis snp-packets, as demon-strated in Example 11-27, could help decipher this sort of problem if it is related to LSP flooding or issues with link-state database synchronization.

Figure 11-8. Flowchart for Troubleshooting Route Advertisement Problems

graphics/11fig08.gif

Step 8 of the flowchart for troubleshooting route advertisement problems calls for enabling the debugging of LSP updates if it is determined that an LSP is not making it to certain remote locations of the IS-IS domain. Example 11-27 shows an output of the debug isis update-packets command. The highlighted lines show RT1 flooding its LSP and also receiving an LSP from RT2. Because the adjacency was just brought up, the output also shows the one-time exchange of CSNPs on point-to-point links between RT1 and RT2.

Example 11-27 Debugging IS-IS Routing Update Problems
 RT1#  debug isis update-packets  Mar  2 23:25:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chp Mar  2 23:25:02: ISIS-Update: Building L2 LSP Mar  2 23:25:02: ISIS-Update: No change, suppress L2 LSP 0000.0000.0001.00-00,0  Mar  2 23:25:03: %CLNS-5-ADJCHANGE: ISIS: Adjacency to RT2 (Serial0/0) Up, newy  Mar  2 23:25:07: ISIS-Update: Building L2 LSP Mar  2 23:25:07: ISIS-Update: TLV contents different, code 16 Mar  2 23:25:07: ISIS-Update: Full SPF required  Mar  2 23:25:07: ISIS-Update: Sending L2 LSP 0000.0000.0001.00-00, seq 160, ht0   Mar  2 23:25:07: ISIS-Update: Rec L2 LSP 0000.0000.0002.00-00, seq 1D16, ht 11,  Mar  2 23:25:07: ISIS-Update: from SNPA *HDLC* (Serial0/0) Mar  2 23:25:07: ISIS-Update: LSP newer than database copy Mar  2 23:25:07: ISIS-Update: No change  Mar  2 23:25:08: ISIS-SNP: Rec L2 CSNP from 0000.0000.0002 (Serial0/0)  Mar  2 23:25:08: ISIS-SNP: Rec L2 PSNP from 0000.0000.0002 (Serial0/0) Mar  2 23:25:08: ISIS-SNP: PSNP entry 0000.0000.0001.00-00, seq 160, ht 1197  Mar  2 23:25:08: ISIS-Update: Sending L2 CSNP on Serial0/0  Mar  2 23:25:08: ISIS-Update: Build L2 PSNP entry for 0000.0000.0002.00-00, se6 Mar  2 23:25:08: ISIS-Update: Build L2 PSNP entry for 0000.0000.0006.00-00, se2 Mar  2 23:25:08: ISIS-Update: Sending L2 PSNP on Serial0/0 Mar  2 23:25:09: ISIS-Update: Building L1 LSP Mar  2 23:25:09: ISIS-Update: Important fields changed Mar  2 23:25:09: ISIS-Update: Important fields changed Mar  2 23:25:09: ISIS-Update: Full SPF required Mar  2 23:25:09: ISIS-Update: Sending L1 LSP 0000.0000.0001.00-00, seq 15A, ht0 Mar  2 23:25:09: ISIS-Update: Sending L1 CSNP on Ethernet0/0 _____________________________________________________________________________________ RT5#  debug isis snp-packets  IS-IS CSNP/PSNP packets debugging is on RT5#  Mar  6 20:02:28: ISIS-SNP: Rec L1 CSNP from 0000.0000.0001 (Ethernet0/0)  Mar  6 20:02:28: ISIS-SNP: CSNP range 0000.0000.0000.00-00 to FFFF.FFFF.FFFF.FFF  Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0001.00-00, seq 15D  Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0001.01-00, seq 104 Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0005.00-00, seq FEA 

The debug isis snp-packets command enables debugging of database synchronization on broadcast interfaces and can troubleshoot route update problems over media such as LANs.

The highlighted line indicates receipt of a CSNP by RT5 from the DIS (RT1). By comparing the contents of the CSNP with the local Level 1 database, RT5 determines that no changes occurred in all known LSPs.

Solution Summary

As depicted by the flowchart in Figure 11-8, there could be many potential causes for problems in which routes are not reaching remote locations in the network. In the extreme case, the route might not be making it into the routing table because of a software glitch relating to IS-IS data structures (Steps 4 and 5). Such situations might need to be reported to the Cisco Technical Assistance Center (TAC) for further assistance.

In other cases, the LSPs might not be reaching some parts of the network because they are getting corrupted in flight (Step 9). Such problems can be determined by a combination of activities, such as debugging the update process and monitoring router logs for LSP errors. If the culprit device is located, it can be isolated or replaced to address the problem.

In most cases, however, the problem could be caused by a trivial configuration error, such as IS-IS not being enabled on an interface (Step 11). Applying the appropriate interface command ( ip router isis ) to the interface should be sufficient to address the problem.

Situations involving route redistribution or route-leaking problems are addressed in the next section.

Route Redistribution and Level 2-to-Level 1 Route-Leaking Problems

Cisco IOS Software allows routes from other routing sources, such as static, connected interfaces, and other routing protocols, to be redistributed into IS-IS Level 1 routing, Level 2 routing, or both as external routes. Technically, external routes should exist only in the Level 2 routing environment, but Cisco provides a configuration attribute that allows redistribution into Level 1 for practical operational purposes. For example, service providers using IS-IS as IGP with flat Level 1-only domains might need this capability. The existence of the IP External Reachability TLV in a Level 1 LSP should not pose any interoperability issues because IS-IS routers are required to skip TLVs that they don't understand when they parse an IS-IS packet.

Redistribution in IS-IS is straightforward with hardly any potential pitfalls except occasional , unexpected software bugs . Sometimes, effective metric assignment becomes a challenge in redistribution scenarios. When configuring redistribution, the operator is asked to specify internal or external metrics. The default is the external metric type, which bumps up the metric by 128 on Cisco routers. The increase by 128 instead of 64 is the result of an incorrect bit setting in the default metric field when using narrow metrics. This issue is discussed in the configur-ation section as a Cisco IOS Software implementation issue that has been retained for backward compatibility.

The rather simple flowchart in Figure 11-9 should provide guidance for troubleshooting redistribution problems.

Figure 11-9. Flowchart to Resolve Redistribution Problems

graphics/11fig09.gif

Route-Flapping Problem

Route flaps normally are caused by unstable links between the source of the route and the location where the flap is being observed . Typically, multiple routes are impacted at the same time because of an adjacency change affecting many LSPs, all of which might carry numerous IP prefixes. Also, route flaps could induce consistent running of the SPF process, causing potentially dangerous high CPU utilization on route processors that might crash affected routers. If the advertised LSP changes affect only IP prefixes, only partial route calculation (PRC) is run instead of full SPF calculations. PRC is less costly in terms of CPU cycles than full SPF runs. The flowchart in Figure 11-10 provides guidance for troubleshooting route-flapping problems.

Figure 11-10. Flowchart for Troubleshooting Route-Flapping Problems

graphics/11fig10.gif

Apart from certain destinations not being reachable ‚ obviously implying routing problems ‚ high CPU utilization by the SPF process certainly should flag instabilities in the network. High CPU utilization can be observed through the IOS command line interface (CLI), network-management applications, or special network-performance analysis tools, such as CiscoWorks 2000, HP Openview, and so on.

From the Cisco IOS Software CLI, the show process cpu command can obtain CPU utilization information. If any indication of high CPU utilization caused by the SPF process is determined, the show isis spf-log command can determine SPF-related events that might cause the high CPU utilization. Example 11-28 provides sample output of this command.

Example 11-28 Tracking SPF Process-Related Events
 RT1#  show isis spf-log  Level 1 SPF log   When   Duration  Nodes  Count     Last trigger LSP   Triggers  03:40:08       0      3      1                        PERIODIC  03:25:08       0      3      1                        PERIODIC 03:10:07       0      3      1                        PERIODIC 02:55:07       0      3      1                        PERIODIC 02:40:07       0      3      1                        PERIODIC 02:25:06       0      3      1                        PERIODIC 02:10:06       0      3      1                        PERIODIC 01:56:08       0      2      1             RT1.01-00  TLVCONTENT 01:55:06       0      2      1                        PERIODIC 01:40:06       0      2      1                        PERIODIC 01:36:31       0      2      1             RT5.00-00  LSPEXPIRED 01:28:31       0      2      2             RT1.01-00  NEWADJ TLVCONTENT  01:28:25       0      3      1             RT5.00-00  NEWLSP  01:25:06       0      3      1                        PERIODIC 01:10:06       0      3      1                        PERIODIC 

The output in Example 11-28 lists SPF process runs by time, trigger, and duration. You might recall that LSPs are refreshed every 15 minutes, triggering periodic SPF calculations. Such events are labeled periodic in the show isis spf-log output. It also shows that at 01:25:25, the process was run over an insignificant period of time because of receipt of a new LSP from RT5. Table 11-2 lists and describes common SPF triggers.

Table 11-2. SPF Process Triggers
Trigger Code Description
AREASET Area change
ATTACHFLAG Attached bit setting change
CLEAR Manual clear
CONFIG Configuration change
DELADJ Adjacency deletion
DIS DIS election
ES End system information change
HIPPITY LSPDB Overload bit state change
IP_DEF_ORIG Default information change
IPDOWN Connected IP prefix down
IP_EXTERNAL Route redistribution change
IPIA Interarea route change
IPUP Connected IP prefix up
NEWADJ New neighbor adjacency up
NEWLEVEL IS-IS process level changed
NEWMETRIC New metric assigned to an interface
NEWSYSID New system ID assigned
PERIODIC Periodic SPF process (LSPDB refresh interval)
TLVCODE LSP with a new TLV code field received
TLVCONTENT LSP with changed TLV contents received

The following IS-IS debugging commands also can narrow SPF-related problems:

  • debug isis spf-events

  • debug isis spf-triggers

  • debug isis spf-statistics

  • debug isis update-packets

  • debug isis adj-packets

Use caution when enabling debugging in situations in which the route processor's CPU already is overtaxed. Example 11-29 provides sample output of the debug isis spf-events command, which displays the events following an interface shut to simulate a link flap. Lines indicating LSPs flagged for recalculation as a result of this event are highlighted. Also, events flagging computation of the Level 1 and Level 2 SPF trees are highlighted.

Example 11-29 debug isis spf-events Command Output Displays Events Following an Interface Shut
 RT1(config-if)#  debug isis spf-events   Mar  6 20:17:26: ISIS-SPF: L1 LSP 1 (0000.0000.0001.00-00) flagged for recalculC   Mar  6 20:17:26: ISIS-SPF: L1 LSP 5 (0000.0000.0005.00-00) flagged for recalculC   Mar  6 20:17:28: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to admin down   Mar  6 20:17:28: ISIS-SPF: Compute L1 SPT  Mar  6 20:17:28: ISIS-SPF: 3 nodes for level-1 Mar  6 20:17:28: ISIS-SPF: Move 0000.0000.0001.00-00 to PATHS, metric 0 Mar  6 20:17:28: ISIS-SPF: Add 0000.0000.0001.01-00 to TENT, metric 10 Mar  6 20:17:28: ISIS-SPF: Add 0000.0000.0001 to L1 route table, metric 0 Mar  6 20:17:28: ISIS-SPF: Move 0000.0000.0001.01-00 to PATHS, metric 10 Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 1 (0000.0000.0001.00-00), version 214 Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 2 (0000.0000.0001.00-00), version 208 Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 3 (0000.0000.0001.01-00), version 207 Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 4 (0000.0000.0002.00-00), version 209 Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 5 (0000.0000.0005.00-00), version 207 Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 6 (0000.0000.0006.01-00), version 112 Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 7 (0000.0000.0006.00-00), version 114 Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 8 (0000.0000.0001.01-00), version 1 Mar  6 20:17:29: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0  Mar  6 20:17:33: ISIS-SPF: Compute L2 SPT   Mar  6 20:17:33: ISIS-SPF: 5 nodes for level-2  Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0001.00-00 to PATHS, metric 0 Mar  6 20:17:33: ISIS-SPF: Add 49.0001 to L2 route table, metric 0 Mar  6 20:17:33: ISIS-SPF: Add 0000.0000.0001.01-00 to TENT, metric 10 Mar  6 20:17:33: ISIS-SPF: considering adj to 0000.0000.0002 (Serial0/0) metric Mar  6 20:17:33: ISIS-SPF:   (accepted) Mar  6 20:17:33: ISIS-SPF: Add 0000.0000.0002.00-00 to TENT, metric 10 Mar  6 20:17:33: ISIS-SPF:   Next hop 0000.0000.0002 (Serial0/0) Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0001.01-00 to PATHS, metric 10 Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0002.00-00 to PATHS, metric 10 Mar  6 20:17:33: ISIS-SPF: Add 49.0002 to L2 route table, metric 10 Mar  6 20:17:33: ISIS-SPF: Redundant IP route 10.1.2.0/255.255.255.0, metric 20d Mar  6 20:17:33: ISIS-SPF: Redundant IP route 11.1.1.2/255.255.255.255, metric d Mar  6 20:17:33: ISIS-SPF: Redundant IP route 11.1.1.6/255.255.255.255, metric d Mar  6 20:17:33: ISIS-SPF: Add 192.168.1.0/255.255.255.252 to IP route table, m0 Mar  6 20:17:33: ISIS-SPF: Next hop 0000.0000.0002/192.168.1.2 (Serial0/0) (rej) 
Solution Summary

As shown in the flowchart for troubleshooting route flapping problems (see Figure 11-10), most route-flapping problems can be traced to unstable links in the network (see Step 2). Physical connectivity problems are addressed by troubleshooting the physical and data link layers .

In more complicated scenarios, however, the flaps could be caused by LSP corruption storms or even a routing loop. This might be the case when there are no unstable links in the network but CPU utilization is high, indicating continuous running of the SPF process (see Step 4). The show isis spf-log command can provide indication of which LSPs are changing the most frequently and are triggering the SPF calculations. Similar clues can be gleaned by enabling debugging of the update process with debug isis update-packets. This should be done with care so that the CPU is not overloaded. The logs also can be observed for LSP error loggings, which could give an indication of packet corruption by a culprit device (see Step 7). Zeroing in on any continuously changing LSPs is critical for determining and addressing the problem.

‚  < ‚  Free Open Study ‚  > ‚  


Troubleshooting IP Routing Protocols
Troubleshooting IP Routing Protocols (CCIE Professional Development Series)
ISBN: 1587050196
EAN: 2147483647
Year: 2002
Pages: 260

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net