Keeping Statistics on Routing Table Changes

Problem

You want to keep statistics on how stable your routing table is.

Solution

IOS Version 12.0 introduced a seldom-used feature that allows you to statistically monitor fluctuations in your routing tables:

Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#ip route profile
Router(config)#end 
Router#

Once this command is enabled, you can check the stability of your network:

Router#show ip route profile
IP routing table change statistics:
Frequency of changes in a 5 second sampling interval
-------------------------------------------------------------
Change/ Fwd-path Prefix Nexthop Pathcount Prefix
interval change add change change refresh
-------------------------------------------------------------
0 327 327 335 335 331
1 4 4 0 0 1
2 2 2 0 0 1
3 0 0 0 0 0
4 1 1 0 0 1
5 1 1 0 0 1 
10 0 0 0 0 0
15 0 0 0 0 0 
20 0 0 0 0 0 
25 0 0 0 0 0 
30 0 0 0 0 0 
55 0 0 0 0 0 
80 0 0 0 0 0 
105 0 0 0 0 0 
130 0 0 0 0 0 
155 0 0 0 0 0 
280 0 0 0 0 0 
405 0 0 0 0 0 
530 0 0 0 0 0 
655 0 0 0 0 0 
780 0 0 0 0 0 
1405 0 0 0 0 0 
2030 0 0 0 0 0 
2655 0 0 0 0 0 
3280 0 0 0 0 0 
3905 0 0 0 0 0 
7030 0 0 0 0 0 
10155 0 0 0 0 0 
13280 0 0 0 0 0 
Overflow 0 0 0 0 0 
Router#

 

Discussion

It is often extremely difficult to tell when there are stability problems in your network. This command doesn't tell you much about what the problems are, but it can at least tell you how serious the problems are. However, to be useful, you must check these statistics periodically. Otherwise, it's difficult to know whether the issues highlighted occurred recently.

To explain what the output means, let's look at the first few lines in more detail:

Router#show ip route profile
IP routing table change statistics:
Frequency of changes in a 5 second sampling interval
-------------------------------------------------------------
Change/ Fwd-path Prefix Nexthop Pathcount Prefix
interval change add change change refresh
-------------------------------------------------------------
0 327 327 335 335 331
1 4 4 0 0 1
2 2 2 0 0 1
3 0 0 0 0 0

The first column indicates the number of changes per five-second interval. So, for example, you can see that the Fwd-path change column has a 4 in the 1 row and a 2 in the 2 row. This means that there were four 5-second intervals that saw one change and two 5-second intervals with two changes of this type. To put it another way, since this is a little confusing at first, the router is ticking along watching for routing table changes. It waits five seconds and notices that there was a single change of this type during this interval, so it increments the counter in the row labeled 1. In the next five-second interval, there may be two changes of this type, so it will increment the counter in the row labeled 2. After a while, this output shows that there have been 327 intervals with zero changes, 4 intervals with 1 change, 2 intervals with 2 changes, and no intervals with 3 changes.

As you go further down in the output of this command, you see more and more serious problems. Ideally, you want to see zeroes in every row except the first few rows. High numbers in the 0 row are normal.

Now let's look at the columns to explain what these numbers actually mean. The first column is labeled Fwd-path change. This represents changes in the forwarding path, and is essentially the total of the Prefix add, Nexthop change, and Pathcount change columns.

The Prefix add column counts the number of new IP route prefixes that have been added to the routing table. Note that the route profile feature does not keep track of deletions, only additions to the table.

Nexthop change shows how often the next hop parameter associated with an existing route changes. This most likely indicates that there has been a topology change. This is particularly useful for troubleshooting because you will only see numbers in this column on devices that are immediately adjacent to the trouble spot in your network.

In the Pathcount change column, you see information on routes with multiple paths. For example, if there are normally 4 redundant paths to a particular destination, and one or more of these paths is flapping up and down, you will see the numbers in this column increment. This is useful because in a highly redundant network, you might not be aware that there are problems, as the routing protocol may be covering them up for you.

Finally, the Prefix refresh column keeps track of how often the routing protocol has refreshed routes, without changing the actual routing information. Different routing protocols handle route refreshes differently, so you should watch this column to get an idea of what is normal for your network.

Although this command is a useful indicator of routing problems, it has several serious shortcomings that you should be aware of. First, it lacks a simple clear command to reset the table and eliminate old data. The only way to do this that we are aware of is to disable and re-enable the feature:

Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#no ip route profile
Router(config)#ip route profile
Router(config)#end 
Router#

A second problem with this feature is that there is no simple way to deduce what route prefixes, next-hops, or even what interfaces have been affected by instability. A final problem is that you can get very high numbers in some columns just from a few changes to every route in a large routing table. So it is difficult to tell from this command alone whether you have a small problem with a lot of routes or a big problem with a few routes.

Despite these shortcomings, this feature can be a helpful adjunct to other troubleshooting methods, as it does track some issuesparticularly network layer issues, which can be much harder to spot.