Correlating Different Router Performance Variables | Performance and Fault Management: A Practical Guide to Effectively Managing Cisco Network Devices (Cisco Press Core Series)

After you've gathered the MIB or CLI data on router performance, what can you do with it? This section provides some ideas on how to correlate the different data points you gathered for use in performance management. A lot of what you gather in performance management can be applied to fault/error management as well, as noted in the preceding discussion of buffer variables. We'll discuss some options to assist performance based on thresholds being exceeded.

Correlating High CPU Values

Based on a high avgBusy5 value, you can look at the individual processes running on the router by executing the following CLI command: show proc cpu. From this output, you can identify what tasks are taking up the majority of the CPU cycles. You may also want to look at the switching paths of the data flowing through the router, such as process, fast, and autonomous switching. By using the CLI commands show interface stat or show interface switch, you can identify how much traffic is process switched versus the other switching mechanisms (fast, autonomous, distributed, and so on.) per interface. Wherever possible, try to offload some of the process switched traffic to other switching paths. Configuration parameters that involve route-caches on the interfaces can be applied to address these process-switching issues. Buffer misses can also be attributed to process-switched packets, which can be corrected by turning on some faster switching mechanisms.

High CPU can also be caused by broadcasts hitting the router, which cannot be controlled by adjusting the switching paths because broadcasts are always process switched. Broadcasts such as routing updates, IPX SAPs, or IP-UDP broadcasts have been known to cause high CPU usage on routers.

NOTE

For more details on performance numbers relating to broadcasts and CPU, see the case study entitled "Broadcasts in Switched Internetworks" on CCO. See the References at the end of this chapter for specific URL locations.

Here are some general suggestions for controlling the amount of broadcasts in your network. To control routing updates, depending on the protocol, you can use a different routing protocol that utilizes neighbor adjacencies to relay information about routes, such as EIGRP or OSPF. It is also a good idea to have a good network design in place to minimize the amount of broadcasts through your network, such as route summarization. To control IPX SAPs, you can use incremental SAP updates by using a subset of EIGRP for IPX and the rsup option, or you can use SAP access-list filters. For UDP broadcasts, there are several options: UDP turbo flooding or IP Helpering. Refer to the case study regarding UDP broadcast flooding on CCO for more details on setting this up in the router.

NOTE

For more details on controlling broadcasts in networks refer to the following case studies on CCO:

"Designing Large-Scale IP Internetworks"

"Reducing SAP Traffic in Novell IPX Networks"

"UDP Broadcast Flooding"

See the References at the end of this chapter for specific URL locations.

Correlating High Memory Usage and Fragmentation

High memory utilization can be caused by several things: instability in routing tables (flapping routes), memory leaks, or just not enough memory to handle the software and processes running on the router. Memory fragmentation in routers typically cannot be cleared unless the router is reloaded. Sometimes, you can free up some contiguous memory by turning off some of the unstable processes. However, the majority of the time the router needs to be reloaded. This is why it is important to trend the lowest fragmented memory and the process holding memory so you can proactively determine the process taking up memory and when the router may start experiencing memory fragmentation.

Correlating Buffer Misses

We won't get into the details of buffer-tuning the routers here because it is not advisable to tune them without consulting Cisco first. But we will discuss some general guidelines and hints for tuning buffers. Do a show memory to see how much memory is free. Creating buffers uses memory. In the low-end boxes, such as the 4000, 2500, etc., the buffers come out of shared I/O memory. In the high-end boxes, such as the 7xxx series, system memory is used. When you add buffers, you can multiply the number of permanent buffers you create by their size to see how much memory you will start off using. You will often see high-end boxes (and now some of the low-end) with 12 or 13 MB of free memory. Don't be afraid of using it if you have a problem.

Check the no memory field at the bottom of the show buffers output. If you see a non-zero number here, it means that you ran out of memory trying to make buffers. Tuning probably won't help. You are most likely to see this in the low-end routers when there is not enough shared I/O memory. You'll need to upgrade the shared memory first and then tune if necessary. In rare cases, a non-zero number in the no memory field could also warn you of a memory leak. A show memory could confirm this.

There are no firm rules for buffer tuning, but here are some useful guidelines:

Take the number of total buffers in a pool, add about 20 percent (depending on how many misses you are getting and how many buffers you currently have free), and make that your Perms for that buffer pool.
Set min to about 20 30 percent of your perms.
Set max to something greater than the sum of perms and mins.

Here's an example from a show buffers output:

 Small buffers, 104 bytes (total 356, permanent 120):      347 in free list (50 min, 350 max allowed)      7254759 hits, 45947 misses, 45351 trims, 45587 created

Here, the router has total of 356 buffers, and it's getting a fair number of misses. Use the following guidelines:

Round up and make the permanent = 430 with the following command (assuming there is enough memory installed):

 buffers small perm 430

You should probably double the mins of 50 to 100, which puts you in the middle of the guideline of 20 30 percent of perms (.25 x 425 = 106), so you would configure the following:

 buffers small min 100

425 + 100 = 525, so you should probably set max to about 600 with the following command:

 buffers small max 600

Some final words regarding buffer tuning:

Buffer tuning won't cure things such as explorer storms, overutilization of the network, or too-slow serial links.
If you create lots of buffers and you still get lots of misses, you probably should try to get an understanding of where the traffic is coming from so you can control it through filtering or loop elimination.
Always try to leave some free memory to allow for changing conditions.
Buffer tuning is not a substitute for good network design, implementation, or management.
Buffer tuning is more of an art than a science, so please consult with an experienced Cisco engineer prior to tuning the buffers.