Tuning Router Buffers

Problem

You want to change your default buffer allocations to improve router efficiency.

Solution

The router maintains two different sets of buffers: public buffers and interface buffers. The router uses these as temporary storage while processing packet data. You can tune the public buffer pools as follows:

Router1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router1(config)#buffers big initial 100
Router1(config)#buffers big max-free 200
Router1(config)#buffers big min-free 50
Router1(config)#buffers big permanent 50
Router1(config)#end
Router1#

And you can adjust the interface buffer pools by using a similar set of commands:

Router1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router1(config)#buffers Ethernet0 initial 200
Router1(config)#buffers Ethernet0 max-free 300
Router1(config)#buffers Ethernet0 min-free 50
Router1(config)#buffers Ethernet0 permanent 50
Router1(config)#end
Router1#

 

Discussion

Before we start this discussion, we need to offer three notes of caution on tuning buffers. First, adjusting your router's buffers is usually not necessary. Second, a poor set of buffer parameters can cause serious performance problems on your router and for traffic passing through the router. Third, if you do find that you need to adjust these parameters, the necessary adjustments will be unique to your network, and perhaps even to each router, so we unfortunately can only offer general guidance, but can't really recommend appropriate parameters.

The router maintains two different sets of buffers: public pools that the router can use for anything, and interface specific pools that it can use only for processing packets on that interface.

The public buffers fall into several different pools, according to their size. They are shown in Table 2-1.

Table 2-1. Public buffer pools

Buffer size	Buffer pool name
104 bytes	Small buffers
600 bytes	Middle buffers
1,536 bytes	Big buffers
4,520 bytes	VeryBig buffers
5,024 bytes	Large buffers
18,024 bytes (default)	Huge buffers

Note that the Huge buffers are 18,024 bytes by default. But unlike the other public buffer pools, you can actually change the size of the buffers in this pool as follows:

Router1(config)#buffers huge size 36048

You can configure any size between 18,024 and 100,000 bytes for your Huge buffers. However, we should mention that it is extremely rare to find that you need to change this buffer size. Since the router can use memory only in buffer-sized chunks, having extremely large buffers can be useful if you find that you need to manipulate extremely large packets for some reason. However, the default value of 18,024 should be large enough to handle the largest MTU values for all standard interface types. So it is extremely rare to find that you actually have to adjust this parameter. The remaining buffer sizes are all fixed and cannot be adjusted.

There are four different parameters that you can adjust on each of the public buffer pools:

Router1(config)#buffers big initial 100
Router1(config)#buffers big max-free 200
Router1(config)#buffers big min-free 50
Router1(config)#buffers big permanent 50

The first of these commands sets the number of buffers of this type that the router will allocate at boot time. If this router is in an extremely high traffic environment, it may take a while to allocate enough buffers to handle the load. So you may find that the router has a few buffer failures right after booting. You can resolve this problem by increasing the number of initial buffers.

The second command uses the keyword max-free to set the maximum number of free buffers of this type that the system should keep. In the router's normal functioning, it will see periodic bursts of activity that may force it to allocate more buffers. When the burst is over, setting a relatively low value for max-free will ensure that the router frees this extra memory to make it available for other purposes. But if you set it too low in an extremely bursty environment, the router may not be able to allocate new buffers quickly enough to meet the demand.

In the third command, we have applied the min-free keyword to take care of the opposite side of the same problem. In order to help ensure that the router is able to handle the rising demand for packets, as soon as the router finds that it has fewer than min-free more unused buffers of a particular type, it will start allocating more from system memory. If you specify a min-free value that is large enough, the router will be able to cope with any demands. But making the value too large will force the router to do additional work by allocating additional buffers that it will never need.

In the final command we set the minimum number of buffers of this type by using the keyword permanent. The router will allocate this many buffers at boot time, and it will not return their memory to the general pool of memory. A good value for this parameter is high enough to reduce the amount of work that the router has to do allocating and trimming buffers, but not so high as to waste precious memory resources.

As you can see from the second example, the parameters for tuning the interface buffer pools are exactly the same as the ones we have just described for the public pools:

Router1(config)#buffers Ethernet0 initial 200
Router1(config)#buffers Ethernet0 max-free 300
Router1(config)#buffers Ethernet0 min-free 50
Router1(config)#buffers Ethernet0 permanent 50

The best way to tell whether your buffers need adjusting is to look at the output of the show buffers command:

Router1>show buffers
Buffer elements:
 498 in free list (500 max allowed)
 760166 hits, 0 misses, 0 created

Public buffer pools:
Small buffers, 104 bytes (total 50, permanent 50):
 50 in free list (20 min, 150 max allowed)
 265016 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)
Middle buffers, 600 bytes (total 25, permanent 25, peak 49 @ 1d09h):
 23 in free list (10 min, 150 max allowed)
 40749 hits, 10 misses, 30 trims, 30 created
 0 failures (0 no memory)
Big buffers, 1536 bytes (total 50, permanent 50):
 50 in free list (5 min, 150 max allowed)
 33780 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
 10 in free list (0 min, 100 max allowed)
 0 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
 0 in free list (0 min, 10 max allowed)
 0 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
 0 in free list (0 min, 4 max allowed)
 0 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)

Interface buffer pools:
Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
 8 in free list (0 min, 32 max allowed)
 24 hits, 0 fallbacks
 8 max cache size, 8 in cache
 30963 hits in cache, 0 misses in cache
Serial0 buffers, 1524 bytes (total 32, permanent 32):
 4 in free list (0 min, 32 max allowed)
 54 hits, 3 fallbacks
 8 max cache size, 7 in cache
 172593 hits in cache, 32 misses in cache
Serial1 buffers, 1524 bytes (total 32, permanent 32):
 7 in free list (0 min, 32 max allowed)
 25 hits, 0 fallbacks
 8 max cache size, 8 in cache
 0 hits in cache, 0 misses in cache


Router1>

First, let us zoom in on one of the public buffer pools to explain what the fields mean:

Small buffers, 104 bytes (total 50, permanent 50):
 50 in free list (20 min, 150 max allowed)
 265016 hits, 0 misses, 0 trims, 0 created
 0 failures (0 no memory)

This section looks at Small buffers, which are 104-byte chunks of memory. The router currently has allocated a total of 50 of these buffers; all 50 of them are permanent, meaning that the router will not attempt to return any of them to the pool of generally available memory.

In the second line, you can see that all 50 of these buffers are currently in the free list, meaning that they are all unused. The numbers 20 and 150 in this line are the min-free and max-free parameters that we discussed above.

In the third line, the number of hits indicates how many times the router has successfully allocated buffers from this pool. The number of misses indicates how many times the router successfully allocated a buffer from this pool, but in doing so had to allocate additional buffers. The field called trims counts the number of dynamically allocated buffers that the router has subsequently returned. And the created field shows how many buffers the router has actually created in response to miss events.

The last line shows serious problems, which are the only reason that you should alter your buffer parameters. The failures field counts the number of times that the router has attempted to allocate a buffer and failed, causing it to drop the packet. The last field is labeled "no memory." It counts the number of times a failure happened because the router had no memory from which to allocate additional buffers. This is clearly an extremely serious problem, which is usually best treated by adding memory to the router.

It is also important to remember that if the router tries and fails to allocate a buffer from one pool, it will request a buffer from the next largest pool. So, for example, if the router is unable to get a Big buffer to handle a 1,500 byte packet, it will use one from the VeryBig pool. This is why you can sometimes see buffer hits in the VeryBig pool, even if every interface on the router has an MTU of 1,500 bytes. So it is a good idea to allocate a few permanent buffers from the pool larger than your highest MTU.

Now let's look at the interface buffers:

Ethernet0 buffers, 1524 bytes (total 32, permanent 32):
 8 in free list (0 min, 32 max allowed)
 24 hits, 0 fallbacks
 8 max cache size, 8 in cache
 30963 hits in cache, 0 misses in cache

This shows a similar set of values to what we just discussed for the public buffer pools, but there are a few differences. The first difference is the fallbacks field. This counts the number of times that the router has needed additional buffers on this interface, and has allocated them from the corresponding public buffer pool of the appropriate size. In this case, the Ethernet buffers are 1,524 bytes, so the router would allocate additional buffers from the big buffer public pool.

The router keeps a cache of buffers on each interface that are effectively in use whether there is data or not. This field varies somewhat depending on the hardware type. But once again, you should watch out for misses. As long as the number of misses and fallbacks are low, there is no need to adjust the interface buffers.

We would like to offer one final warning about adjusting buffers. Always be sure to look at your router's free memory with the show memory before and after making any adjustment:

Router1#show memory 
 Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 17DA4C 13112756 2308632 10804124 10577100 10663072
 I/O E00000 2097152 336980 1760172 1740988 1759812

Keep close track of how much the Free memory in particular changes when you adjust your router's buffers. Both the Processor and the I/O memory can be affected by these changes. If you inadvertently over allocate your buffers while trying to improve system performance, you may find that the router does not have enough memory to operate properly when the load increases.