Memory Load Diagnostics | Troubleshooting Linux Firewalls

Sometimes everything is right about the firewallthe rules are fine, the drivers are installed correctly, and in fact everything seems perfect with the firewall except that it is just not quite working properly. Often, these hard to pin down problems can stem from the firewall simply being overloaded and not having enough memory or the connection tracking settings not being optimized for the network or system on which the firewall is running.

The connection tracking engine requires a certain amount of RAM to function properly, and, probably not too obviously, it takes more RAM to track more connections. The more connections and the more users, the more memory the firewall needs. To determine if this might be part of the problem, let's start with the formula used to determine memory usage of the state engine:

(memory used per tracked connectionxmaximum number of tracked connections)

(memory used per bucketsxnumber of buckets)=nonpagable memory needed by the connection engine

To figure out the solution to this formula, the place to look is /proc/slabinfo. This will show you the amount of memory allocated to elements of the connection tracking engine.

 cat  /proc/slabinfo slabinfo - version: 1.1 kmem_cache            93     93    124    3    3    1 : 252 126 ip_conntrack         322    570    384   47   57    1 : 124  62 ip_mrt_cache           0      0     96    0    0    1 : 252 126 tcp_tw_bucket        320    320     96    8    8    1 : 252 126 tcp_bind_bucket      226    226     32    2    2    1 : 252 126 tcp_open_request     169    295     64    4    5    1 : 252 126 inet_peer_cache      177    177     64    3    3    1 : 252 126 ip_fib_hash           56    226     32    2    2    1 : 252 126 ip_dst_cache         426    552    160   23   23    1 : 252 126 arp_cache            150    150    128    5    5    1 : 252 126 uhci_urb_priv          1     63     60    1    1    1 : 252 126 blkdev_requests     4096   4120     96  103  103    1 : 252 126 journal_head         352    858     48    8   11    1 : 252 126 revoke_table           2    253     12    1    1    1 : 252 126

For the state engine, the value we are interested in is ip_conntrack. For those readers not familiar with the layout of data in slabinfo, here is a quick tutorial:

              active-objects                |      allocated-objects                |      |     object-size                |      |     |   active-slab-allocations                |      |     |   |     total-slab-allocations                |      |     |   |     |     alloc-size                |      |     |   |     |     | ip_conntrack  322    570   384  47   57     1 :124      62                                                 |       |                                               limit     |                                                     batch-count

It's the object size that will start to tell us how much memory the state engine will be using to track connections. You also can use this shortcut to find this information:

 grep ip_conntrack /proc/slabinfo | tr -s " " | cut \ -d " " -f 4

The kernel will try to automatically set the maximum number of connections it can handle based on the amount of RAM in your system. You can see this by looking at your logs on bootup. If your machine has not been rebooted in a while, you might need to check messages.1, messages.2, and so on.

 cat /var/log/messages | grep ip_conntrack ip_conntrack version 2.1 (4095 buckets, 32760 max) - 360 bytes per conntrack

After you have this information, you can tell what the maximum recommended setting for your system should be and then compare that with what your system is set to support by looking once again in /proc:

 cat /proc/sys/net/ipv4/ip_conntrack_max 32760

On our system, the value returned was "32760," which is exactly the "safe" limit. In reality, the system can handle significantly more than 32K connections with 512MB of memory. For 32,760 connections, with a connection size of around 360 bytes on our test system, that's only 11,793,600 bytes, or around 11.8MB of memory used. But that's just for the memory used per connection and the maximum number of connections. It might be tempting to change the value in /proc/sys/net/ipv4/ip_conntrack_max, but don't do this without reading on. Changing this value alone will only increase the number of connections to track but not the amount of buckets allocated for those connections. This will actually cause a decrease in performance by making the collision chains longer.

According to published information about the engine, the ultimate ratio for performance is 2:1 for buckets to connections. So if you want to track 32,760 connections, you need to increase the amount of buckets as well.

If you want to increase the number of connections the engine is managing, you can modify it by increasing the hashsize allocated for the module ip_conntrack.

 modprobe ip_conntrack hashsize=16380

This will need to be run when the ip_conntrack module is first loaded. In 2.4 with a monolithic kernel (for example, a kernel without module support), the only way to change this value is by altering the kernel source. For 2.6, module parameters can be specified in a generic manner from the boot command line. Returning to the ip_conntrack_max variable, it adds the ip_conntrack_max value into /etc/sysctl.conf in this format so that the system is configured in this manner on its next boot.

 #max connections net.ipv4.ip_conntrack_max = 32760

In the short term you can also modify this value by echoing 32760 into

 /proc/sys/net/ipv4/ip_conntrack_max echo 32760 > /proc/sys/net/ipv4/ip_conntrack_max

We must also add in the memory used per bucket and the number of buckets. For a 32-bit system, each bucket uses eight bits of memory; for a 64-bit system, it's 16 bits. In our example, we have 16,380 buckets, which means the system is only using 131,040 bytes. It's not a whole lot in the grand scheme of things, but it can add up if you don't do your math. In total, the state tracking engine in this example is using just a little over 11.9MB of RAM.

In theory at least, this means that you could increase the number of connections the conntrack engine is managing pretty substantially, but this assumes that your firewall isn't using memory for much else. Also it's important to keep in mind the conntrack engine uses nonpagable memory, which also reduces the amount of memory available for other processes.

System memory is also used to handle things such as fragment reassembly, which might seem to be insignificant, but fragment reassembly on a network that is supporting clients with ever-changing MTU's can actually consume a substantially more significant amount of memory than the connection tracking engine. In earlier kernels, it was even possible to exhaust all of the system's memory by generating a flood of fragments and filling up the system's buffers while waiting for the rest of the fragments to arrive. This caused the kernel to start dropping connections, and in some cases could cause a DoS of the firewall's IP stack. The point here is that memory is quite important for a firewall, and because it's so cheap these days, there is great opportunity and need to increase the memory on a heavily used firewall. If your firewall is engaging in some extremely hard to diagnose problems, consider that it might simply be overworked and check the systems accordingly to determine if that is the case.