3.3 Pretuning Your Kernel

Squid requires a fair amount of kernel resources under moderate and high loads. In particular, you may need to configure your system with a higher-than-normal number of file descriptors and mbuf clusters. The file-descriptor limit can be especially annoying. You'd be better off to increase the limit before compiling Squid.

At this point, you might be tempted to get the precompiled binaries to avoid the hassle of building a new kernel. ^[1] Unfortunately, you need to make a new kernel, regardless. Squid and the kernel exchange information through data structures that must not exceed the set file-descriptor limits. Squid checks these limits at runtime and uses the safest (smallest) value. Thus, even if a precompiled binary has higher file descriptors than the kernel, the kernel value takes precedence.

^[1] Not all operating systems require building a new kernel. Some may be tunable at runtime.

To change some settings, you must build and install a new kernel. This procedure varies among different operating systems. Consult Unix System Administration Handbook ( Prentice Hall ) or your operating-system documentation if necessary. If you're using Linux, you probably don't need to recompile your kernel.

3.3.1 File Descriptors

File descriptors are simply integers that identify each file and socket that a process has opened. The first opened file is 0, the second is 1, and so on. Unix operating systems usually impose a limit on the number of file descriptors that each process can open . Furthermore, Unix also normally has a systemwide limit.

Because of the way Squid works, the file-descriptor limits may adversely affect performance. When Squid uses up all the available file descriptors, it is unable to accept new connections from users. In other words, running out of file descriptors causes denial of service. Squid can't accept new requests until some of the current requests complete, and the corresponding files and sockets are closed. Squid issues a warning when it detects a file-descriptor shortage.

You can save yourself some trouble by making sure the file descriptor limits are appropriate before running ./configure . In most cases, 1024 file descriptors will be sufficient. Very busy caches may require 4096 or more. When configuring file descriptor limits, I recommend setting the systemwide limit to twice the per-process limit.

You can usually discover your system's file-descriptor limit from your Unix shell. All C shells and similar have the built-in limit command. Newer Bourne shells and similar have a command called ulimit . To find your file-descriptor limits, try running these commands:

 csh% limit descriptors unlimited csh% limit descriptors descriptors    4096 

or:

 sh$ ulimit -n unlimited sh$ ulimit -n 4096 

On FreeBSD, you can also use the sysctl command:

 % sysctl -a  grep maxfiles kern.maxfiles: 8192 kern.maxfilesperproc: 4096 

If you can't figure out the file-descriptor limit, Squid's ./configure script can do it for you. When you run ./configure , as described in Section 3.4, watch for output like this near the end:

 checking Maximum number of file descriptors we can open... 4096 

If either limit , ulimit , or ./configure report a value less than 1024, you should invest the time to increase the limit before compiling Squid. Otherwise, Squid's performance will be poor under a moderate load.

Increasing the file descriptor limit varies from system to system. The following sections offer some tips to help get you started.

3.3.1.1 FreeBSD, NetBSD, OpenBSD

Edit your kernel configuration file, and add a line like this:

 options       MAXFILES=8192 

On OpenBSD, use option instead of options . Then, configure, compile, and install the new kernel. Reboot your system so the change takes effect.

3.3.1.2 Linux

Configuring file descriptors on Linux is a little complicated. You must edit one of the system include files, and execute some shell commands before compiling and running Squid. Start off by editing the file /usr/include/bits/types.h . Change the value for _ _FD_SETSIZE as follows :

 #define _ _FD_SETSIZE    8192 

Next, increase the kernel file descriptor limit with this command:

 # echo 8192 > /proc/sys/fs/file-max 

Finally, increase the process file-descriptor limit in the same shell in which you will configure and compile Squid:

 sh# ulimit -Hn 8192 

This command must be executed as root and only works from the bash shell. There is no need to reboot on Linux.

With this technique, you must execute the echo and ulimit commands each time your system boots, or at least before starting Squid. If you use an rc.d script to start Squid (see Section 5.6.2), that is a good place to stick these commands.

3.3.1.3 Solaris

Add this line to your /etc/system file:

 set rlim_fd_max = 4096 

Then, reboot the system for the change to take effect.

3.3.2 Mbuf Clusters

The BSD-based networking code uses a data structure known as an mbuf (see W.R.Stevens' book, TCP/IP Illustrated, Vol 2) . Mbufs are typically small (e.g., 128 octets) chunks of memory. The data for larger network packets are stored in mbuf clusters . The kernel may enforce an upper limit on the total number of mbuf clusters available in the system. You can find this limit with the netstat command:

 % netstat -m 196/6368/32768 mbufs in use (current/peak/max):         146 mbufs allocated to data         50 mbufs allocated to packet headers 103/6182/8192 mbuf clusters in use (current/peak/max) 13956 Kbytes allocated to network (56% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines 

In this example, there are 8,192 mbuf clusters available, but there are never more than 6,182 used at once. When the system runs out of mbuf clusters, I/O routines such as read( ) and write( ) return the "No buffer space available" error message.

NetBSD and OpenBSD don't display mbuf usage in netstat -m output. Instead, they report "WARNING: mclpool limit reached" via syslog.

To increase the number of mbuf clusters, you need to add an option to your kernel configuration file:

 options         NMBCLUSTERS=16384 

3.3.3 Ephemeral Port Range

Ephemeral ports are the local port numbers the TCP/IP stack assigns to outgoing connections. In other words, when Squid makes a connection to an origin server, the kernel assigns a port number to the local socket. These local port numbers fall within a certain range. On FreeBSD, for example, the default ephemeral port range is 1024-5000.

A shortage of ephemeral ports may adversely affect performance for very busy proxies (i.e., hundreds of requests per second). This is because some TCP connections enter a TIME_WAIT state when they are closed. An ephemeral port number can't be reused while the connection is in the TIME_WAIT state.

You can see how many connections are in this state with the netstat command:

 % netstat -n  grep TIME_WAIT Proto Recv-Q Send-Q  Local Address          Foreign Address        (state) tcp4       0      0  192.43.244.42.19583    212.67.202.80.80       TIME_WAIT tcp4       0      0  192.43.244.42.19597    202.158.66.190.80      TIME_WAIT tcp4       0      0  192.43.244.42.19600    207.99.19.230.80       TIME_WAIT tcp4       0      0  192.43.244.42.19601    216.131.72.121.80      TIME_WAIT tcp4       0      0  192.43.244.42.19602    209.61.183.115.80      TIME_WAIT tcp4       0      0  192.43.244.42.3128     128.109.131.47.25666   TIME_WAIT tcp4       0      0  192.43.244.42.3128     128.109.131.47.25795   TIME_WAIT tcp4       0      0  192.43.244.42.3128     128.182.72.190.1488    TIME_WAIT tcp4       0      0  192.43.244.42.3128     128.182.72.190.2194    TIME_WAIT 

Note that this example has both client- and server-side connections. Client-side connections have 3128 as the local port number; server-side connections have 80 as the remote (foreign) port number. The ephemeral port numbers appear under the Local Address heading. In this example, they are in the 19,000s.

Unless you see thousands of ephemeral ports in the TIME_WAIT state, you probably don't need to increase the range. On FreeBSD, you can increase the range with this command:

 # sysctl -w net.inet.ip.portrange.last=30000 

On OpenBSD, the command is almost the same, but the sysctl variable has a different name :

 # sysctl -w net.inet.ip.portlast=49151 

On NetBSD, things work a little differently. The default range is 49,152-65,535. To increase the range, change the lower limit:

 # sysctl -w net.inet.ip.anonportmin=10000 

On Linux, simply write a pair of numbers to the following special file:

 # echo "1024 40000" > /proc/sys/net/ipv4/ip_local_port_range 

Don't forget to add these commands to your system startup scripts so that they take effect each time your machine reboots.