16.4 Tuning Qmail

More often than not, qmail doesn't need any tuning. It's designed to work well on typical Unix systems. For local deliveries, qmail is usually disk-bound, because it syncs files and directories to disk to avoid losing mail if the system crashes. Although it's possible on some systems to set filesystem parameters to subvert the syncs, that's usually a poor economy. If you want your local mail delivered faster, get a faster disk.^[2] If your system has a lot of unusually slow local delivery programs, or it runs really slow spam filters (Spamassassin can fall into that category), it's possible that local deliveries could be CPU-bound. The easiest way to find that out is with a utility like top that shows what's running. Much of the slowness in slow spam filters is due to DNSBL lookups, which are in fact network bound. Modern CPUs are so fast that it's a rare mail system that is even occasionally compute-bound.

^[2] If you haven't priced 15K RPM SCSI disks or 10K RPM ATA disks on eBay, you may be amazed how cheap they are. Be sure to get a drive cooler, too.

Remote deliveries are invariably network-bound. If the goal is to deliver mail as fast as possible, crank the concurrency up as high as possible. Looking at the zoverview results, it completed deliveries of 309400658 to 65158 recipients, for an average of a little under 5 Kbps per message. The average xdelay was 5.8 seconds, so each delivery was sending under 1 Kbps. This system happens to be on a T1 line, which can transmit 192 Kbps (that's 1.5 megabits divided by 8 bits per byte). So if each delivery sends 1 Kbps and the channel is 192 Kbps, it takes about 192 simultaneous deliveries to fill up the T1. Note that the ddelay, the time from when a message enters the queue to when a delivery finishes, is 35 seconds, while the average xdelay, the time from the beginning to end of a delivery, is only 5 seconds, which means messages wait 30 seconds to get a delivery slot. The mail traffic on this system is very bursty; a message comes in for a majordomo list and is queued for delivery to the 900 members of the list. The remote concurrency is 110, so the 110 slots immediately fill up and the other 790 deliveries have to wait for slots to be available as deliveries finish. Increasing the concurrency speeds overall deliveries. (I don't do this, because there are web and other servers on the network, and I don't want to squeeze them out every time there's a mailing list message.)

These numbers are fairly typical; if the channel ran at an Ethernet-like 10 megabits, the useful concurrency would be over 1000. Of course, most networks aren't entirely dedicated to email, but these sorts of estimates remain useful for setting up a system to use as much email bandwidth as the system manager wants to use.

16.4.1 Tuning Small Servers

Usually the only tuning needed on a small server is to adjust concurrencylocal and concurrencyremote. On very small systems with slow deliveries (Spamassassin run from procmail), it may be useful to decrease concurrencylocal to limit the hit on system performance from a lot of incoming mail, at the cost of slower deliveries. Set concurrencyremote using 1 K per second per delivery so that, for example, a DSL connection with 256 Kbits/sec of outbound bandwidth is 64 Kbytes/sec, so it would make sense to set concurrencyremote to 64 to use all of the bandwidth or to 32 to use up to half of it.

16.4.2 Tuning Large Servers

Large servers can be tuned and patched to increase the concurrency past what's normally possible. All of the necessary patches are at www.qmail.org in the section "Patches for high-volume servers."

For systems with a very large number of injected messages, the big-todo patch improves performance. In qmail's mail queue, most of the queues are divided into 23 subdirectories, with the files distributed pseudo-randomly into the 23 directories, but incoming mail goes into a single todo directory. If mail is injected at a high enough rate, the todo directory becomes inefficiently large and qmail-send falls behind. The big-todo patch by Russ Nelson and Charles Cazabon splits todo and the parallel intd directory into 23 subdirectories. The patch changes the format of todo but not the rest of the queue, so to install it without losing mail, you must ensure that nothing's queued in todo. After applying big-todo and rebuilding, use svc -td to stop qmail-smtpd and stop any local daemons likely to inject mail, then use svc -td to stop qmail-send, then make setup check to install the patched qmail, and use svc -u to restart qmail-send and qmail-smtpd.

An alternative, more complex, big-todo patch by André Oppermann is available at http://www.nrg4u.com/. (Look for the "silly qmail syndrome" patch.) It was written for use with qmail-ldap, described in Chapter 13, but it works equally well with regular qmail. It splits the qmail-send daemon into two separate processes, qmail-todo, which processes newly queued messages, and qmail-send, which schedules local and remote deliveries. This patch doesn't change the queue file format, so it can be installed merely by building the patched version, stopping qmail, make setup check to install the new version, and restart qmail. Normally I prefer simpler patches to more complex ones, but in this case, the Oppermann patch does a better job of dealing with a lot of incoming mail, so it's a better choice for systems busy enough that todo is an issue.

Normally qmail is built with a maximum delivery concurrency of 120 local deliveries and 120 remote deliveries. For large servers on fast networks, that's nowhere near enough remote deliveries. To raise the concurrency limit to 255, edit conf-spawn and change the 120 to 255, and recompile. This doesn't change the queue format, so to install it, stop qmail-send, install, and restart qmail-send. You don't need to stop qmail-smtpd.

If you need concurrency of more than 255, another patch found at www.qmail.org increases the maximum concurrency to 65,000. As distributed, the components of qmail pass delivery numbers to each other as single bytes. This patch changes them to pass the components as two-byte numbers. It doesn't change the queue format, so to install it, apply the patch, rebuild qmail, stop qmail-send, install the patched version, and restart qmail-send. The patch sets the concurrency limit to 1,000, which should be enough for most systems. As the patch file notes, if the concurrency limit is 1,000 and a message has 1,000 recipients in the same domain, qmail might try to open 1,000 simultaneous connections to the same server, which managers of some recipient systems might misinterpret as a denial-of-service attack. It's not likely in practice unless you happen to have a mailing list with all of the recipients in the same domain. (Recall that deliveries are made in random order, so a list with 1000 recipients in each of 10 domains does about 100 deliveries at time to each of the 10 domains.)

The final patch for large systems makes the queue bigger. If your system sends a lot of mail that takes a long time to deliver, you may need to enlarge the queue directories. Most Unix filesystems perform poorly with more than 1,000 files in a directory, so the default 23-way split will have trouble with more than 23,000 queued files. If you find yourself in this condition, the code change is easy; just edit conf-split to a larger prime number (at least as great as 1/1000 of the number of queued files you expect) and rebuild. But this changes the queue format, so the new version won't work with the existing queued files. Qmail.org tells how to make a smooth transition: before changing the split, stop qmail, move the existing set of qmail files from /var/qmail to /var/qmail2, edit conf-qmail to refer to /var/qmail2, and build and install a new temporary copy of qmail with make setup. Run /var/qmail2/rc to start up the temporary version, which will continue delivering mail out of the old queue. Now change conf-qmail back to /var/qmail, edit conf-split to increase the split, build and install it with make setup, and restart qmail. All future mail will be handled by the new copy of qmail. After a week or so all the mail in the old copy of qmail will be delivered, at which point you can kill off the qmail-send started from /var/qmail2/rc and delete /var/qmail2.