14.4 Sending Bulk Mail That s Not All the Same
14.4 Sending Bulk Mail That's Not All the SameQmail does a magnificent job of sending identical copies of a single message to thousands of recipients.[6] It does a considerably less magnificent job of sending thousands of messages all of which are different each to a single recipient. The overhead of passing a message to qmail-queue, storing it in the queue/todo directory, then moving it into the delivery queue, is substantial. When large numbers of messages arrive quickly, qmail-send can fall behind to the point where it's so busy dealing with injected mail it doesn't schedule deliveries as fast as it should, the so-called "silly qmail syndrome." The big-todo patch discussed in Chapter 16 helps somewhat, but the fastest way to deliver lots of unique messages is to avoid asking qmail to deal with them in the first place. To test this theory out, I wrote a small Perl module Qspam, available as http://www.iecc.com/Qspam.pm, to send lots of unique messages fast.
The program sending the mail starts by calling qspam_start(N, &donefunc), where N is the number of deliveries to handle at once (analogous to concurrencyremote) followed by a callback routine that's called each time a delivery attempt finishes. To send a message, the program calls qspam_send("to", "from", mfile, code), where to and from are the envelope addresses, mfile is the name of a file containing the entire message to send, headers and body, and code is an optional code string that identifies the message. When the delivery is done, it calls the callback as donefunc(mfile, code, resultflag, resultmsg) where mfile and code are from qspam_send, and resultflag is "y" if the message was delivered, "n" if the delivery failed (in which case resultmsg is the error message), or a null string if the delivery was deferred until later. At the end of the program, qspam_flush( ) waits for all of the delivery attempts to complete. How does this all work? Qspam_send forks and calls qmail-remote to deliver the message. The module keeps a table of all of the deliveries in progress and won't start more than the delivery limit at once. When an instance of qmail-remote completes, if it either delivered the mail or got a permanent error, the delivery is done. If there was a temporary error, Qspam forks again to call qmail-queue to use the standard qmail delivery scheme, which always succeeds (from Qspam's point of view). Because qmail-remote can't deliver local mail, qspam_send checks the delivery address of each message against locals and virtualdomains to see if an address is local, and if so calls qmail-queue immediately. In practice, most remote delivery attempts succeed or fail on the first try, so only a small fraction of the messages need to be queued. Some mail is accepted by the remote MTA only to be bounced back later, and qmail returns its usual bounce messages if a queued delivery eventually fails, so the application needs to use envelope return addresses that can be handled by a companion bounce processor, just like list mail sent directly through qmail. Although Qspam wasn't written for maximum efficiency (it opens and closes temporary files rather than using pipes), it's pretty fast. On a modestly sized PC sending lightly customized mail to a list of several thousand users selected from a MySQL database, it has no trouble keeping 100 simultaneous deliveries going at once. The entire application is written in Perl, but it spends nearly all of its time waiting for qmail-remote processes to finish so there's little reason to rewrite it to be faster. This approach, try one delivery attempt before queueing, has proven to be a simple but effective way to handle customized list mail. |
