14.1 Sending Mail to Lists

The easiest and most common way to handle a small list is to put the list in a .qmail file. To reiterate an example from Chapter 10, assume a user's name is fred, and the list is about fishing. Then the list goes into ~fred/.qmail-fishing, one address per line like any other .qmail file (see Example 14-1).

Example 14-1. Fred's fishing list

[e-mail] [e-mail] [e-mail] &/fn=hunt/ln=dash/@bigcorp.com

Note that the third address, an X.509 address that contains slashes, is preceded by an ampersand to keep it from being interpreted as a filename. Also, Fred's address is in the list so he gets copies of messages sent to it. To send mail to this list, one needs only to send a message to fred-fishing, and it's redistributed to all of the list members.

14.1.1 Maintaining List Files

Qmail provides a small but useful set of functions to maintain list files. To edit a file safely, set the otherwise unused "sticky" bit in the user's home directory, edit the file, then unset the sticky bit:

$ cd $ chmod +t . $ emacs .qmail-fishing $ chmod -r .

Should any mail arrive for addresses handled by a .qmail file in the directory while the sticky bit is set, qmail-local notices the sticky bit and exits with code 111 so the delivery is retried later.

This list file example highlights a possible security hole when an address looks like a filename.^[1] There are three ways to solve the problem. The simplest, but most error prone, is to put an ampersand in front of each name, or at least in front of each name that might look like a filename or command. The second is to set the owner execute bit on the file, which tells qmail-local that the file should only contain forwarding addresses, so any file or program deliveries fail. The third (undocumented) is to put a line containing +list in the file, which tells qmail-local that subsequent lines have to be forward addresses. This permits a few setup lines at the beginning before the addresses. For example, to require that each message's subject line has a keyword, see Example 14-2.

^[1] This isn't a new problem; in some ancient versions of Unix you could send mail to /etc/passwd and it'd add your message to the end of the password file.

Example 14-2. Fred's fishing list with subject checking

| egrep -qi "^Subject:.*(largemouth|smallmouth|squid)" || bouncesaying "Not fishy enough." +list [e-mail] [e-mail] [e-mail] &/fn=hunt/ln=dash/@bigcorp.com

In the examples so far, the list has an address that is a subaddress of a user address. List files can equally well live in ~alias in which case they have regular addresses; the list file ~alias/.qmail-fishing has the address fishing.

14.1.2 Bounce Handling and VERP

One of the most tedious and difficult parts of mailing list management is bounce handling, identifying and removing addresses that are no longer valid. The most difficult aspect of bounce management turns out to be identifying the address that's bouncing, and a secondary problem is getting the bounces sent to an address that can do something useful with them.

When a message bounces, the host doing the bounce, which may be the one where the message was injected or another one to which the message was relayed by SMTP, sends back a failure report to the message envelope sender address. On qmail systems, all the bounces from the injecting system are sent back in one message in QSBMF (qmail-send Bounce Message Format, described at http://cr.yp.to/proto/qsbmf.txt). Bounces from remote systems arrive one per bouncing address, because qmail sends remote mail to one address at a time. Remote bounces arrive in whatever format the remote system chooses to use. Qmail systems use QSBMF; some MTAs use DSNs (delivery status notices), a complex format originally described in RFCs 3461-3464; and a lot of systems use ad-hoc formats not standardized or documented anywhere. Also, the envelope address on outgoing list mail needs to be the address of the mailing list manager (human or software), not the address of the original sender only the list manager can update the list.

14.1.2.1 Manual bounce handling

The way to set up a qmail list for manual bounce handling is simply to create an owner mailbox. That is, if the list's qmail file is .qmail-fishing, create .qmail-fishing-owner and set it up to deliver mail someplace that the owner will see it, usually either forwarding to the owner's regular address or putting the mail in an mbox or Maildir.

When qmail-local processes .qmail-fishing, it checks to see if .qmail-fishing-owner exists, and if so, changes the envelope sender to [e-mail], or more generally to LOCAL-owner@ HOST where LOCAL and HOST are the local and host part of the original address. When bounces arrive, it's up to the list owner to read them and update the list appropriately by removing addresses that consistently bounce.

14.1.2.2 Automated bounce handling

For a list of any size or with a significant amount of traffic, manual bounce handling is an impossible amount of work. Fortunately, software does as good a job of bounce handling as people can, particularly when it uses qmail's VERP to identify the bouncing addresses.

Variable Envelope Return Path (VERP) encodes the recipient's address in the envelope sender of each message sent out, so if a message bounces, the address that bounced can be recovered from the address the bounce message is sent to. The recipient's address is placed at the end of the mailbox part with the @ sign changed to an = sign. For example, VERP would arrange that mail from Fred's fishing list to recipient [e-mail] has return address fred-fishing-owner-mary=[e-mail]. If her mail bounces, the bad address is recovered from the bounce address by picking out the text at the end of the local part and changing the = back to an @ sign.

To use automated bounce handling, along with a -owner file, create a -owner-default file, which delivers to the bounce handling program. If qmail-local sees both of those files,^[2] it rewrites the sender to LOCAL-owner-@HOST-@[ ]. This peculiar sender address turns on VERP, by telling qmail-send to rewrite the address again on each remote delivery to RUSER@RHOST so the envelope sender is LOCAL-owner-RUSER=RHOST@HOST. The overall effect of this is that all bounce mail is delivered to the -owner-default address, with local bounces delivered to LOCAL -owner- (note that trailing hyphen).

^[2] Both files have to be present, even though nothing will be delivered to the plain -owner address. This is debatably a buglet, although the owner address should exist anyway for humans to write to.

The bounce script can now easily determine the bouncing addresses, by parsing the QSBMF message in local bounces and picking the return address out of the address in remote addresses. The code to do so isn't very complex. It's wordy in C because of all of the string processing, so Example 14-3 shows it in Perl.

Example 14-3. Sample Perl code to handle bounces

$addr = $ENV{DEFAULT} # set by qmail-local if ($addr) {     $addr =~ s/=/\@/;         # VERP bounce, pick up address     while(<>) {         # ignore bounces that aren't really bounces         exit 99 if /THIS IS A WARNING MESSAGE ONLY/;         exit 99 if /^Subject: WARNING: message delayed at/;         exit 99 if /^Subject: Returned mail: Deferred/;     }     dobounce($addr); } else {     # locally generated bounce, must be QSBMF     $/=""; # slurp up a paragraph at a time     $_=<>; # get rid of the email header.     $_=<>; # get the QSBMF     /^Hi. This is the/ || die "This is not a qmail bounce message";     while(<>) { # handle each address section          last if /^-/;         /^<(.*)>/ || die "No recipient address";         dobounce($1);     } }

Once the bounce code has the address, it should remove addresses from the list that bounce too often, for an appropriate definition of "too often" that has to depend on the nature of the traffic to the list.

This scheme won't handle 100% of all bounce mail, because some MTAs act in hostile ways, sending bounces other than to the envelope sender, but this gets about 90% of the effect of more comprehensive bounce handlers with about 5% of the work.

14.1.2.3 Bounce handling for mail without .qmail file forwarding

Mailing list software that doesn't keep the list in a .qmail file can also take advantage of qmail's automated bounce handling by setting the return address appropriately. No matter how mail is injected into qmail, whether it's via qmail-inject, by SMTP, or by calling qmail-queue directly, any envelope return address that ends with -@[ ] receives automatic VERP handling. (It's also possible for list software to generate 100 messages with 100 return addresses for 100 recipients, but that's pointless unless the messages differ in more than the envelope address.) At the moment the only mailing list software that takes advantage of qmail's automated VERP are ezmlm^[3] and majordomo2,^[4] but it wouldn't be hard to add it to other list management software.

^[3] Because it was written to work with qmail

^[4] Because I wrote the qmail VERP code myself.