6.2 Cleaning Up Injected Mail

Unlike some other MTAs, qmail distinguishes between injected mail, new messages entered into the mail system, and relayed mail, which is delivered from somewhere else. The difference is that injected mail needs to have its headers cleaned up, while relayed mail doesn't. Configuring qmail to clean up injected mail isn't hard, but depending on your setup, there are several possible ways to handle it.

The new-inject package contains two programs: new-inject, which is a replacement for qmail-inject, and ofmipd (Old Fashioned Mail Injection Protocol Daemon), an SMTP daemon that includes the functions of new-inject. Although you can survive without new-inject, it's easy to install and I encourage you to use it.

6.2.1 Accepting and Cleaning Up Locally Injected Mail

The usual ways to inject local mail are to feed it to qmail-inject or sendmail. Both do the cleanup automatically. (The qmail version of sendmail is a small wrapper that runs qmail-inject.)

Because new-inject is almost completely upward compatible with qmail-inject, use it in place of qmail-inject:

# cd /var/qmail/bin # mv qmail-inject qmail-inject.old # ln new-inject qmail-inject

(I've saved the old qmail-inject as qmail-inject.old in case there turned out to be some application that needed exactly qmail-inject's features, but after a year, I have yet to need it.)

Some programs inject local mail by opening an SMTP connection to the loopback address 127.0.0.1. If you've installed an SMTP listener following the instructions in Chapter 4, injecting mail via that route already works, but without any cleanup. There are two alternatives to clean up mail injected by SMTP: adjusting the setup of the regular SMTP server to detour locally injected mail through a cleanup program or setting up a separate SMTP daemon running ofmipd to receive locally originating mail. I discuss both options later in the chapter.

The standard way to give a freshly created message to qmail for delivery is to use qmail-inject or its replacement new-inject. Both programs accept a message from the standard input, clean up and complete the headers without modifying the message body, construct the envelope information from the message and command arguments, and pass the result to qmail-queue for delivery. A combination of flags on the command line and environment variables give you some control over the header rewriting and control where it gets the envelope addresses. In the following discussion capitalized names refer to environment variables passed to qmail-inject or new-inject.

The QMAILINJECT environment variable, if it exists, contains a string of letters from the set cfimrs that control the header rewriting, as described later. new-inject also accepts the uppercase letters FIMRS with the equivalent meanings and also accepts command-line --FIMRS flags.

For testing purposes, the --n flag causes the rewritten message to be copied to standard output rather than queued. To show the envelope addresses, new-inject prefixes Envelope-Sender: and Envelope-Recipients: headers, while qmail-inject puts the sender address in a Return-Path: line but doesn't do anything with the recipient addresses.

6.2.1.1 Setting the envelope addresses

Like sendmail, qmail-inject and new-inject can take the recipient addresses from the command line, from the message itself, or both. In the absence of any flags or with the --A flag, they deliver the message to the addresses on the command line if there are any, otherwise to the addresses in the To:, Cc:, Bcc:, and Apparently-To: (a sendmail-ism). new-inject uses Envelope-Recipients: line(s), if any exist, in preference to those headers. The --a flag says to use only command-line addresses, --h says to use only the header recipients, and --H says to use both. All addresses are rewritten as described in Section 6.2.1.3.

The envelope sender address is taken from the --f flag if present. Otherwise, unless the environment flag "s" is set, it uses Envelope-Sender: (new-inject only) or Return-Path:. If those headers aren't present, the user part of the sender is taken from QMAILSUSER, QMAILUSER, MAILUSER, USER, or LOGNAME. The host part is taken from QMAILSHOST, QMAILHOST, or MAILHOST, or if none of those are set, the defaulthost control file.

The environment flags "m" and "r" handle Variable Envelope Return Paths (VERP), a way to encode information about the message and its sender in the envelope return address to aid bounce processing. (VERP is discussed at length in Chapter 14.) If environment flag "m" is set, it appends a dash, the time in seconds, a dot, and the process ID as a per-message VERP. If environment flag "r" is set, it rewrites the sender address for the per-recipient VERP (described in Chapter 14). Either or both kinds of VERP can be present; for example, if the sender [e-mail] might be turned into [e-mail]-@[ ]. Note that neither kind of VERP is done if the sender is set explicitly with --f; in that case it's up to you to put whatever you want into the sender string.

6.2.1.2 Header rewriting

Both qmail-inject and new-inject rewrite most message headers:

From:

If environment flag "f" is set, any existing From: header is discarded; if not, an existing From: header is passed through. When creating a From: header, the user part is taken from QMAILUSER, MAILUSER, USER, or LOGNAME, and the host part is taken from QMAILHOST, MAILHOST, or the defaulthost control file. (Note that QMAILSUSER and QMAILSHOST don't affect the From: line, providing the occasionally useful ability to concoct different header and envelope return addresses.) The comment on the From: line is taken from QMAILNAME, MAILNAME, or NAME. If the environment flag "c" is set, qmail-inject uses the address (comment) style; otherwise it uses comment <address>. new-inject ignores the "c" flag and always uses comment <address>.

To: and Cc:

Addresses are rewritten as described in the next section, and put into standard format with commas between the addresses and address comments in the preferred form. new-inject combines multiple To: lines or multiple Cc: into one. If there are no To: or Cc: addresses at all, it adds a syntactically valid Cc: group address of "recipient list not shown: ;".

Bcc: and Apparently-To:

These lines are deleted.

Notice-Requested-Upon-Delivery-To:, and Mail-Reply-To: and Reply-To:

The addresses are rewritten but otherwise don't affect delivery.

Date:

If there's an existing date header with a date that it can decode, the date is standardized into the form 23 Jun 2004 12:02:00 -0500. If not, it adds a Date: header with the current time and date.

Message-ID:

If there's an existing header, it's passed through, unless the environment variable "i" is set, in which case any existing Message-ID: header is deleted. In the absence of a passed-through header, it creates a new one. The domain part of the new Message-ID: comes from QMAILIDHOST if present, otherwise the idhost or me control files. The user part is always a combination of the date and PID in the form 20020623170200.2345.qmail.

Resent- headers

new-inject moves these to the top of the message with the Received: headers but doesn't otherwise rewrite them. If qmail-inject sees any of them, it adds Resent-Date: and Resent-Message-ID:, and treats the Resent-To: as the header addresses to which the message is delivered. This is the most significant incompatibility between qmail-inject and new-inject, although it rarely causes trouble in practice because MUAs tend to put either a full set of Resent- headers on messages or none of them.

Mail-Followup-To:

If this header is not already present, and the environment variable QMAILMFTFILE is the name of a file that contains a list of mailing list posting addresses, one per line, and one of those addresses appears in the To: or Cc: line of the message, it adds a Mail-Followup-To: header containing all of the To: and Cc: addresses. This is a little-used feature intended for some varieties of mailing list software.

Content-Length:

If this field is present, it's removed. Some MUAs attempt to use it to make it faster to scan mboxes, but it's not useful in mail in transit.

6.2.1.3 Address rewriting

Addresses in the message headers are rewritten into a standard form. (Envelope addresses aren't rewritten, other than with the VERP options discussed earlier. If the envelope recipients are taken from the headers, it uses the rewritten versions.)

The rewriting involves adding or completing the domain. Qmail's rewriting rules work best in an environment with multiple subdomains, such as a university where each department has its own subdomain (so [e-mail] and [e-mail] are different addresses, and on-campus users would likely abbreviate them as fred@alchemy and fred@phrenology).

If an address has no host part at all, it adds a default hostname from QMAILDEFAULTHOST, or the contents of defaulthost or me.

If the host part (whether it came from the previous step or was already present) contains no dot, it adds a dot and QMAILDEFAULTDOMAIN, or the contents of defaultdomain or me.

If the host part ends with a plus sign, it changes the plus to a dot and adds QMAILPLUSDOMAIN, or the contents of plusdomain or me.

In the usual case of "flat" addressing where all the addresses are in the second or third-level domain, both defaulthost and defaultdomain should contain that domain. In the aforementioned campus example, defaulthost should contain the name of the local subdomain (such as alchemy.bigu.edu), and defaultdomain should contain the main domain (such as bigu.edu) so that short addresses like fred@alchemy and fred@phrenology work. Plus addresses are for the more esoteric situation where there are multicomponent subaddresses, so a user can type [e-mail]+ and have that turn into [e-mail].

new-inject has a more elaborate rewriting system controlled by patterns from rewrite (or if it exists, the file named by QMAILREWRITE.) See the rewriting(5) manpage for details. I don't recommend doing more elaborate rewriting, because that makes the addresses your users type into their MTAs different from the ones that the rest of the world uses, causing great confusion when they tell their friends to write to jerry@boam and it doesn't work because that address is locally rewritten into [e-mail]. However, rewriting is useful to compensate for users who insist on writing to stevec@aol when they mean [e-mail].

6.2.2 Passing in Large Numbers of Addresses

The simplest way to send mail to a list of addresses is to put all the addresses in a file, and then either directly or, more likely, via a mailing list manager (such as ezmlm, majordomo, or mailman) type:

new-inject -a $(cat mylist) << EOF To: mylist Subject: Free beer Look behind the coffee machine at 5 PM EOF

This works as long as the list of addresses remains small enough to fit on a command line. The To: address in the message, which is normally the address of the list, is ignored for delivery purposes. But what happens when the list doesn't fit on a command line? A typical command-line limit is 20 K, which only fits a thousand 20-character addresses not a very big list.

The usual way to get around the command-line limit is to queue the message directly, either by running qmail-queue or by connecting to the local port 25 SMTP daemon to send the mail. They both work, but they have the disadvantage of doing no header cleanup. Can we run new-inject and give it thousands of addresses? Yes.

The obvious approach is to put all the addresses on Bcc: lines, because they're normally copied into the envelope and deleted. But the problem is that addresses on the To: and Cc: lines are copied into the envelope as well. The To: address is generally the address of the list itself, so this is a fairly efficient route to a mail loop.^[1] Instead, put the addresses into Envelope-Recipients: headers at the top of the message, either one huge line with addresses separated by commas (like all parts of qmail, it allocates line buffers to be as big as they need to be so there's no limit on line length) or in multiple header lines. Either way, all of the recipients will be extracted from those headers, and then the rest of the message will be cleaned up and sent on its way to all the recipients.

^[1] A mail loop, for the fortunate few who have never encountered one, is a chain of forwarding addresses that forms a loop so that mail keeps recirculating forever, frequently growing at each stage as forwarders add headers or comments to the message. Qmail breaks mail loops by scanning the Delivered-To: headers at delivery time and bouncing any mail that has a Delivered-To: that's the same as the address it's delivering to now, but avoiding loops is far preferable to breaking them.