5.3 Draining Queues

At the outset, I should point out that this third strategy for the mass mailing of files is the one I recommend least. It would be very easy to get a little sloppy with this method and cause a considerable amount of heartache, either in terms of work that needs to be redone or in the creation of an email mess for the intended recipients of the messages. The reader should consider this method to be included primarily for completeness.

Another option to consider for mass mailing is to precreate sendmail qf and df files in directories (queues) and, when the appropriate time to send comes, run sendmail -q -oQ/path/to/directory to drain the queues. This strategy requires a solid knowledge of the format of the qf files, which isn't too difficult to determine. Much of the information is available in Costales' sendmail book [CA97]. More up-to-date but terse information can be found in a comment inside the sendmail/queue.c file. Good prototypes for these files can be generated using the following method:

1. Start sendmail on a test machine in queue-only mode: sendmail -odq -bd.

2. Send several pieces of email to the test machine with the destination of addresses that belong to the sender. We don't want to spam anyone in case we make a mistake.

3. Shut down sendmail, and then move the messages out of /var/spool/mqueue for inspection and use as templates.

Here's an example qf file that we might use as a prototype. It was sent from a machine using sendmail version 8.12.0.

 V6  T1001371039  K0  N0  P120381  Fbs  $_root@localhost  ${daemon_flags}cu  Snpc  [e-mail]  C:[e-mail]  rRFC822; [e-mail]  RPFD:[e-mail]  H?P?Return-Path: <<\081>g>  H?x?Full-Name: Nick Christenson  H??Received: (from npc@localhost)         by discovery.gangofone.com (8.12.0/8.12.0/Submit) id         f8OMbJVu087571 for [e-mail]; Mon, 24 Sep 2001 15:37:19         -0700 (PDT)  H??Date: Mon, 24 Sep 2001 15:37:19 -0700 (PDT)  H??From: Nick Christenson <[e-mail]>  H??Message-Id:<[e-mail]>  H??To: [e-mail]  H??Subject: test  . 

Let's take a closer look at some of these lines and what they represent.

The first line begins with "V", which denotes the version number for the format of the file. The version number for sendmail 8.12 files is 6. This file cannot be parsed by sendmail 8.11 or earlier releases.

The second two fields, "T" and "K", denote the time that the message was created and the time that it was last processed, respectively. Because the qf file in this example was never processed, the file includes K0. These times are written in the UNIX time_t format, so they represent the number of seconds since the beginning of 1970. This number is used when determining whether a message is too old for delivery. If a sendmail process fails to deliver this message, and the current time minus the T value is greater than the Timeout.queuereturn value in the /etc/mail/sendmail.cf file, the message will be bounced. If the current time minus the T value is greater than the Timeout.queuewarn value, then a warning message will be sent. For mass mailings, such a warning message is probably undesirable. The sender can easily determine which messages haven't been sent yet by inspecting the queues, so Timeout.queuewarn should be set to 0 and Timeout.queuereturn should be set either very high or to 0. In pregenerated qf files, T should be very close to, but not more than, the actual time at which the message will be sent. The K value should be set to either T or 0.

The "N" value indicates the number of times delivery has been attempted for this message. It should be set to 0 as in this case, or the line should be omitted.

The "P" value is the message's priority. Priority isn't quite as useful a parameter as its name might imply. Under most circumstances, it's a good idea to set the priorities of all messages in a single queue to the same integer, such as 1, or to omit the field from the file.

The "S" line is required; it contains the sender's email address in RFC 2821 format. In this case, because the message was set from the command line, it has not yet been canonified. Once canonified, it will be used as the argument to the MAIL command in the SMTP envelope of the message transmission.

The "r" line contains the original recipient. This information provides the address to which DSNs would be sent. The "R" line contains the recipient address in RFC 2821 format. One of these lines should appear for each recipient of the message. The second letter "P" in this field stands for "primary," which is usually the appropriate choice. The "D" indicates that this message requests delay DSNs, if appropriate. An "F" indicates that this message requests failure DSNs.

After the "R" line come the message headers, in the formats specified in the sendmail.cf file. Take special care when altering these lines, as changes to them can have side effects. More information on headers can be found in Appendix B of the Sendmail Installation and Operation Guide.

If pregenerated qf files will be used, one may as well sort the recipients to optimize delivery. If multiple recipients of a message reside in a single domain, put as many of these recipients as is practical in a single message. This clustering will save resources on both the sending and receiving server, as the message will need to be transmitted only once for all of these recipients. To discourage spammers, some sites will set the sendmail MaxRecipientsPerMessage parameter in their configuration files. After this threshold is exceeded, the receiving server will issue temporary failure codes for succeeding recipients of the message. These messages will have to be re-sent. This threshold will differ for each site, but on servers that have it set, it typically does not exceed 100, but rarely is set to less than 10. Therefore, it should be safe for the time being to construct messages to be sent to 10 simultaneous recipients. With these sorts of numbers, if some recipients are initially rejected, the duplicate effort will be minimized. If information on the policies of each domain can be compiled, it would make sense to use it when pregenerating messages in the future.

Pregenerating qf files offers the advantage of allowing one to finely control numbers of recipients per message, queue sizes, and numbers of queues, although this level of granularity is rarely necessary. Going through these gymnastics makes sense only when one has a lot of email to send, interactive delivery mode with sendmail 8.12 isn't realistic, a considerable amount of preparation time is available before the messages need to go out, and the messages must be sent out very quickly when that moment arrives. In fact, pregenerating qf files is a dangerous business in general. Before going live with such a scheme, messages should be carefully tested to make sure their results are sane. It is easy to get wrong, so it's not recommended that email be sent this way at all.