Managing the Queue

---

If qmail-send is qmail's brain, the queue is qmail's heart. Every message that qmail delivers passes through the queue, at least momentarily. It's the queue that allows qmail to guarantee that it won't lose messages, even if the system crashes. It's the queue that allows qmail to retry deliveries that temporarily fail.

To quickly assess the health of a qmail system, take its pulse by checking the state of the queue. qmail automatically maintains its queue, and the mail administrator is rarely called on to perform heart surgery by manually modifying the queue.

For security and privacy reasons, the queue, which is stored in /var/qmail/queue, is accessible only by the qmail users and the superuser. Queue management should be performed as the superuser.

Checking the Queue

The two main queue diagnostic tools are qmail-qstat and qmail-read. qmail-qstat takes the queue's pulse: It shows the total number of messages in the queue and the number of messages in the queue that haven't been preprocessed by qmail-send.

qmail-stat

On a lightly used, properly functioning system, qmail-qstat will often report an empty queue:

 # /var/qmail/bin/qmail-qstat messages in queue: 0 messages in queue but not yet preprocessed: 0 # 

All that means is that qmail has delivered every message it has received. On a busier, properly functioning system, you might see something like this:

 # /var/qmail/bin/qmail-qstat messages in queue: 87 messages in queue but not yet preprocessed: 0 # 

which shows there are 87 messages in the queue, all of which have been pre-processed by qmail-send. These are messages that have not yet been delivered to all of their recipients. The fact that none require preprocessing shows that qmail-send is running and doing its job.

The same system could easily show a small number of messages that aren't preprocessed if qmail-qstat happens to be run immediately after one or messages were injected:

 # /var/qmail/bin/qmail-qstat messages in queue: 88 messages in queue but not yet preprocessed: 1 # 

However, if the number of unpreprocessed messages remains constant or even grows, this indicates that qmail-send is either not running at all or is not working correctly. For example, on a system where qmail was not started properly, you'd see something like this:

 # /var/qmail/bin/qmail-qstat messages in queue: 5 messages in queue but not yet preprocessed: 5 # 

Another situation that can be spotted with qmail-qstat is that of a server that's being overwhelmed or, in other words, asked to deliver mail faster than qmail-send can handle it. You'll usually only see this on busy servers. The symptoms of this problem are many messages in the queue and significant numbers of unpreprocessed messages.

For example:

 # /var/qmail/bin/qmail-qstat messages in queue: 8254 messages in queue but not yet preprocessed: 73 # 

If this condition persists, see Chapter 7 for tips on configuring your server for maximum performance.

The qmailctl stat function is a handy way to run qmail-qstat because it also reports on the status of the qmail services. For example:

 # qmailctl stat /service/qmail-send: up (pid 857) 13356 seconds /service/qmail-send/log: up (pid 862) 13356 seconds /service/qmail-smtpd: up (pid 859) 13356 seconds /service/qmail-smtpd/log: up (pid 861) 13356 seconds messages in queue: 0 messages in queue but not yet preprocessed: 0 # 

The last two lines are qmail-qstat's output. If qmail-send wasn't running, for example, the first line of output would say down.

If qmail-qstat indicates any problems, the first step is to verify that qmail-send is running. If it's not, you'll need to figure out why (probably an error in the startup scripts) and restart it. If qmail-send is running, you should check its logs (see Chapter 6, "Troubleshooting qmail").

qmail-qread

If qmail-qstat takes the queue's pulse, qmail-qread takes an electrocardiogram (ECG). qmail-qread looks at every message in the queue and shows the envelope sender, time of creation, queue ID, and the status of delivery to each recipient. If the queue is empty, so is its output. But on a busy mail server—especially a list server—it can easily generate hundreds of thousands of output lines.

For example, on a mail list server, qmail-qread outputs something like this:

 # /var/qmail/bin/qmail-qread 21 Jun 2001 14:49:40 GMT   #119439   4190   <[e-mail]-@[]> bouncing done      remote        [e-mail] done      remote        [e-mail] done      remote        [e-mail]           remote        [e-mail]           remote        [e-mail] done      remote        [e-mail] # 

This shows that the queue contains one message, queue ID 119439, which is a mailing list mailing from the xlist list that was sent on June 21, 2001, to six remote recipients. Delivery to four of the recipients is complete, as indicated by the done flag.

In most cases, you can significantly prune qmail-qread's output by stripping the completed deliveries. Repeating the previous example with this method yields this output:

 # /var/qmail/bin/qmail-qread | grep -v "   done" 21 Jun 2001  14:49:40 GMT  #119439  4190  <[e-mail]-@[]> bouncing          remote   [e-mail]          remote   [e-mail] # 

make check

The check target of the Makefile in the qmail source directory checks the consistency of a qmail installation, including the queue structure. It doesn't check all of the files in the queue, but it will find some major problems.

For example:

 # cd /usr/local/src/qmail-1.03 # make check ./instcheck instcheck: warning: /var/qmail/queue/local/22 does not exist # 

Any problems reported by make check indicate that the queue is corrupt (damaged or inconsistent). You should shut qmail down until you have corrected the problem.

qmail-qsanity

Russell Nelson has written a Perl script called qmail-qsanity that checks the consistency of the queue (http://www.qmail.org/qmail-qsanity-0.52). If you're seeing unusual errors from qmail-send, you might want to run this script to identify or rule out queue corruption.

For example:

 # qmail-qsanity local/ has too few subdirectories at /usr/local/sbin/qmail-qsanity line 87. cannot read local/22 at /usr/local/sbin/qmail-qsanity line 43. # 

Modifying the Queue

qmail automatically manages the queue, but occasional situations may require human intervention. For example, one of your users sends a message with a 50-megabyte attachment to a few hundred of his closest friends. Or maybe junk mailers pollute your queue with mass mailings. Or even worse, a junior system administrator tries to manipulate the queue and ends up corrupting it.

A mail administrator has basically four kinds of queue modifications to perform: adjusting message lifetimes/retry schedules, removing messages from the queue, making a corrupt queue consistent, and re-creating an empty queue.

Adjusting Message Lifetimes/Retry Schedules

Normally, undeliverable messages remain in the queue for queuelifetime seconds before being returned as permanently undeliverable. The longer a message has been in the queue, the less frequently qmail tries to deliver it. (The actual retry schedule is documented in Appendix A, "How qmail Works.")

qmail tracks a message's age using the creation time of the info queue file, the file under /var/qmail/queue/info/split/queueid, that stores the envelope sender of each message. A message's age is the current time minus the creation time of the info file. Using the touch command to adjust this creation time, the mail administrator can prematurely age a message, causing it to be retried less frequently and bounce sooner, or give it a sip from the fountain of youth, causing it to be retried more frequently and delaying it from bouncing.

With the default queuelifetime of one week, setting a message's age to at least a week will mark it for one final delivery attempt. For example, to age the message with queue ID 119439, which resides in the 0 split directory, do this:

 # cd /var/qmail/queue/info/0 # ls -l 119439 -rw-r--r--   1 qmailq   qmail      4190 Jun 21 10:49 119439 # touch -d "1 week ago" 119439 # ls -l 119439 -rw-r--r--   1 qmailq   qmail      4190 Jun 17 07:23 119439 # 

Likewise, to grant the message with queue ID 119364 a temporary reprieve, the administrator could lower its age by doing this:

 # cd /var/qmail/queue/info/17 # ls -l 119364 -rw-r--r--   1 qmailq   qmail      1936 Jun 22 08:12 119364 # touch 119364 # ls -l 119439 -rw-r--r--   1 qmailq   qmail      1936 Jun 23 09:20 119364 # 

Adjusting message ages is safe, even while qmail-send is running.

Removing Selected Messages from the Queue

OK, so there are messages in the queue that have to be removed before qmail delivers them. The first thing you should do is stop qmail-send using either the qmailctl stop command or svc –d, like this:

 # qmailctl stop Stopping qmail. . .   qmail-smtpd   qmail-send # 

or this:

 # svc -d /service/qmail-send/ # 

Warning 

qmail-send must be stopped before attempting to remove messages from the queue. Otherwise, qmail-send's internal knowledge of the queue won't match the actual queue on disk. This will result in qmail-send logging errors about the messages that were deleted behind its back.

Once qmail-send has been stopped, the queue files associated with the messages you want to remove can be deleted. But before you can do that, you'll need to find the queue IDs of the messages you want to remove using the logs, qmail-qread, or by greping the mess queue files. For example, to search the queue for all messages containing a string, for example, "warez," use a find command:

 # find /var/qmail/queue/mess -exec grep warez {} /dev/null \; /var/qmail/queue/mess/18/119457:Subject: Best warez site ever! # 

Tip 

Include /dev/null in the grep command to force grep to show the names of the files containing matches. If grep only searches one file, such as the argument supplied by find in place of the {}, it will output matching lines without prefixing the filename. By including /dev/null, this tricks grep into thinking it's searching multiple files, so it includes the filename in its output.

Once you've identified the queue IDs of the target messages, use find to locate and remove the files:

 # find /var/qmail/queue -name 119457 /var/qmail/queue/bounce/119457 /var/qmail/queue/mess/18/119457 /var/qmail/queue/info/18/119457 /var/qmail/queue/remote/18/119457 # find /var/qmail/queue -name 119457 -exec rm {} \; # 

Repeat the find command as necessary until all of the files associated with the target message are removed.

Finally, restart qmail using the qmailctl script:

 # qmailctl start Starting qmail # 

or using svc, do this:

 # svc -u /service/qmail-send/ # 

Once qmail is restarted, check the end of the qmail-send log file for any messages about problems with the queue:

 # tail /var/log/qmail/current 

Removing All Messages from the Queue

If it's necessary to delete all messages in the queue, you have two approaches: delete only the message files, as in the previous section, or delete/rename the entire queue and build a new one.

Whichever approach you use, stop qmail-send first. Using qmailctl, do this:

 # qmailctl stop Stopping qmail. . .    qmail-smtpd    qmail-send # 

or using svc, do this:

 # svc -d /service/qmail-send/ # 

To delete all of the message files, do this:

 # find /var/qmail/queue -type f | grep -v /lock/ | xargs rm -f # 

To rebuild the queue, do this:

 # mv /var/qmail/queue /var/qmail/queue.old # cd /usr/local/src/qmail-1.03/ # make setup check ./install ./instcheck # 

Finally, restart qmail. Using the qmailctl script, do this:

 # qmailctl start Starting qmail # 

or using svc, do this:

 # svc -u /service/qmail-send/ # 

Once qmail is restarted, check the end of the qmail-send log file for any messages about problems with the queue:

 # tail /var/log/qmail/current 

Both of these methods will remove all messages from the queue immediately and without generating bounces. Using the rebuild method, the old queue can be preserved for extracting and re-injecting important messages.

Making a Corrupt Queue Consistent

If qmail-send generates error messages about missing queue files or the inability to read or write queue files, the problem is likely to be queue corruption, which is usually caused by system or mail administrators directly manipulating the queue. Another cause of queue corruption is file system corruption, which is often caused by a failing disk drive. The qmail-qsanity script (see the "Checking the Queue" section earlier in this chapter) will check the consistency of the queue and report any problems it finds, but it won't fix them.

Eric Huss has written a utility called queue-fix that fixes most easily repairable queue corruptions (http://www.netmeridian.com/e-huss/queue-fix.tar.gz). It can also be used to move the queue to another file system. If you've installed Russell Nelson's big-todo patch, there's a patch for queue-fix that must be installed before building queue-fix (http://www.qmail.org/queue-fix-todo.patch).

For example:

 # queue-fix /var/qmail/queue Creating directory [/var/qmail/queue/local/22] Changing permissions of [/var/qmail/queue/local/22] to [700] Changing ownership of [/var/qmail/queue/local/22] to uid 507 gid 502 queue-fix finished. . . # 

queue-fix takes two options: -N, which causes it to show the actions it would take, without actually taking them, and –i, which causes it to enter "interactive mode" where the user must confirm each action before queue-fix will perform it.

If the queue is corrupt beyond queue-fix's repair abilities, you might be able to manually fix the problem using information provided by qmail-qsanity, queue-fix, and qmail-send.

If all else fails, you might have to start a new queue from scratch. See the next section.

Re-creating an Empty Queue

There are two ways to create an empty queue. The first is to run make setup from the qmail source directory used to install qmail originally. The second method is to use the queue-fix utility covered in the previous section.

Whichever approach you use, stop qmail-send first. Using qmailctl, do this:

 # qmailctl stop Stopping qmail. . .    qmail-smtpd    qmail-send # 

or using svc, do this:

 # svc -d /service/qmail-send/ # 

Next, move the corrupt queue aside—or remove it completely if you don't need to recover any messages in the queue:

 # mv /var/qmail/queue /var/qmail/queue.bad # 

Now generate the new queue.

To generate a new queue using make setup, do this:

 # cd /usr/local/src/qmail-1.03/ # make setup check ./install ./instcheck # 

To generate a new queue using queue-fix, do this:

 # queue-fix /var/qmail/queue Creating directory [/var/qmail/queue/] Changing permissions of [/var/qmail/queue/] to [750] Changing ownership of [/var/qmail/queue/] to uid 505 gid 502 Creating directory [/var/qmail/queue/info] . . .lots more output. . . Creating fifo [/var/qmail/queue/lock/trigger] Changing permissions of [/var/qmail/queue/lock/trigger] to [622] Changing ownership of [/var/qmail/queue/lock/trigger] to uid 507 gid 502 queue-fix finished. . . # 

Finally, restart qmail. Using the qmailctl script, do this:

 # qmailctl start Starting qmail # 

or using svc, do this:

 # svc -u /service/qmail-send/ # 

---