Recipe 9.2 Using qf, df, and xf Subdirectories

Problem

Special configuration is required to use multiple mail queues for the different queue file types.

Solution

Create a new queue directory structure with separate directories for df , qf , and xf queue files. Here is an example:

 #  umask 077  #  mkdir /var/spool/mqueue.new  #  cd /var/spool/mqueue.new  #  mkdir df qf xf  

Move the old queue directory out of the way so that the new queue directory structure can be given the pathname defined by the QueueDirectory option in the sendmail.cf file. Here is an example:

 #  mv /var/spool/mqueue /var/spool/mqueue.old  #  mv /var/spool/mqueue.new /var/spool/mqueue  

Start another copy of sendmail as a queue runner to drain the old queue. Use the -O argument on the sendmail command line to point the queue runner to the old queue. Here is an example:

 #  sendmail -L sm-oldq -q15m -O QueueDirectory=/var/spool/mqueue.old  

Discussion

sendmail stores df , qf , and xf files in separate directories when the default queue directory contains subdirectories named df , qf , and xf . As with the multiple queues described in Recipe 9.1, df , qf , and xf subdirectories reduce the size of any individual queue directory, and they allow the subdirectories to be placed on separate devices to overcome the I/O limitations of a single device. These benefits aid in the processing of a very large mail queue. Additionally, using separate subdirectories for the different queue files allows you to select a device that is optimized for the type of queue file that it will handle. These are the file characteristics to consider when selecting a device:

df

A df file contains the body of the mail message. Usually, it is the largest of the three files. A df file is written to the queue once and read once for each delivery attempt, so it is usually handled less than the other files. Storage capacity is the primary consideration when selecting a device for df files.

qf

A qf file contains the delivery and queuing instructions for the message, as well as the message headers. A qf file is small, but it is rewritten each time a delivery is attempted. When a message has many recipients or is requeued several times, sendmail handles the qf file many times. Performance is an important consideration when selecting a device to handle qf files.

xf

An xf file is created for each delivery attempt and is valid only for that attempt. xf files are small and they have a short life. Performance is the primary consideration when selecting a device for xf files.

These subdirectories can be created in any combination. You could, for example, create just an xf directory ”sendmail would place xf files in that directory and place df and qf files in the base directory. This recipe creates a temporary directory, /var/spool/mqueue.new , as the base directory. All three subdirectories, df , qf , and xf , are then created within that directory. The old queue directory is moved out of the way and the new directory is renamed to the value defined by the sendmail.cf QueueDirectory option, which, in the example, is /var/spool/mqueue . sendmail will begin to use the new directory structure as soon as it is renamed.

As Nick Christenson explains in his book, sendmail Performance Tuning (Addison Wesley), the apparent race condition caused by renaming sendmail's queue directory while sendmail is running is minimal. Mail sent by the MSP uses the queue directory defined in submit.cf , which is usually /var/spool/clientmqueue , so it is unaffected when mqueue is renamed. ^[1] A process that is running when the directory is renamed is unaffected because it uses the inode of the queue directory not the logical name . Therefore, if the process started with the old queue, it will continue with the old queue even after the queue directory is renamed. After the master sendmail process is restarted, all subsequent sendmail processes will use the new queue.

^[1] sendmail prior to Version 8.12 did not use the submit.cf file.

Of course, mail queued in the old directory will remain there after the directory is renamed unless a process is started to drain the old queue. Use the -O argument on the command line to point the queue runner to the old queue, as shown in Recipe 9.2.2. Periodically check the old queue to see if it has drained using a command such as the following:

 #  sendmail -bp -O QueueDirectory=/var/spool/mqueue.old  

When the old queue is empty, kill the special queue runner and remove the unneeded directory.

The df , qf , and xf subdirectories can be placed on separate physical devices. For example, on a Linux system, the xf subdirectory could be placed on /dev/shm , which is a shared memory device using the Linux tmpfs filesystem format. Placing the xf subdirectory on a volatile, memory-based filesystem that provides very high performance at the cost of long- term storage works because xf files do not need to be preserved between queue runs or reboots.

Multiple queues, such as those created in Recipe 9.1, can be combined with df , qf , and xf subdirectories to gain the advantages of both multiple queues and subqueues for a system that handles a very large mail queue. Recipe 9.1 created three queue directories. Within each of those, we could create df , qf , and xf subdirectories for a total of nine different directories, which could then be allocated to physical devices in any manner you wished.

See Also

Recipe 9.1 also creates multiple queues that can be combined with this recipe. The sendmail book covers the df , qf , and xf subdirectories in Section 11.3.2 and in Section 6.5. sendmail Performance Tuning , by Nick Christenson (Addison Wesley), covers this topic in Section 3.4.4.