3.4 File Space

Like any other types of data that can accumulate on disks, email queues come with their own data storage management issues. This section provides some advice on how to deal with these issues on busy email servers where many pending messages may be queued up for eventual delivery and where I/O performance is important.

3.4.1 Large Directory Issues

Two issues related to disk storage can affect performance of an email gateway: disk space and the number of files in any given directory. In almost all cases these days, disk space isn't an issue for mail queues. Large disks are cheap. Instead, for mail systems I/O bandwidth is a greater concern than storage capacity. For performance-sensitive systems, an entire disk is likely to be devoted to the mail queue, if not an entire RAID system to take advantage of the NVRAM. It would be quite a trick to fill one of the disks available today with queued email under normal operations without performance becoming a nightmare first. The one scenario where a little caution remains warranted regarding total queue capacity is if network connectivity goes away, even to only a restricted set of sites. In this case, large amounts of email may back up.

As an example, suppose that we are running an ISP with a single machine that aggregates all email sent by our subscribers to the rest of the Internet. Suppose also that we have 100,000 subscribers, each of whom sends, on average, 10 pieces of email to the Internet each day, and that each message averages 10KB in size. Further suppose that 30% of our outbound traffic goes to America Online email addresses, not an implausible supposition. In the unlikely event that AOL's mail system goes off the air for 24 hours, our mail queue may grow to about 3GB in capacity before it starts to drain. While this amount isn't enough to fill a disk, it's still a lot of email. If we increase the number of subscribers at this ISP to 1 million, then we can easily run into sizing problems with contemporary equipment.

Just a few years ago, large ISPs going offline were a much more common occurrence than they are now, and situations where mail queues filled up due to the volume of mail bound for downed sites were not unusual. While the reliability of ISPs and their network connections, as well as the capacity of typical disks, makes this scenario less likely with each passing year, it is not beyond the realm of possibility. Given the low cost of disk capacity, there's no good reason to leave oneself vulnerable to these sorts of situations.

A more likely scenario might involve a corporate environment in which Company A has a close strategic partnership with Company B, but the two have disjoint email systems. An average company's network connection is almost certainly much less fault tolerant than that of a large ISP, so situations in which a company is completely offline are, unfortunately, more common. In this environment, if employees from Company A and B are collaborating on refining PowerPoint presentations or are exchanging business projections created with Excel, CAD drawings, and other large data sets, it doesn't require thousands of people to fill up mail queues when individual messages can easily exceed tens of megabytes in size.

In either case, an email administrator would be well advised to check the log files to find out which domain is the most common destination for email from company servers, and to figure out how many messages and how much storage might be required if that site went offline for an extended period of time. Use experience as a guide for determining the duration of the outage to use in the calculations, but planning for a 24- to 72-hour event (depending on the site) may be a reasonable estimate for a worst-case scenario. If the quantities of email that might be generated during an outage of this magnitude would cause problems for an organization's system, adding capacity might be in order.

It's not just storage capacity that can pose a problem. On most email systems, the number of files in a directory will cause the system to bog down long before disk space is in danger of being exhausted. Therefore, during such a crisis, an email system administrator needs to remain vigilant about the depth of the system's email queues.

Besides space, deep queues can cause another, often overlooked problem. As more entries are added to a directory, the filesystem will obviously need to allocate more space to that directory to hold the information on all of its files. Afterward, even if the number of directory entries returns to a manageable size, the space allocated to the directory itself does not contract. The blank records that used to contain file information must still be scanned as a process walks through the directory looking for files. Certainly, it takes less bandwidth to scan a directory that once contained 10,000 files but now holds 100 files than it does to scan a directory with 10,000 files. However, scanning a large, albeit mostly empty, directory can itself contribute a great deal of overhead to the time it takes to perform directory operations. Therefore, if a directory has become uncommonly large at some point, once the number of entries in that directory has become more manageable it's a good idea to remake that directory to make future lookups more efficient.

Many of the advanced filesystems that use a hash or tree structure for their directory entries can speed lookups on large directories, but they remain adversely affected by large directory sizes. The effect is just not as pronounced. Silicon Graphics' XFS [SDH⁺96] filesystem uses B+ trees to store directory entries, and ReiserFS [REI] uses B* trees for its directory storage. On both of these filesystems, a lookup of a file in a directory at its maximum size with 1,000 entries will likely go faster than a lookup in a directory that currently has 1,000 entries but once contained 100,000 entries.

In advanced filesystems, directory lookups traverse a tree or hash table searching for their entries. The time it takes to perform a lookup operation is typically O(log(N)), where N is the number of entries in a directory. Lookup time grows much more slowly than the time for a linear, O(N), search as N increases. However, it's still expensive to list all entries in a directory, and searches on larger directories always take longer than searches on smaller directories. Therefore, smaller directories are always better it's just that these advanced file systems mitigate the consequences of having larger directories.

3.4.2 Multiple Queues

Starting with sendmail version 8.10, mechanisms are built into the MTA that help keep queue sizes manageable namely, the ability to create and manage multiple simultaneous queues. In the sendmail.mc file, one could add a line like

 define('QUEUE_DIR', '/var/spool/mqueue/*') 

that would specify every directory underneath /var/spool/mqueue/ as an additional queue. Presumably, the email administrator had already created some number of directories, perhaps a dozen, over which queued messages will be divided. As one might expect, the directory path can be made more explicit for example, define('QUEUE_DIR', '/var/spool/mqueue/queue*'). File types also can be split out by making subdirectories in these directories that begin with qf, df, and xf, which house each file type respectively, further restricting the number of files per directory. In this case, sendmail will automatically place the appropriate queue file types in the appropriate subdirectories. Each of these directories and subdirectories can be a symbolic link, which would allow smaller and more frequently modified qf files to be stored on a disk with a small amount of NVRAM or allow multiple disks to be used, for example.

Because the xf files contain information relevant only to the process that created the file, they may be safely written to a memory-based filesystem, such as a Solaris tmpfs. A symbolic link pointing from the xf directory to /tmp would be one way to accomplish this goal on Solaris, as well as on some other platforms. The tf files cannot be separated out in this manner, however, as they act as placeholders for qf files that are being modified. When a sendmail process has completed writing to a tf file, it is rename()ed to supercede its original qf file. If the tf file does not reside on the same filesystem as the qf file, this operation will not occur atomically. For this reason, sendmail always creates tf files in the same directory as the qf file it will replace.

If multiple queue directories are available, with each SMTP connection sendmail determines the queue in which any message will reside pseudorandomly. Starting with sendmail 8.12, it is possible to use rules to segregate traffic bound for a certain host or domain to a special queue. Mechanisms for doing so are discussed in more detail in Section 6.1.3. If multiple queues are used, the mailq command, which lists all entries in the mail queue, will list the queue name and its contents for each queue it can find from listings given in the sendmail.cf file.

While this mechanism might seem little more than a fairly minor workaround for some filesystem's directory deficiencies, multiple queues offer one other advantage: One can gain effective parallelism from multiple queue runners. Recall that when sendmail starts, one parameter given to it on the command line (typically something like -q30m) specifies how often a queue runner should start. This parameter indicates that every 30 minutes, the master sendmail daemon will fork a child process; this child process scans the entire queue looking for messages to deliver, sorts the messages, and then sequentially tries to deliver them. With multiple queues, one queue runner process exists per queue directory. As most of the time during a queue run is spent waiting for responses from email servers across the Internet (or lack of responses remember that most items encountered by a queue runner were undeliverable in a previous attempt), it may take as little as 1/N times as long to complete a queue run over N queues as it does to attempt to deliver every item in a single queue, a significant speedup. This effort helps keep the total number of concurrently queued messages to a minimum.

3.4.3 Queue Migration

One tactic that is sometimes used to speed up the processing of deep queues involves moving entries that aren't being processed out of the main queue(s) and into other directories. This process of queue migration may seem like a good idea, and in some cases it may help. Nevertheless, most of the time better ways exist to manage the situation. Further, queue migration can be a perilous operation.

Queue migration operates by moving files out of one directory and into another. In its most straightforward incarnation, the UNIX mv command is used to relocate qf and df files. A danger exists that one might try to move a message while a sendmail process is operating on it. The sendmail processes keep from colliding with each other by using an operating system call, usually flock(), to place an advisory lock on the qf for the message on which they are working. Migration can work on operating systems, such as FreeBSD, that provide a command such as /usr/bin/lockf, which can obtain a lock before executing another command. On many operating systems, however, such a utility doesn't exist. In these cases, unexpected results can occur while a queued message is being moved unless some other mechanism can guarantee that inconsistencies won't occur, such as killing off or at least sending a STOP signal to all queue runners before manipulating queue entries. Starting with version 8.10, the sendmail distribution comes with a utility written in Perl and located in the contrib subdirectory called qtool.pl that can be invoked from scripts to safely migrate items from queue to queue. Moving messages from one queue to another can prove problematic when using sendmail 8.12's queue group features.

As an aside, note that the FreeBSD lockf command uses the flock() system call to lock a file, not the lockf() library call. This situation is more confusing than it ought to be.

If the destination queue resides on a filesystem different from the queue where the message to be migrated resides, the files must be copied and then deleted. If the two directories are located on the same filesystem, then mv can create entries pointing to the old inodes in the new directory and unlink() the old entries, saving a data copy. In the former case, copying results in many unnecessary I/O operations. Copying the files requires more total I/O (on both disks) than delivering them. Further, even when both directories reside on the same filesystem, the migrator must still perform the same (synchronous) I/O operations in the queue directory that would need to be performed if the message were successfully delivered in its next attempt. Even after the migration takes place, the same I/O operations must occur on that same disk when the message is eventually delivered, resulting in no net savings of disk I/O. Therefore, the only savings from queue migration is a reduced number of directory entries in the queue on future queue accesses. On the downside, queue migration will save very little, or perhaps nothing, in aggregate disk I/O, is potentially risky, and won't result in faster message delivery. Thus moving messages from one queue to another is rarely beneficial. Instead, queue rotation should be used to achieve the same goal.

3.4.4 Queue Rotation

In migrating entries out of a queue, the goal is to not have current and future messages encumbered by having to perform their operations in a large, slow-to-drain queue. Queue rotation is a safer and less I/O-intensive mechanism by which this end can be achieved. In queue rotation, one renames a queue (or queue hierarchy) by running a shell script similar to the following example:

 #!/bin/sh  # Set my umask to match what I want on the queue directory.  umask 077  # Make sure the old queue directory name isn't currently in use.  if [ -d "/var/spool/mqueue.old"]  then          echo "$0 : Old queue directory exists."          exit 1  fi  # Make the new queue directory.  mkdir /var/spool/mqueue.new  # If multiple queues are being used, then subdirectories need  # to be made:  # cd /var/spool/mqueue.new  # mkdir qf df xf  # Quickly move the old directory out of the way and remake it.  # A real script should make sure the mkdir and mv commands  # succeed.  mv /var/spool/mqueue /var/spool/mqueue.old  mv /var/spool/mqueue.new /var/spool/mqueue  # Restart the master sendmail process.  kill -HUP 'head -1 /etc/mail/sendmail.pid'  # Start a queue runner to drain the old queue.  /usr/sbin/sendmail -oQ/var/spool/mqueue.old -q30m 

This script will work with multiple queues under the /var/spool/mqueue directory, but it may not be sufficient if several sendmail 8.12 queue groups have been defined. At the very least, the script must be altered to create the extra directories. Once the mqueue.old directory is empty, its queue runner can be killed. If the machine reboots, a queue runner must be started for every queue that still contains email. Also, note that once queues have been rotated, the mailq command won't display the contents of queue directories that aren't explicitly listed in the sendmail.cf file. This display can be done explicitly: sendmail -bp -oQ/var/spool/mqueue.old. Finally, if this script is used with sendmail prior to version 8.10, a queue identifier might be reused, which could make parsing the log files more difficult.

When sendmail starts, it changes its working directory to the mail queue, /var/spool/mqueue, from which it performs all of its operations. If this directory is renamed, the process remains associated with the directory's inode, not the logical directory name. Thus, after the earlier script executes, all currently running sendmail processes would continue to operate normally in the /var/spool/mqueue.old directory. We then restart the master sendmail process, which will operate in the newly created queue directory. While preexisting sendmail processes will continue to operate in the old queue, once they finish their queue runs, they will exit, leaving mqueue.old unprocessed. For this reason, we start a queue runner specifically to continue processing the old queue. Note that new message processing is not affected by the size of the old queue, already queued messages are processed just as quickly as they were before the rotation, and our cost in terms of I/O operations is a single rename() system call in the message queue's parent directory.

Examining the script closely reveals a race. Between the two mv commands, there exists a window where there is no queue directory. If a sendmail process starts from the command line during this interval, it will fail, giving the following error: can not chdir(/var/spool/mqueue/): No such file or directory. Ofcourse, new sendmail processes spawned due to SMTP connections will not have this problem, as their parent process sits happily in the renamed directory. This problem will generally arise under only two circumstances: (1) if the master sendmail daemon is killed and restarted for some other reason while this script is being run, and (2) if someone interactively sends email from a shell or other mail-related application. The first situation is easy to avoid by policy that is, by making sure nobody restarts this daemon while the rotation script is run. The second case is either easier or more difficult depending on the circumstances. If people log on to the email server to interactively send email, this scenario could manifest itself. If no one sends email from the command line on this machine, then very little risk arises. Generally, it's good policy not to let people log on to an institution's mail gateway. If this is unavoidable, then any queue rotations should occur only during idle hours or in an emergency. Note that the window for this race condition is extremely short, almost certainly much less than one second, so even someone who was looking for this phenomenon would be unlikely to find it under normal operation. Finally, this last problem disappears under the default installation of sendmail 8.12, as a different queue is used when sendmail is invoked from the command line one that isn't being rotated.

Renaming the queue directory is straightforward only if the directory itself is not a filesystem mount point. This consideration is important in setting up an email server. Generally, mounting the queue filesystem on /var/spool/ is a better idea than mounting it on the /var/spool/mqueue directory. On a busy email gateway, not much else should be using general spool space, and this approach allows for rotation of the queue.

Alternatively, an entirely different mount point can be created, such as /queue, to be used solely for email queueing. Subdirectories, such as /queue/current, can be set up and defined in the appropriate .mc file as follows:

 define('QUEUE_DIR','/queue/current') 

It is also possible to mount the queue somewhere else on /queue, for example and to use a symbolic link to accomplish the queue rotation without changing the sendmail.cf file. Under this directory one could put in a number of subdirectories, perhaps creating one for each day, such as 2002-02-23. Then create a symbolic link pointing from /var/spool/mqueue to the appropriate directory. This link could even be automatically rotated by cron each day. The following script accomplishes this goal by using some command options specific to the FreeBSD operating system:

 #!/bin/sh  # Define directories.  yesterday='date -v -1d +%Y-%m-%d'  today='date +%Y-%m-%d'  # Make today's directory.  mkdir /queue/$today  # Rotate the symbolic link.  # Race condition possible between the rm and the ln.  cd /var/spool  rm queue; ln -s /queue/$today queue  # Restart the master daemon.  kill -HUP 'head -1 /etc/mail/sendmail.pid'  # Start a queue runner for yesterday's queue.  /usr/sbin/sendmail -oQ/queue/$yesterday -q30m 

Some more work remains to be done. Something will need to eventually clean out and eliminate empty queues. Also, queue runners for the old queues need to be started when the server reboots. This script should suffice as an example, however.

This technique has the same window of vulnerability as the mv and mkdir technique. This problem is unavoidable, as no operating system allows the atomic repointing of symbolic links. Also, on the downside, the use of symbolic links generally indicates a kludge of some sort. The upside is that one no longer needs to have the rest of /var/spool reside on the same filesystem as the mail queues, an appealing idea. Either of these methods can be used in practice, and preference is largely a matter of personal taste.

With the symbolic link method, it might be tempting to perform disk housecleaning by simply deleting directories older than the largest Timeout.queuereturn.* value in the sendmail.cf file, perhaps plus a couple of days to provide a margin for error. This deletion should not be done blindly. If the directory contains any qf files, the Internet mail standards require that a "bounce message," a special type of Delivery Status Notification (DSN), be returned for each undelivered message. If an old directory that is not being accessed by any sendmail processes which may receive new messages to handle has no qf files in it, then it can be safely deleted. If one runs sendmail -q on a directory in which all files are older than Timeout.queuereturn.*, and qf files remain after the process completes its queue run, then something that shouldn't happen has happened and a person should investigate. Thus, a script written to clean up and remove old queue directories under these circumstances should perform the following steps:

1. Make sure the last file created in that directory is older than the largest Timeout.queuereturn.* value in the sendmail.cf file.

2. Make sure no sendmail process is currently operating on the directory (using the fuser or lsof utility). If a process is operating on the queue, it should be killed.

3. If the directory still contains any qf files, run sendmail -q -oQDirectoryName to make sure bounce messages are sent for all messages in that queue.

4. If, after the queue runner has completed, qf files remain in that directory, inform a person who can investigate. The script should then exit, reporting an error condition.

5. If no more qf files appear in that directory, the directory may be safely removed.