Message Store Formats

When Postfix makes local deliveries it transfers the contents of messages to the local message store. The most common types of message stores are the traditional mbox format and the newer maildir style. Both use regular files to store messages, but they are structured in different ways. In Postfix, you specify maildir style by including a trailing slash when you configure any mail file or directory parameters (see configuration information later in this chapter).

7.2.1 The Mbox Format

Historically, Unix systems have used a single file to store each user's email messages. This type of message store format is commonly referred to as mbox. Each message within the file starts with a line that begins with the word From. It is important that the string start on the first character of the line, and that there is a space after the end of the word. The From line is commonly referred to as From_ with an underscore character to indicate the space following the word. Don't confuse the From_ line used for separating messages within an mbox file with the From: line included in email message headers. The last line of a message is always a blank line.

A complete From_ line looks like the following:

From [e-mail] Sun Feb 3 16:54:01 2002

As described, the line starts with the word From followed by a space. Following the space is an email address that is usually the envelope address of the message. Following the envelope address is the date of delivery in the common Unix date format occupying 24 characters. The mbox format allows for an optional comment string following the date, but it is generally not used.

When Postfix delivers a message to an mbox file, it first creates the From_ line using the envelope sender and the current date. Postfix then copies the contents of the delivered message into the mbox file. If Postfix encounters any lines that begin with From followed by a space, it has to quote them by adding a > to the beginning of the line, so that they won't be confused with the start of the next message.

When a POP/IMAP server reads messages from the mbox file, it scans the file, looking for From_ lines, which mark the beginning of each message. It can read to the next From_ line (or the end of the file) to know when a message is finished. The POP/IMAP server may unquote any of the ">From" quoted lines, or they may remain in the quoted form.

Since both Postfix and the POP/IMAP servers access the mailbox file, they must use file locking. Postfix must obtain an exclusive lock on the file when it is delivering a message, so that it can write the message to the file. Postfix offers a variety of locking mechanisms, depending on the platform. You can use the postconf -l command to see which mechanisms Postfix can use on your system:

$ postconf -l
flock
fcntl
dotlock

If you want more information about the locking types listed by Postfix on your system, check your system's man pages for the specific lock name:

$ man flock

The dotlock type, which should be available on all systems, is probably not documented on your system, because it is not a function of the operating system or supporting libraries as flock and fcntl are. The dotlock is simply a file. The lock file name is made up of the name of the file to be locked with a .lock extension appended to it. If such a lock file exists, then Postfix knows that another process is using the mail file. If the file does not exist, Postfix creates it to signal other processes that it is using the file. When Postfix is finished, it removes the lock file, making the mail file available again. The drawback of dotlock locking is that it is susceptible to stale locks, and it is not very efficient.

For the most part, you do not need to worry about locking, and the lock types available, because Postfix does a good job of figuring out the best option.

7.2.2 The Maildir Format

The maildir mailbox format differs from mbox in that it uses a structure of directories to store email messages. It was designed to solve some of the reliability and locking problems of the mbox format. For example, if a system crashes at the instant an email message is being delivered to an mbox file, it is possible that the message will be truncated at the point where the delivery was interrupted. When the system comes back online, the mail transport agent will attempt to deliver the message again. The partially written message at the bottom of the mbox file may cause problems when the next message is appended to the file.

Other problems can occur if a POP/IMAP server tries to access the mbox file at the same time as the SMTP server. If the programs do not use the same locking mechanism, the mail file will most likely be corrupted. There are several possible mail file locking mechanisms (see above), which are not necessarily used by all mail programs. With the maildir format, no locks are necessary because each message gets its own file. Different mail processes do not need access to the same files at the same time.

A maildir-style directory has three subdirectories, which must all be on the same filesystem: tmp, new, and cur. These subdirectories are usually below a mail directory in a user's home directory:

$ ll /home/kdent/maildir
total 12
drwxr-x--- 2 kdent kdent 4096 Mar 13 12:24 cur
drwxr-x--- 2 kdent kdent 4096 Mar 13 12:24 new
drwxr-x--- 2 kdent kdent 4096 Mar 13 12:24 tmp

Files in the new directory are messages that have been delivered but have not yet been read. The modification time of the file is the delivery date of the message. The file usually contains the message in RFC 2822 format, and no From_ line is needed.

Once a message has been viewed, it is moved to the cur directory. The tmp directory is used during message delivery to store the contents of a file before it can be confirmed to have been written to the new directory.

7.2.3 Mbox Versus Maildir

There is no simple answer to help you decide which type of mailbox format is best for you. The mbox format has the advantage of being almost universally supported, but has the file-locking problems that prompted the development of the maildir format. On the other hand, there are concerns about the ability of the maildir format to scale to handle large numbers of messages on some filesystems. There are performance arguments to support both formats: locating and accessing or deleting a particular message is probably quicker with maildir, but delivery by simply appending the text of a message to the end of a single file is probably quicker in the mbox format. Your choice will most likely be driven by your selection of a POP/IMAP server. If you settle on a POP/IMAP server that requires the maildir format, the choice is made for you. Postfix easily supports either format, so you can safely allow other considerations to drive your decision. If you think it will be significant in your environment, you should run tests of both formats, simulating your own mail tasks as closely as possible.