How Email Is Sent and Received

Email is transmitted as plain text across networks around the world using the SMTP protocol (Simple Mail Transfer Protocol). As the name implies, the protocol itself is fairly basic, and it has been extended to add further authentication and error reporting/messaging to satisfy the growing demands of modern email. Mail transfer agents, or MTAs, work in the background transferring email from server to server allowing emails to be sent all over the world. You may have come across such MTA software such as Sendmail, Postfix, Fetchmail, Exim, or Qmail.

SMTP allows each computer that the email passes through to forward it in the right direction to the final destination. When you consider the millions of email servers across the world, you have to marvel at how simple it all seems.

Here is a simplified example of how email is successfully processed and sent to its destination:

1.	[e-mail] composes and sends an email message to [e-mail].
2.	The MTA at hudson.org receives Andrew's email message and queues it for delivery behind any other messages that are also waiting to go out.
3.	The MTA at hudson.org contacts the MTA at hudzilla.org on port 24. After hudzilla.org acknowledges the connection, the MTA at hudson.org sends the mail message. After hudzilla.org accepts and acknowledges receipt of the message, the connection is closed.
4.	The MTA at hudzilla.org places the mail message into Paul's incoming mailbox; Paul is notified that he has new mail the next time he logs on.

Of course, several things can go wrong during this process. Here are a few examples:

What if Paul does not exist at hudzilla.org? In this case, the MTA at hudzilla.org will reject the email and notify the MTA at hudson.org of what the problem is. The MTA at hudson.org will then generate an email message and send it to [e-mail], informing him that no Paul exists at hudzilla.org (or perhaps just silently discard the message and give the sender no indication of the problem, depending on how the email server is configured).

What happens if hudzilla.org doesn't respond to hudson.org's connection attempts? (Perhaps the server is down for maintenance.) The MTA at hudson.org notifies the sender that the initial delivery attempt has failed. Further attempts will be made at intervals decided by the server administrator until the deadline is reached, and the sender will be notified that the mail is undeliverable.

The Mail Transport Agent

Several MTAs are available for Ubuntu, each with its pros and cons. Normally they are hidden under the skin of Ubuntu, silently moving mail between servers all over the world with need for little or no maintenance. Some MTAs are extremely powerful, being able to cope with hundreds of thousands of messages each day, whereas some are more geared toward smaller installations. Other MTAs are perhaps not as powerful, but are packed full with features. In the next section, we take a look at some of the more popular MTAs available for Ubuntu.

Postfix

Postfix has its origins as the IBM Secure Mailer, but was released to the community by IBM. Compared to Sendmail, it is much easier to administer and has a number of speed advantages. Postfix offers a pain-free replacement for Sendmail, and you are able to literally replace Sendmail with Postfix without the system breaking a sweat. In fact, the applications that rely on Sendmail will automatically use Postfix instead and carry on working correctly (because Postfix uses a Sendmail wrapper, which deceives other programs into thinking that Postfix is Sendmail). This wrapper, or more correctly interface, makes switching to Postfix extremely easy if you are already running Sendmail. Postfix also happens to be the MTA of choice for Ubuntu, so it is this one that we spend more time on later in this chapter.

For enhanced security, many Postfix processes used to use the chroot facility (which restricts access to only specific parts of the file system) for improved security, and there are no setuid components in Postfix. With the current release of Ubuntu, a chroot configuration is no longer used and is, in fact, discouraged by the Postfix author. You can manually reconfigure Postfix to a chroot configuration, but that is no longer supported by Ubuntu.

If you are starting from scratch, Postfix is considered a better choice than Sendmail.

Sendmail

Sendmail handles the overwhelming majority of emails transmitted over the Internet today. It is extremely popular across the Linux/UNIX/BSD world and is well supported. A commercial version is available that has a GUI interface for ease of configuration.

As well as being popular, Sendmail is particularly powerful compared to some of the other MTAs. However, it is not without its downsides, and you will find that other MTAs can handle more email per second in a larger environment. The other issue with Sendmail is that it can be extremely complicated to set it up exactly as you want it. A few books are available specifically for Sendmail, but the most popular one has more than a thousand pages, reflecting the complex nature of Sendmail configuration.

We can be thankful, however, that the default configuration for Sendmail works fine for most basic installations out of the box, making further configurations unnecessary. Even if you want to use it as a basic email server, you only need to do some minor tweaks. The level of complexity associated with Sendmail often leads to system administrators replacing it with one of the other alternatives that is easier to configure.

Qmail and Exim

Qmail is a direct competitor to Postfix but is not provided with Ubuntu. Postfix is designed to be easier to use than Sendmail, as well as faster and more secure. However, Qmail isn't a drop-in replacement for Sendmail, so migrating an existing Sendmail installation to Qmail is not quite as simple as migrating from Sendmail to Postfix. Qmail is relatively easy to administer, and it integrates with a number of software add-ons, including web mail systems and POP3 servers. Qmail is available from http://www.qmail.org/.

Exim is yet another MTA, and it is available at http://www.exim.org/. Exim is considered faster and more secure that Sendmail or Postfix, but is much different to configure that either of those. Exim and Qmail use the maildir format rather than mbox, so both are considered "NFS safe" (see the following sidebar).

MDIR Versus Mailbox

Qmail also introduced maildir, which is an alternative to the standard UNIX method of storing incoming mail. maildir is a more versatile system of handling incoming email, but it requires your email clients to be reconfigured, and it is not compatible with the traditional UNIX way of storing incoming mail. You will need to use mail programs that recognize the maildir format. (The modern programs do.)

The traditional mbox format keeps all mail assigned to a folder concatenated as a single file and maintains an index of individual emails. With maildir, each mail folder has three subfolders: /cur, /new, and /tmp. Each email is kept in a separate, unique file. If you are running a mail server for a large number of people, you should select a file system that can efficiently handle a large number of small files.

mbox does offer one major disadvantage. While you are accessing the monolithic mbox file that contains all your email, suppose that some type of corruption occurs, either to the file itself or to the index. Recovery from this problem can prove difficult. The mbox files are especially prone to problems if the files are being accessed over a network and can result in file corruption; one should avoid accessing mbox mail mounted over NFS, the Network File System, because file corruption can occur.

Depending on how you access your mail, maildir does permit the simultaneous access of maildir files by multiple applications; mbox does not.

The choice of a mail user agent (MUA), or email client, also affects your choice of mail directory format. For example, the pine program does not cache any directory information and must reread the mail directory any time it accesses it. If you are using pine, maildir is a poor choice. More-advanced email clients perform caching, so maildir might be a good choice, although the email client cache can get out of synchronization. It seems that no perfect choice exists.

Ubuntu provides you with mail alternatives that have both strong and weak points. Be aware of the differences among the alternatives and frequently reevaluate your selection to make certain that it is the best one for your circumstances.

Choosing an MTA

Other MTAs are available for use with Ubuntu, but those discussed in the previous sections are the most popular. Which one should you choose? That depends on what you need to do. Postfix's main strengths is that it scales well and can handle large volumes of email at high speeds, not to mention that it is much easier to configure than the more cryptic Sendmail. However, you may find that there are specific things that you need that only Sendmail can provide. It is easy to switch between MTAs when you need to.

The Mail Delivery Agent

SMTP is a server-to-server protocol that was designed to deliver mail to systems that are always connected to the Internet. Dial-up systems only connect at the user's command; they connect for specific operations, and are frequently disconnected. To accommodate this difference, many mail systems also include a mail delivery agent, or MDA. The MDA transfers mail to systems without permanent Internet connections. The MDA is similar to an MTA (see the following note), but does not handle deliveries between systems and does not provide an interface to the user.

Note

Procmail or Spamassassin are examples of MTAs; both provide filtering services to the MTA while they store messages locally and then make them available to the MUA or email client for reading by the user.

The MDA uses the POP3 or IMAP protocols for this process. In a manner similar to a post office box at the post office, POP3 and IMAP implement a "store and forward" process that alleviates the need to maintain a local mail server if all you want to do is read your mail. For example, dial-up Internet users can intermittently connect to their ISP's mail server to retrieve mail using Fetchmailthe MDA recommended by Ubuntu (see the section "Using Fetchmail to Retrieve Mail" later in this chapter).

The Mail User Agent

The mail user agent, or MUA, is another necessary part of the email system. The MUA is a mail client, or mail reader, that allows the user to read and compose email and provides the user interface. (It is the email application itself that most users are familiar with as "email.") Some popular UNIX command-line MUAs are elm, pine, and mutt. Ubuntu also provides modern GUI MUAs: Evolution, Thunderbird, Mozilla Mail, Balsa, Sylpheed, and KMail. For comparison, common non-UNIX MUAs are Microsoft Outlook, Outlook Express, Pegasus, Eudora, and Netscape Messenger.

The Microsoft Windows and Macintosh MUAs often include some MTA functionality; UNIX does not. For example, Microsoft Outlook can connect to your Internet provider's mail server to send messages. On the other hand, UNIX MUAs generally rely on an external MTA such as Sendmail. This might seem like a needlessly complicated way to do things, and it is if used to connect a single user to her ISP. For any other situation, however, using an external MTA allows you much greater flexibility because you can use any number of external programs to handle and process your email functions and customize the service. Having the process handled by different applications gives you great control over how you provide email service to users on your network, as well as to individual and small office/home office (SOHO) users.

For example, you could do the following:

• Use Evolution to read and compose mail

• Use Sendmail to send your mail

• Use xbiff to notify you when you have new mail

• Use Fetchmail to retrieve your mail from a remote mail server

• Use Procmail to automatically sort your incoming mail based on sender, subject, or many other variables

• Use Spamassassin to eliminate the unwanted messages before you read them