9.1 About Electronic Mail

As with regular postal mail, a properly functioningelectronic-mail system depends on a series of distinct and often geographically-separated facilities and processes working together. Typically, each of these parts is handled by one or more programs specifically designed to perform the corresponding tasks.

In general, on Unix systems, the electronic mail facility is composed of the following components:

Programs that allow users to read and write mail messages

In the jargon, such programs are known as mail user agents . There are a variety of such programs available, ranging from the traditional (and primitive) mail command to character-based, menu-driven programs such as elm, mutt, pine, and the mh family, to Internet-integrated packages such as Netscape (some users also prefer the mail facilities embedded within their favorite editor, such as emacs). These programs require only a little administrative time and attention, usually consisting of setting system-wide defaults for the various packages.

Programs that accept outgoing email (submission agents), send it along its way, and begin the delivery process

Delivering mail to its final destination is the responsibility of mail transport agents , which relay mail messages within a site or out onto the Internet toward their final destinations. Transport agents run as daemons, and they generally use the directory /var/spool/mqueue as a work queue to hold messages waiting for processing.

sendmail is the traditional Unix transport agent. sendmail usually functions as the submission agent as well, although some mail programs (user agents) now incorporate this capability themselves. Current estimates indicate that sendmail handles about 75% of all email. Other available transport agents include Postfix, qmail, and smail. At present, transport agents most often use the Simple Mail Transfer Protocol (SMTP) to exchange data, although other transport protocols are seen occasionally (e.g., UUCP).

Programs that transfer messages to the user's mailbox

Once mail arrives at its destination, the transport agent hands it off to a delivery agent that actually places messages into the appropriate user's mailbox (among other tasks). User mailboxes are located in /var/mail (/var/spool/mail under AIX, FreeBSD, and Tru64) and consist of text files named for the corresponding user account.

There may be different delivery agents for the various classes of messages (e.g., local versus remote) and different transport protocols (e.g., SMTP versus UUCP). Commonly used delivery agents include procmail, mail, rmail, and mail.local (the latter is part of the sendmail package).

Programs that retrieve stored messages from an ISP or other holding location

When a user or organization has only an intermittent connection to the Internet, incoming remote messages are usually stored on their ISP's server until they are ready to collect them. Periodically, such users/sites must establish a connection to the ISP and send out all new messages and retrieve those waiting for them. The program that performs these actions might be termed a retrieval agent ,^[1] and fetchmail is the most common. Once messages have been downloaded, they are usually handed off to a transport agent for local routing and delivery.

^[1] What I'm calling a retrieval agent can also be thought of as a type of access agent (see the following paragraph).

Programs to access delivered messages from a different computer

Some organizations and individual users choose to access email from a computer other than the one where their mailbox is located (the target location for the delivery agents). For example, a user at a site with a central mail server may prefer to read his mail on his own workstation rather than on the designated server. Such schemes use a message store to hold accumulated messages. They may be stored in traditional user mailboxes files within the designated mail spool director or as records in a database. The user agent must connect to the message store to view, access, manipulate, and potentially download the messages. When doing so, the user agent is functioning as an access agent . The message retrieval processes uses the Post Office Protocol (POP3) or Internet Message Access Protocol (IMAP) for communication.

Figure 9-1 illustrates some of these components and concepts via a sample mail message sent from Hamlet (user account hamlet at uwitt.edu) to his friend Ophelia ([e-mail]).

Figure 9-1. Example email configuration

Hamlet composes his message to Ophelia using a mailer program like pine or mutt on one of the workstations in his department (hostname philo). Depending on his user agent and its exact configuration, it may forward the message to the local sendmail process using port 587, allowing sendmail to submit it to the mail facility, or it may do the submission operation itself, communicating with sendmail via SMTP on port 25 (the transport agent standard port). In our example, pine has been configured to function as a submission agent as well as a user agent, while mutt relies on sendmail for mail submission.

At this site, all outgoing mail is funneled through a single mail relay host named apollo, so the sendmail process on philo passes the message along to the corresponding process on apollo, which in turns relays it to the Internet. From there, the message will eventually be sent to [e-mail], which is redirected (via DNS MX records) to some system at the ISP used by the elsinore.gov site.

When convenient, the incoming mail server at elsinore.gov, named poste, connects to the ISP and uses the fetchmail program to retrieve waiting messages. fetchmail then forwards the data to sendmail, using the SMTP protocol and port 25, thereby simulating normal incoming TCP/IP mail. The sendmail process on poste sends the messages for user ophe624 to the sendmail process on polonius, where the procmail program places it in the correct mailbox, /var/spool/mail/ophe624.

From the point of view of the sendmail transport agent, the mail is now delivered. However, Ophelia still hasn't seen the message. She typically reads her mail on her laptop. To do so, she has configured Netscape's email component to connect to the message store in this case, her mailbox on polonius (providing the appropriate username and password for authentication). Once she's connected, Netscape displays information about the messages in her mailbox, showing her the actual message when requested, retrieving all data via the IMAP protocol. At her option, Ophelia can delete the message, download it to her laptop, or file it away into one of her mail folders on polonius (or even leave it in her incoming mailbox).

If elsinore.gov had a direct Internet connection, the initial delivery of mail messages to their site would be somewhat different. Instead of using fetchmail to retrieve messages from a remote ISP site, mail would arrive at the computer designated for that domain via DNS MX records. Most often, this means the site's firewall, where some extra precautions are taken. Instead of running sendmail on the firewall, which involves significant security risks, the firewall can run a much simpler, unprivileged daemon that forwards SMTP packets to a designated host inside the firewall (in our example, poste could again be used for the latter at elsinore.gov). Such a daemon is known as an SMTP proxy.

For added security, this function can be split into two noncommunicating processes. In fact, the most widely used SMTP proxy facility is the combination of smtpd to receive and store incoming SMTP data and smtpfwdd to forward SMTP data to the incoming mail server (available from http://www.obtuse.com). The smtpd daemon simply accepts SMTP packets, constructs mail messages, and writes them into a spool directory on disk (e.g., /var/spool/smtpd). Sometime later, smtpfwdd reads messages from that location and invokes a program to submit them to the mail system. Usually, this program is sendmail, and it forwards the messages to a transport agent inside the firewall. On the firewall system, however, sendmail does not run as a daemon and is configured to accept mail only from smtpfwdd. This configuration is illustrated in Figure 9-2.

Figure 9-2. An SMTP proxy at a firewall

Both daemons implement only the minimal set of SMTP commands to perform their tasks. By limiting the proxy processes' functions to simple reading and writing, any potential troubles arising from malicious SMTP commands are avoided. In addition, smtpd can optionally filter messages based on a variety of access control settings, and smtpfwdd can filter messages based on content.

smtpd is designed to be invoked on demand by inetd, so adding an entry for it to the /etc/inetd.conf configuration file is part of the installation process. smtpfwdd runs as a daemon and is, accordingly, started at boot time. Consult the accompanying documentation for more information about these programs.

9.1.1 Mail Addressing and Delivery

So far, we have considered only the most straightforward mailaddressing case: a message is addressed to a user at a particular site. However, several complications can arise, making real-world delivery of actual mail messages much more complex:

• DNS MX records can redirect a message to a host other than the one to which it was directed.

• Name-mapping functions in the transport agent can map public email addresses to local user accounts and/or hosts (e.g., [e-mail] to chavez@dalton).

• Email aliases can redirect incoming messages for a user to a different host and/or user (or even to a group of users).

• Mail-forwarding mechanisms can also redirect mail to a different destination address, a facility typically used for users who are away from their home site for an extended period (e.g., on vacation) or who have left an organization altogether.

We will consider the first, third, and fourth items in this section. Transport agent name mapping is discussed later in this chapter.

9.1.1.1 DNS MX records

DNS MX records specify the host(s) that handle email for a given computer. They cause email addressed to that host to be sent to a new destination system rather than being delivered on that host itself. MX records have the general format:

host [ttl] IN MX n destination

host is the computer to which the record applies, n is a number indicating the record's priority level (lower numbers indicate higher priority), and destination is the name of the host to which mail should be (re)directed. Note that the destination can be the same as the host itself, and it often is. The hostname specified must be the one used in the corresponding A record; CNAMEs are not allowed. (ttl is the usual, optional caching time-to-live parameter.)

Here are some examples for the domain ahania.com:

dalton      IN  MX  10  dalton             IN  MX  20  postal             IN  MX  90  remote.ahania.com.    newton      IN  MX  10  apple             IN  MX  20  postal    ahania.com. IN  MX  10  granada.ahania.com.             IN  MX  20  laguna.ahania.com.

Host dalton normally receives its own mail since it is listed as its own highest-priority destination host. That is, mail addressed to someone@dalton or [e-mail] is delivered to host dalton. If dalton is unavailable, mail is redirected first to host postal (i.e., postal.ahania.com) and then to host remote.ahania.com, if postal is also down.

In contrast, email destined for host newton is redirected to host apple under normal circumstances. In other words, mail to someone@newton or [e-mail] is actually delivered to host apple. If apple is unavailable, mail goes to postal instead. Thus, in this example, postal serves as a backup mail server for both hosts.

The final two lines specify a default mail destination system for mail addresses of the form [e-mail]. By default, mail addressed to a user at this site without including any hostname in the address is routed to the system granada, which serves as the incoming mail server for that site. System laguna is specified as a backup mail destination.

9.1.1.2 Mail aliases

Mail aliases are another way of rerouting email. In contrast to DNS MX records, these operate on a per-user basis. Mail aliases are usually defined in the file /etc/aliases (or occasionally /etc/mail/aliases); this facility is provided by the transport agent. These aliases are automatically applied to the local recipients of incoming mail. Names specified in email message addresses are compared against the entries in the aliases file and are translated according to its directives.^[2]

^[2] This is true for the most common mail system configurations. If NIS or LDAP is in use, the situation can be a bit more complicated. This issue is covered in detail in the discussion of transport agents later in this chapter.

Some mailer programs also allow users to define personal mail aliases, but these apply only to outgoing messages created by that specific user, and they won't be considered here since they are expanded before the message enters the larger mail system.

Entries in the aliases file have the following format:

local-name: user [, user  . . . ]

Aliases may be continued onto as many lines as needed by indenting the second and subsequent lines. This line has the affect of translating the specified local-name into its corresponding expansion (whatever follows the colon) whenever it is encountered as an email address by the transport agent on the local system.

Here are some example entries:

eve:      ewood ewood:    ewood@altos  ike:      \issac@newton    chem:     enzo, nadia, vala phys:   ike,    enzo,    kip  science:  chem, phys, max  vala:     [e-mail]

The first three entries illustrate user account aliases. In this case, mail for eve is redirected to ewood. The name ewood is itself an alias, and it expands to ewood@altos, so mail coming to this system for eve would go to ewood@altos (at least to start with). Aliases continue to be expanded up to ten levels deep. In the same way, the third entry defines an alias for ike: \issac@newton. This is a terminal alias: the initial backslash prevents any further expansion on the local system, including via .forward files (see below).

The next three sample entries are used to define some local mailing lists. The first two lists have three members each. The third list, science, has two other mailing lists as its members (along with max). Any duplicates in the resulting list are automatically removed by sendmail (user enzo in our example). Note also that entry order is irrelevant in the aliases file. Thus, the alias defining vala does not need to precede its use in the chem mailing list.

The component email addresses for mailing lists may also be listed in an external file; the alias itself is then defined via an include directive in the aliases file, as in this example:

curry: :include:/usr/local/mail_lists/curry_lovers.list

The full path to the include file must be specified (the directory location in the example is arbitrary). In this case, curry_lovers.list is a text file containing the list of email addresses for this mailing list. You will also see aliases such as owner-curry and similar names, which are used for mailing list administration.

Names encountered in email addresses which are not defined in the aliases file are assumed to be usernames on the local system under normal circumstances. You can also configure some transport agents to perform other kinds of address lookups (as we'll see). Similarly, unqualified names (i.e., without an @host part) in alias definitions within the aliases file are also interpreted as local usernames.

The sendmail facility and other transport agents do not access the aliases file directly. Instead, they use binary random access databases to speed up the alias expansion process. Whenever you edit the aliases file, you must update these binary files by running the newaliases command (no arguments required).^[3] However, newaliases does not need to be run when you edit a list file specified with an include directive.

^[3] newaliases is equivalent to sendmail -bi, which may be used in those rare cases in which no newaliases command is provided.

Aliases may also be used to redirect mail messages to a file or a program, via entries like the following:

help:  help-list, /data/help/incoming info:  "|/usr/local/admin/send_info"

The first alias directs mail addressed to help to help-list and also appends it to the file /data/help/incoming. The second alias pipes mail messages to the specified program.

Any file specified in an aliases file entry must already exist. For sendmail, the file must also be writable by the package's default user (a configuration option discussed later in this chapter), and it must be setuid to owner but not be executable (i.e., chmod -x,u+s). This unusual permission requirement makes it quite unlikely that any file will be accidentally overwritten. Postfix also requires the file to be writable by its default user.

Defining a pipe as an alias sends mail messages to the standard input of the specified program. The program runs as the transport agent's default user, and the program's working directory is set to the mail queue directory (usually /var/spool/mqueue ). By default, the program is executed by /bin/sh for sendmail, although you can (and should) specify a different shell for security reasons (discussed later). Postfix attempts to run the program directly but falls back to /bin/sh if necessary.

NOTE

Using a single aliases file has a number of advantages, including limiting alias administration to a single point and preventing some sorts of mail bouncing problems. Such a file lists every user at the site and defines an alias for each that points to the system where he receives or collects email. This master aliases file can be distributed using one of the methods described in Chapter 14.

9.1.1.3 Mail forwarding

Mail forwarding is the third mail redirection mechanism we will consider. Mail forwarding for specific users can be specified at the site level using features of the transport agent, or it can be accomplished by an individual user himself. Mail redirection using sendmail and other transport agents is usually performed when a user has permanently left an organization. We will discuss these facilities when we consider sendmail and Postfix in detail, later in this chapter.

User-specified mail forwarding uses the same basic idea as mail aliases. A user can cause his mail to be automatically forwarded to a different address by creating a file named .forward in his home directory.^[4] This file contains one or more email addresses to which email should be forwarded (the simplest format is to put each address on a separate line). For example, if the .forward file in user chavez's home directory contained the single line [e-mail], her email would be forwarded to the specified address. If she wanted to keep a local copy of the mail as well, she could use this .forward file:

^[4] Actually, the mail-forwarding file path is a configurable list within the transport agent.

[e-mail], "/home/chavez/mail_pile"

This file would forward the mail to the same address and also place a copy of each forwarded message into the file mail_pile in her home directory. The target file must already exist and be owned by user chavez, and common sense dictates that it should be writable only by the user herself, as should every component subdirectory of the directory tree in which it resides.

In some configurations/versions, sendmail enforces these file protection requirements and will not append mail to files that are group- or world-writable or are placed in an insecure directory location. Postfix has similar requirements.

With sendmail, forwarding messages to pipes or files also requires that the user's login shell be listed in the /etc/shells file. If this file is not used (e.g., under AIX), you must create it manually (or rely on the internal default list of /bin/sh and /bin/csh). You can disable this requirement by including the following line within the shells file:

/SENDMAIL/ANY/SHELL/

Such an entry is necessary to enable forwarding for users whose shell prevents logging in to the mail server (e.g., having nologin as their shell).

9.1.1.4 Putting it all together

So how do all of these separate pieces interact to deliver mail? First, the MX records are examined to see whether email is rerouted at the DNS level. If so, the mail is sent to the same user at the new host.

If no MX record causes the mail to go to another host, the address is processed for aliasing via the aliases file and then the forwarding mechanism. Either of these has the potential to redirect the mail to a different user and/or host. If the host changes, the message is routed to the specified host (where MX record checking begins again). On the other hand, if the aliasing does not redirect the message to a different host, the message is delivered to the appropriate user on the local system.

Let's consider an example (illustrated in Figure 9-3).^[5] Consider a message addressed to [e-mail] (arriving from some remote sender). The message is directed first to the incoming mail server, poffice.ahania.com, the destination designated by the MX record for the domain ahania.com.

^[5] To illustrate the various mail redirection possibilities, this example violates many design principles for an effective and efficient email system: a central aliases file, a logical and well-ordered set of DNS MX records, and so on.

Figure 9-3. Mail redirection

The sendmail configuration on poffice specifies that all incoming mail addressed to ahania.com be sent to the same user at host incognito. On incognito, an alias for jane on incognito points to jane@dalton. Finally, on dalton, jane is aliased to jsmith, a local user account. However, user jsmith has a .forward file in her home directory consisting of the entry [e-mail]. So the mail is readdressed accordingly, and the mail system in the ahania.com domain sends the message back out onto the Internet. When the message arrives at the zoas.org domain, the entire process begins again.

9.1.2 Electronic Mail Policies

Electronic mail frequently raises as many social issues as it does technical ones. Part of the system administrator's job is to educate users about using email properly and its risks. Many sites implement an email policy to specify appropriate and inappropriate uses of user email accounts, as well as to inform users about their rights (and limitations to them).

The following is a list of items you should consider including in an educational or policy document about email:

• Reminders to keep email brief and to-the-point and to strive for the same level of politeness one would use in verbal communications (avoid "flaming"). Wait a day or so before replying to or sending an emotionally charged message (in the latter case, having someone else read the message before you send it is also a good idea).

• Limit a message's recipients as much as possible. In other words, be conservative about copying (CCing) to additional people. When replying, send your response only to the message's author or at least remove extraneous recipients from the list. Avoid making Reply All the default in your mailer program.

• Any policy your site has with respect to blind copying (BCCs).

• Any forbidden uses of email at your site: e.g., commercial ventures not related to your organization, chain letters, spam, and so on.

• Any policies you have made with respect to mail attachments (e.g., quarantining ones possibly containing viruses).

• Email should not be considered private, so confidential information should not be sent via email. You should also mention any organizational policy on users' email ownership and privacy (i.e., whether management reserves the right to examine any email message).

• Politeness dictates that email messages not be forwarded to third parties without the original sender's approval.

• Email is not considered legal notification in most instances. Use written communication (a memo or letter) instead of or in addition to email when you need to officially/legally convey information.

• Email can be forged, so trust your instincts about suspicious messages and investigate them before taking any action.

Email Ethics for System Administrators

Anyone with root access on a system can obviously read any file on it, including users' electronic mail. However, system administrators should obviously not do so in general, and they should make every effort to avoid looking at actual messages even when they must debug the mail system. Most times, test messages can be sent to simulate actual mail traffic. Even if you do need to examine real mail messages, examining the mail headers is sufficient in almost all cases.