1.8 General Tuning Ideas

There are some general tuning tips that can improve the performance of email service applications in a broad way. Some are simple, common-sense ideas, but they bear mentioning nonetheless. Note that modern UNIX-like operating systems generally do a much better job of tuning themselves these days than they once did. That fact, coupled with the relatively large amount of RAM one can economically install in a system, has meant that the most common sorts of performance-tuning tricks that are described in many books on the subject are much less effective in bringing about an actual performance increase than they were, say, 15 years ago. These tricks might include removing kernel modules to free up memory or increasing kernel table sizes. This is not to say that one shouldn't take the time to remove unneeded kernel modules or check kernel table sizes, but these optimizations are less likely to produce noticeable performance increases than they once would. Other avenues of attack can probably yield much more substantial gains.

Performance gains via general operating system tuning can still be obtained, however. Documentation on general performance tuning is available for most operating systems from the vendor, on the Internet, or both. Some sources of general information can also prove quite useful, such as the second edition of System Performance Tuning [ML02]. Even if general tuning advice doesn't directly contribute to better email throughput, it often provides insights about what's happening "under the hood" on a server. Gaining a better understanding of how an important system functions is always a good thing.

This book cannot hope to be as thorough as these other sources when it comes to operating system specifics, nor can it cover every system. Instead, the book focuses primarily on the types of hardware that can be used and on ways to improve performance at the application level. The rest of this section examines other methods by which performance improvements may be obtained.

1.8.1 DNS

One area in which significant performance gains can be achieved is in the configuration of DNS services. Any server that handles a large number of connections to and from arbitrary sites on the Internet, such as a Web or email server, will typically perform reverse DNS lookups on the IP addresses from which connections are received to perform authorization checks, properly log the name of the connecting host, and so on. Depending on the software and the level of paranoia in the configuration, once this reverse lookup is done, a forward lookup may also be performed to confirm that the two results match. Many MTAs take this step to make certain that the remote machine to which an email server is talking is really who it says it is. Besides providing identity verification, such checks properly identify the correct machine to which a particular piece of email will be sent. Consequently, a server will likely perform a very large number of DNS lookups, the majority of which will apply to hosts outside the local domain. The processes that make these lookups will sit around waiting for responses to be returned before they proceed. Methods that cut down on the time required to complete these DNS requests will reduce the total duration of an email protocol session, which means that resources are freed up more quickly to handle other tasks.

The most obvious way to reduce the latency of DNS requests on an Internet server is to run a name service daemon on the server itself. For high-volume email servers, this is an excellent idea. One of the most commonly cited reasons why folks resist doing so is because DNS servers are perceived to increase the administrative effort required to maintain a server. It doesn't have to be this way. Busy servers should run their own name daemon and reference it as their first choice in the /etc/resolv.conf file.

To reduce the maintenance costs, the daemon should be of the caching-only variety, and no computer other than the server itself should query its name service daemon. This can be done by administrative fiat or packet filtering; alternatively, using BIND version 8, it can be enforced in the name server configuration. This step is worth taking. The caching-only nature of this name server means that the only zone files on the name server are (1) the cache zone file, which lists the root name servers, and (2) the loopback zone file, which contains host name-to-IP address translations for the loopback network interface, 127.0.0.1. The server does not hold zone files for the local domain. When an SMTP connection is made to a host in the local domain, the name server must query the root name servers to find authoritative sources of DNS information about these hosts. It will need to make this query only once to determine which name servers are authoritative, and then it will need to perform a query on each particular host only once before that information will become locally cached. Thus the fact that this DNS server maintains no explicit knowledge of its own domain information only marginally affects the performance of the first query for a local host, while keeping the number of requests that it must make over the network to a minimum.

A sample named.conf file that lists these two zones and administratively keeps remote hosts from querying it for BIND version 8 follows:

 options {           allow-query { 127.0.0.1; };           allow-transfer { none; };  };  zone "0.0.127.in-addr.arpa" IN {           type master;           file "REV.127.0.0";  };  zone "." IN {           type hint;           file "named.root";  }; 

The appropriate contents for these zone files can be found on any properly running DNS server running BIND.

While it may not seem like a lot of traffic, the DNS load and the latency introduced by making these queries to another machine can be substantial. Consequently, with a caching-only name server that maintains almost no local state, very little increased maintenance overhead arises. Very little can go wrong in this situation.

Instead of running caching-only name servers, some DNS administrators like to run caching-forwarding name servers. They differ from caching-only name servers in two ways:

1. They do not contain a zone for the root name servers, nor do they ever query the root name servers directly.

2. When a server cannot answer a query directly (because the query asks for information that cannot be found in a locally maintained zone or because the information isn't in its cache), the request is forwarded to a predefined server or set of servers for resolution.

With this strategy, if the servers to which the caching server forwards its queries are authoritative for local domains, the extra round-trip request across the Internet for querying the root name servers when the caching server needs to look up DNS information about local hosts does not occur.

As has already been discussed, while it is more expensive to query the root name servers for local domain information, this happens very infrequently (unless the Time to Live [TTL] records on one's own domain are set unusually low for example, less than 5 minutes), so it doesn't cost very much. On the other hand, every request that a caching-forwarder makes for information not in its cache, including the lookups for all remote servers with which it will communicate, must go through an extra hop: the forwarder. If the forwarder is sufficiently fast, is dedicated to this task, and is "near" the caching-forwarder ("near" in a network sense, meaning that sufficient bandwidth exists between the two servers to support their connection, and the latency between these two hosts is very low), then this effort may not cost the email server very much. Whenever possible, however, an email server should take the extra time to look up local zone information by querying the root name servers as caching-only servers than to have them forward every request through another host. Forwarding both slows down the query process and provides additional perhaps even a single points or point of failure.

The primary exception to this general rule arises when an organization's firewall configuration precludes an email server from making direct DNS queries to the root name servers. In this case, a caching-forwarding configuration must suffice, and is a much better option than directly querying the forwarder for all DNS information.

Servers based on sendmail are less susceptible to DNS-induced slowdowns than some other MTAs might be, because each sendmail process caches the DNS information it receives. Note: This cache is maintained on a process-by-process basis; it is not shared between separate instances of sendmail, even between parent and child processes. The results of this functionality are most pronounced when a sendmail process performs a queue run. For each host that has email bound for it in the queue, host canonicalization will be performed exactly once. Starting with sendmail 8.12, the MX record resolution of each host will also be performed exactly once (unless the resource record TTL is exceeded, at which point the request must be repeated). This point is also evident during the processing of sendmail rulesets. As recipient information is processed, sendmail must translate between host name and IP address, often parsing the same email address multiple times. Nonetheless, a DNS lookup on each unique host name or IP address is performed only once per sendmail instance.

In some environments, it would clearly be better for the name daemon running on an email server to be authoritative for its own domain. It's also certainly possible to envision a situation in which an email server should not run a name daemon at all. Even so, in the vast majority of situations, running a caching-only name daemon on busy email servers can improve performance considerably without creating a maintenance headache.

1.8.2 General Server Tuning

The arrangement of the server hardware acts as an important foundation for providing efficient email services. An email administrator will often decide that moving the message store and email queue to their own disks or, if necessary, disk controller, might be a good idea. It's important to understand that the reason for doing so is not only to improve the throughput of email I/O, but also to insulate the email system from things that might go wrong on the rest of the server. That is, if someone fills the / filesystem due to some administrative error, it would be a good thing if that mistake didn't directly affect the flow of email. Dedicating a disk (or at least a disk partition) for the storage of temporary files created by the email server software can be beneficial for the same reasons.

In general, regardless of the system's performance characteristics, I recommend one disk (or more if appropriate) for the general operating system, one disk (or disk system) for the message queue, and, if the server provides a message store for local mailboxes, one disk or disk system for the message store. Even if the message throughput or volume does not require multiple disks, it is worth taking this step on any dedicated server just to have the separation of the disks themselves available to act as a barrier against something going wrong on one part of the server affecting another.

With larger email servers, at some point it becomes prudent to provide a separate disk devoted to swap space, moving these data off the operating system disk, although in almost all cases these two disks will share the same disk controller. Swap areas should never be allocated on disks devoted to the email queue or message store, regardless of available space or apparent short-term need.