6.1 Configuration

This section explicitly focuses on the options available within the sendmail.cf file to see how modification of various parameters can improve throughput in different situations. I've waited until this point to discuss these issues because tuning the system's file I/O is almost always more effective in speeding email server throughput, but some configuration options can result in significant changes.

6.1.1 Timeouts

One set of configuration parameters that can lead to performance speedups involves changes to sendmail's timeouts, although altering some parameters in this area can prove perilous. RFC 2821 specifies some minimum timeouts for MTAs at different phases of an SMTP protocol exchange. While sendmail will allow timeout parameters to be changed to values below the RFC recommendations, this step is rarely advisable. Some of these parameters may seem rather long and archaic on today's Internet. For example, it may not seem necessary to have to wait for 10 minutes to receive a "250 OK" message at the end of a DATA phase. However, these are the rules by which we play on the Internet, and breaking them should not be undertaken lightly or, in almost every case, at all. Besides, most of these timeout situations happen rarely and therefore have relatively little influence on overall performance. If the goal is to improve the total throughput of an email server, it is important to focus on speeding up those situations that arise frequently, as large cumulative improvements can be made there.

Timeout.ident

As noted earlier, operations that result in disk movement or network round trips are likely to be the tall tent poles when it comes to the total amount of time required to process an email message. Therefore, if an opportunity presents itself to remove a network query to a remote host, then that strategy should be seriously considered. A network round trip can be safely eliminated by setting the Timeout.ident variable to 0 from its default, 5s. sendmail waits this amount of time for IDENT information to return from a remote server during an SMTP connection. If the variable is set to zero, the IDENT query is never attempted.

IDENT is a security protocol designed to return information regarding which user owns the client-side socket used in an Internet connection. This protocol is defined in RFC 1413 [JOH93]. It has little use in today's Internet because (1) it is applicable only to email sent by a user logged into a multiuser machine, relatively few of which exist these days, and (2) one cannot trust IDENT information coming from a server administered by someone else.

Most email servers don't run IDENT servers; on those without user accounts, the information IDENT provides is meaningless. For example, on all UNIX-based email gateways or proxies, we would expect the user identified in an IDENT request to always be root, mailnull, daemon, or some other generic system account. Strangely enough, the real delays that sendmail sees due to IDENT problems don't relate to servers that choose not to run IDENT daemons. Rather, they come from servers that reside behind firewalls that don't return a "connection refused" response for protocol requests that aren't supported IDENT is almost always on the "unsupported" list.

Here is how this scenario works. My email server receives an SMTP connection from another server. After the connection is established but before my server returns the "220" greeting message, signaling that I am willing to carry out an SMTP conversation with the connecting server, I send an IDENT request back to the host originating the SMTP session. The originating host's firewall, which sits between the two servers, intercepts this request. The firewall then silently discards the packet. I now wait until the timeout value has expired before allowing the SMTP session to proceed. At best, the firewall displays antisocial behavior. In fact, I would go so far as to say this behavior is flat-out wrong. It's not uncommon on today's Internet, however, and each time a connection is made to a remote site that behaves in this fashion the duration of the SMTP session will be lengthened by the timeout value plus the time it takes for a network round trip. I always set Timeout.ident to 0 on all my email servers:

 define('confTO_IDENT','0') 
Timeout.iconnect

Some sites that send out a great deal of email will set the Timeout.iconnect variable to be lower than allowed in the RFCs, such as 10 seconds, but then provide a FallbackMX host with the same timeout set to an RFC-compliant value, minimally 5 minutes. Section 6.3 discusses the advantages of using a FallbackMX host more thoroughly. With this mechanism, the first attempt to send the message either succeeds or fails very quickly. If it fails because the remote server runs slowly, the message is shunted off for a more leisurely delivery on a server running at a lower load, thereby freeing resources on the "fast" relay to attempt the next delivery. While this method technically violates the RFCs, it does adhere to the spirit of these documents by allowing the remote server to be appropriately slow while minimally penalizing the sending host for this fact. In general, these sorts of optimizations should not be undertaken without careful consideration of all their ramifications.

Queue Processing Timeouts

Another set of timeouts that can modified are Timeout.queuewarn and Timeout.queuereturn. In some cases, messages cannot be sent to any server for some period of time. After Timeout.queuewarn (default of 4 hours) passes, the system generates a DSN warning the sender that the message hasn't yet been successfully delivered, but the server will keep trying. While this result can be convenient, it's just one more piece of email for the server to process, which adds to its overall burden. Indeed, this message often ends up being more confusing than helpful to many email users, especially in an ISP setting where the customers may not have much experience in these matters. In my ISP days, I saw more than one piece of irate email sent to the mailer-daemon demanding that it expedite mail delivery after a warning message was sent back to the user. I've even received an occasional piece of mailer-daemon fan mail, thanking this kind person for trying to help the sender get a message to a family member or friend. Especially in this environment, but also in a legitimate mass-mailing endeavor, increasing Timeout.queuewarn, even to a value higher than Timeout.queuereturn, or setting it to zero is entirely appropriate. However, DSNs signaling that email has been delayed is a feature that may be missed in an environment with more savvy Internet users.

Timeout.queuereturn denotes the amount of time that a message will remain queued before being bounced as undeliverable. The default sendmail value for this parameter is 5 days. Each truly undeliverable piece of email represents one extra item in the queue that is processed with each queue run. Each time it is processed, it slows down the processing of the rest of the queue, because the connection to the remote server can't be made. Therefore, decreasing this number may be appropriate. I would rarely recommend shortening this period to less than 3 days (the amount of time a corporate mail server might be down if it fails on Friday and service is restored Monday morning). This value doesn't account for the fact that this outage would be noticed and someone would scream long before the weekend concluded at most places, nor does it account for longer weekends that include vacation days. Of course, one can also mitigate this problem in a straightforward manner by rotating queues more frequently than Timeout.queuereturn for example, daily.

Good information on SMTP timeouts appears in the Sendmail Installation and Operation Guide [ASA] and RFC 2821 [KLE01]. The reader is strongly advised to peruse the pertinent information in both of these documents before altering queue processing timeouts.

Resolver Timeouts

As has already been discussed, the nature of sendmail's interaction with DNS services can have a significant effect on an email server's performance characteristics. Few Internet services provide an interface to alter the resolver library's behavior in response to differing network or server circumstances. Parameters such as timeouts are usually hard-coded into the resolver library itself; even in cases where applications may override these numbers, few elect to do so. Because the interactions between DNS and sendmail are so important, beginning with version 8.10, it became possible to alter some of the resolver's behavior within the sendmail.cf file.

Because DNS requests typically take the form of UDP packets, the only indication to a DNS client that a server received its request is a response. It's generally impossible to know how to interpret the lack of a timely reply. It might mean that the name daemon is not running on the remote host, that the server itself is down or overloaded, or that the server is up and operating but the specific information requested isn't immediately available. Typically, a resolver library will try to make a DNS query some small number of times two to five attempts is typical and will space these requests in time. Every 5 seconds is a default value present in some recent versions of BIND, although much longer timeouts, often exceeding 60 seconds, have historically been commonplace on some versions of UNIX.

The Resolver.retry.* parameters control how many different queries will be made of each name server listed in the /etc/resolv.conf file, usually with a maximum of three name servers listed. The Resolver.retrans.* parameters control the interval between retransmissions. In sendmail, the default for all Resolver.retry.* parameters is four attempts, and the default for all Resolver.retrans.* parameters is 5 seconds. More fine-grained control of these parameters is available. Each comes in a Resolv.*.first and Resolv.*.normal variety, as well as the default Resolver.{resolv,retrans} mechanism. The distinction between "first" and "normal" refers to the message delivery attempt, not to the particular DNS request attempt.

On a typically well-configured email server, the resolv.conf file will probably list two name servers. The first will be 127.0.0.1, referring to the locally running caching-only name server, and the second will be a dedicated name server with an interface on the local network, on the off chance that the local name server stops responding for some unforeseen reason. If the locally running name server stops responding, we expect that condition to be detected in very short order by locally running automated processes and that fact to be reported to a human being very quickly. In the usual case, we expect the local name server to respond to the first attempt, and to respond very quickly. If IP packets are getting lost on the loopback interface, then the server has bigger problems than slow DNS queries. Even for the locally attached "backup" DNS server, the server should generally respond to the first request, or at least one of the first two, so it's probably acceptable to lower Resolv.retry to 2. While this change may yield a slight performance improvement, it probably won't make a noticeable performance difference under most circumstances, so no harm arises from leaving the default values intact.

As soon as they have an answer, both the name server on the loopback interface and the backup name server on the local network will respond to DNS requests very quickly. Even if the local name server is running on a machine a few network hops away, the bulk of the time spent waiting for a general DNS resolver request would likely relate to the query across the Internet from the name server to the root name servers and then the appropriate top-level name servers, not to any local data connection. Just because especially fast name servers are available, it doesn't mean that decreasing the Resolv.retrans value is a good idea. On today's Internet, 5 seconds is a pretty reasonable yet conservative period in which to expect a DNS response. Note that if the value of Resolv.retry is cut from 4 to 2, it will halve the maximum amount of time a given sendmail process will wait on DNS requests for a domain that is not responding from 20 seconds to 10 seconds. Decreasing this value, which is fairly safe, will have a more beneficial effect than trying to trim the Resolv.retrans interval. Similarly, if all DNS servers listed in /etc/resolv.conf are fairly reliable, and they should be, dropping the number listed from 3 to 2 will decrease the amount of time spent waiting for an answer from name servers that cannot be contacted by 33% without measurably affecting the service's reliability. Just because three (or more) name servers can be listed in the resolv.conf file doesn't mean that this choice is a good idea. Also, remember that this performance tuning will be meaningful only if the local name server does not have the information in its cache.

Much like the Timeout.iconnect example given earlier, some mass-mailing servers may wish to cut the Resolv.*.first parameters to lower the latency imposed on the average injector, figuring that most of the DNS responses will come very quickly. For example, the following lines added to a .mc file might be quite reasonable under these circumstances:

 define('confTO_RESOLVER_RETRY_FIRST','2')  define('confTO_RESOLVER_RETRANS_FIRST','2s') 

Thus a message bound for a recipient whose MX record cannot be resolved quickly would remain in the queue. Queue runners, no longer attempting to make the first delivery, would use the default values during redelivery, in case the name servers are up but responding slowly. If the system uses a FallbackMX host, then the mass-mailing machine should never attempt a redelivery, so it might seem tempting to set these values for all situations, rather than just for the first attempt. Of course, the FallbackMX host might not respond all the time, so some reasonably polite behavior during redelivery attempts by the mass-mailing host is appropriate.

On a strategic note, while these tuning parameters may speed up email delivery, one technique may help even more, especially in the mass-mailing case: priming the DNS cache before sending the email. In fact, during less busy times, running a script on the email server that will do MX lookups on each domain for a site's large client or mailing list say, once a day can really improve performance. It will guarantee that for every site whose DNS service remains up and that uses TTLs of no less than 24 hours, the DNS information will always remain in cache, and no queries over the Internet during the mass-mailing phase need to be made. For those servers receiving email, this strategy will prove less productive.

Note that the *.normal and *.first parameters are consulted only when sending email. The Resolv.retrans and Resolv.retry parameters affect all DNS lookups.

6.1.2 Load Limits

Few things are more annoying for an email administrator than dealing with a mail bomb. With a mail bomb, some user or server sends an inordinate amount of email, often very large messages, to a single user to create a nuisance. If the amount or rate of email is large enough, and adequate safeguards are lacking, a mail bomb can progress from being merely obnoxious to becoming a denial-of-service attack against the entire server.

One can try to mitigate this possibility by setting MaxMessageSize so that sendmail will automatically reject messages over a certain size limit. Some (well-constructed) MTAs will specify the message size during the envelope exchange so that the rejection may occur before any resources have been consumed in transmission, which is a good thing. The following is an example of such an ESMTP exchange. Lines beginning with "S:" indicate the protocol provided by the server, and those beginning with "C:" indicate the client's portion of the exchange. Lines that begin with neither of these letters are continuations of the previous line.

 S: 220 discovery.gangofone.com ESMTP Sendmail 8.12.2/8.12.2;     Tue, 9 Feb 2002 17:59:55 -0800 (PST)  C: EHLO streaker.gangofone.com  S: 250-discovery.gangofone.com Hello streaker.gangofone.com,     pleased to meet you  S: 250-ENHANCEDSTATUSCODES  S: 250-PIPELINING  S: 250-8BITMIME  S: 250-SIZE 1000000  S: 250-DSN  S: 250-ETRN  S: 250-STARTTLS  S: 250-DELIVERBY  S: 250 HELP  C: MAIL FROM:<npc@streaker.gangofone.com> SIZE=5000000  S: 552 5.2.3 Message size exceeds fixed maximum message     size (1000000) 

In the exchange, the client suggested it would senda5million byte message, and the server rejected the message as too large. Details of this SMTP extension appear in RFC 1870 [KFM95].

With a pest or an older MTA that doesn't support the SIZE message extension, a server will have to accept at least part of the message before discarding it, but resources may still be saved. The problem is that people have become used to sending the most inappropriate things via email, to the extent that 100MB messages are not out of the ordinary at many sites, so this tactic has limited effectiveness. Many ISPs limit messages they handle to 10MB or smaller. This decision can reduce some of the more egregious offenses, and every little bit helps.

It is also possible to set these parameters for different mailers using the SMTP_MAILER_MAX, LOCAL_MAILER_MAX, UUCP_MAILER_MAX, or other configuration parameters. In this way, one could allow the relay of relatively large messages through a machine without accepting them for local delivery:

 define('SMTP_MAILER_MAX','10000000')  define('LOCAL_MAILER_MAX','1000000') 

In the .cf file, this task is accomplished by specifying the message size using the M= parameter of the appropriate mailer definition.

Some sendmail configuration parameters are based on the nebulous concept of the system load average. Explaining exactly what this average entails would be complex and outside the scope of this book, but roughly it consists of an average count over some time period of the number of runable jobs. On some operating systems, "jobs" means processes; on others, it means threads of execution. If the value of the load average exceeds the number of processors in a system, then some jobs are unable to run because they are waiting for some resource to become available. These resources might include waiting for a CPU time slice or waiting for access to a special device file, such as a tape drive. Performance shortcomings on an email server are rarely well represented by this metric.

One thing that doesn't count toward load average on most systems is a process that's blocked waiting for file I/O, something that happens quite frequently on a busy email server. Under most operating systems, the load average generally sky-rockets on a dedicated email server only if a system becomes so busy that process tables fill or it runs out of memory, or if something else runs into problems on the machine that either shouldn't be running or shouldn't be having problems. In the latter case, the solution is clear. In the former case, a high load average indicates that something is wrong long after the situation has reached a dire state. Because load average is so rarely useful for measuring the business of an email server, altering sendmail's behavior based on these parameters typically provides fewer benefits than we might hope. Nonetheless, sendmail configuration options such as QueueLA and RefuseLA deserve mention here.

With Linux, jobs that are blocked waiting for I/O do count in the load average. While load averages rarely reach very high numbers on operating systems such as Solaris or FreeBSD, busy email servers using Linux can have load averages that are quite high in fairly typical use, and exceptionally high when under heavy load. Therefore, when setting sendmail configuration parameters based on load average, don't assume that they can be cavalierly transferred between Linux and non-Linux systems they cannot. Even between different Linux systems what constitutes a high load average will depend greatly on the specifics of the filesystem and the storage system in use. In general, I recommend setting these parameters very high, perhaps in the hundreds, and lowering them only if the system's total throughput demonstrably decreases as the load average increases.

The RefuseLA variable indicates the load average at which the MTA will stop accepting new connections via SMTP. Once the load average drops below this mark, new connections will be permitted. Because this event usually indicates an email server that simply cannot keep up with demand, having this machine stop responding for a period of time is not a long-term solution. It generally creates a greater pent-up demand by having email that has not yet been delivered back up across the Internet. Therefore, while traffic remains high, once the server's load average dips below the RefuseLA watermark, a flood of connections will come in, sending the load average skyrocketing. This, in turn, will cause the MTA to refuse connections again for some time. Unfortunately, changing the email server's configuration parameter will not diffuse the situation very much. The solution is to improve the throughput of the email server.

As the amount of email handled by a server increases, the total throughput (in bytes per second or messages per second) will increase in tandem. This situation continues until the server reaches its saturation point, where throughput is maximized. As demand increases beyond this threshold, total throughput will begin to decline. Depending on how large the load is and which operating system is running, the load average may then increase substantially. A good number for RefuseLA would be about half of the load average value at the point where total throughput begins to decline precipitously. Generally, one might be surprised at how high a number is appropriate here. Depending on the server's operating system and my direct experience with that particular load profile, I'll typically set this variable in a range between 12 and 20 on a dedicated non-Linux email server, and higher on a Linux server. Note, however, that the appropriate cutoff number will depend heavily on the resource being starved on the machine, so it's impossible to give hard and fast guidelines. One scenario where this variable does come in handy is if a busy email server near its capacity goes offline for whatever reason (network outage, hardware repair, power outage, and so on). It likely will get slammed with deferred connections when it first comes back up, but will eventually work its way out of the jam. An appropriately chosen RefuseLA parameter may help it do so in a minimum amount of time.

It is impossible to build a server that accepts connections from the Internet that cannot be pushed to saturation by external load. No matter how powerful an email service one builds, it can be brought to its knees by the cooperating horsepower of even a small fraction of "the rest of the Internet." Every significantly busy email service on the planet has been pushed beyond its saturation threshold at some point. Most of this book has discussed techniques that will raise the bar for this threshold, but if a high-capacity email server never reaches the saturation point, it probably has been overbuilt. As annoying as these situations can be, they're a fact of life on today's Internet.

Besides total capacity, another important characteristic of a well-designed, high-performance email server is its response to load saturation. If the demand for a service temporarily increases beyond the server's ability to respond to it, only minimal degradation in total throughput should occur. It's not sufficient to build an email gateway that handles 10 Mbps of email service gracefully, but crashes when the load reaches 15 Mbps. It should be expected that total throughput would be reduced by a few percentage points even 10% or 20% is probably acceptable when demand significantly exceeds capacity. If the server becomes so heavily loaded that it ceases to do work effectively, however, it may end up in a state from which it cannot recover. This outcome represents an unacceptably fragile situation.

The solutions adopted on an email server that risks occasionally running in a state where its throughput capability becomes saturated need to respond gracefully to overload. For example, even if the queue's filesystem doesn't need to support a hashed directory structure to handle the expected maximum load, having this capability is beneficial because it responds more gracefully to overload than a filesystem with a linear directory lookup. For the same reason, the system should have more RAM than one anticipates needing. Extra memory can act as a buffer against several types of resource shortages. It may be useful to perform a periodic queue rotation even if we don't expect this directory to grow very large under normal circumstances. Having more CPU horsepower than one expects to need can occasionally come in handy, as can having an additional unused disk or two on a server, which can be allocated toward swap space or used for an extra queue in case of emergency. Skilled email administrators don't just consider how their server will behave when demand is reasonable and within predicted values, but imagine what life might be like if demand exceeds available bandwidth. The best indicators of preparation are if a situation has the capacity to become a catastrophe yet turns out to be a mere disaster, or if a potential disaster ends up as a mere annoyance. Minimizing such damage rarely grabs headlines or results in awards, but these victories are very real when it comes to evaluating the bottom line.

The partner of RefuseLA is QueueLA, the load average at which an email server will simply queue incoming messages rather than trying to deliver them. On almost all busy email servers, it is an extremely dangerous parameter, and I strongly advise that it be set to a number not merely larger, but much larger, than RefuseLA. That is, a busy email server never wants to accept email and just queue it. When we consider that I/O to and from the queue constitutes one of the scarcest resources on a typical email server, and as the queue grows the server tends to run more slowly, it becomes clear why going into queue-only mode can have disastrous consequences. It will cause the queue to swell, reducing capacity in the one place most likely to cause the system to run slowly in the first place! The reader has been warned.

The last load average variable worth considering is DelayLA, which was introduced in sendmail 8.12. Exceeding this threshold introduces one-second delays into the SMTP session. Clearly, this variable should be set lower than RefuseLA to slow down the rate of incoming email without eliminating the flow entirely. In most situations, this tactic will not be very effective. If it's set, however, a value of about half of RefuseLA is probably a reasonable starting point. Setting this parameter tends to offer more benefits in situations where the email server receives messages from an SMTP injector or other fixed number of machines and bulk-mails them out to the Internet. This technique can help slow down a group of synchronously operating injectors if they're driving the senders faster than they can optimally operate.

On a machine receiving email from the Internet, this parameter proves less useful. Suppose an email server that receives email messages from the Internet has exceeded its DelayLA load average threshold and is now slowing down incoming connections. The rate at which the Internet will initiate new SMTP sessions won't be significantly affected by the fact that each individual SMTP session runs slowly. Instead of decreasing the rate at which a few machines send email to the overloaded server, connections will be started at the same rate, but each will take longer to complete. This result doesn't help the server to deliver email any faster.

If DelayLA is set, as long as the load average exceeds the parameter's threshold, a one-second delay will be introduced before any SMTP command is accepted and before a sendmail process performs an accept()call on an incoming connection. Therefore, when DelayLA is in effect, connections could potentially back up in the server's listen() queue. If DelayLA is set, it would probably be a very good idea to increase the depth of the listen() queue in the system's kernel. Generally, DelayLA should be viewed as a parameter that can help smooth out a transient surge in load over a short period of time, perhaps measured in minutes. It will not help an email server very much if it remains in a saturated state for hours.

DelayLA has an interesting side effect: If it is set, the system's load average is checked far more frequently than if just other *LA parameters are defined. Therefore, if the load average for RefuseLA is not checked as frequently as one might like, setting DelayLA to a large number might help mitigate this problem. On most contemporary operating systems, load average checking involves a very inexpensive lookup into a kernel data table. On an operating system where calculating the load average consumes a significant amount of system resources, it may not be advisable to use DelayLA.

SMTP Connection Cache

When looking through the copious list of features provided by sendmail, two leap out as possible sources for tuning: ConnectionCacheSize and ConnectionCacheTimeout. These parameters control the number of outgoing SMTP connections that a process can hold open at once. The dictum that "more is better" for performance is certainly true, but it's not nearly as big a help as one might think.

The reason for this small benefit is that these connections are considered on a per-process basis. They cannot be pooled among sendmail processes. Thus, if a process is spawned to relay email to a busy site such as aol.com, there's no sense in trying to hold this connection open. After all, the process will finish and exit after the SMTP session, and no other delivery will be able to take advantage of this open pipeline. This parameter rarely comes into play except when running the queue. Even then, however, the effect will be minimal if the queue is sorted by destination host. (The effect isn't zero because some messages may be bound to multiple recipients, so a later recipient on the list may take advantage of a cached connection. Nonetheless, as one would expect, this gain is a marginal win at best.) Generally, it would be rare to see a measurable performance enhancement from setting this parameter higher than "2" on most servers, unless a high percentage of the email traffic goes to each of several domains, and most messages handled by the server are destined for multiple recipients or the queue is not sorted by destination host. The Sendmail Configuration and Operation Guide gives fairly dire warnings about setting this value too high. These warnings are probably overwrought, but it's best to keep the value fairly low unless compelling reasons exist to think that a higher value would help significantly.

Some Other Parameters

Some other parameters are designed to help prevent an email server from failing under the weight of its own processes. MaxDaemonChildren, for example, limits the maximum number of sendmail processes that can be concurrently active that have been spawned by the master daemon process listening on TCP port 25. If MaxDaemonChildren is set and this threshold is reached, it does not prevent more sendmail processes from being spawned from the command line for the purposes of message submission or running the queue. The master sendmail process has no mechanism by which to detect these other processes. Instead, it simply imposes limitations on the number that it spawns itself.

The MaxQueueRunSize parameter is normally not set by default. Once a queue runner sorts the queue, it will attempt to deliver only as many messages as given by the value of this parameter. For busy and clogged queues, messages at the end of a queue that could be delivered may not be, because the first N messages cannot currently be delivered. Therefore, this parameter rarely proves useful in practice. When it is used, it is likely to be less detrimental when used in conjunction with the "random" queue sorting strategy. Queue sorting strategies are discussed in more depth in Section 6.1.3.

The ConnectionRateThrottle parameter controls the maximum number of new SMTP sessions that the master sendmail daemon will accept per second. Once a good threshold is known, setting this parameter may help smooth out the load on a mass-mailing server where the messages to be delivered are injected via SMTP. On other mass-mailing servers, this parameter will likely have no beneficial effect. It is also one of the few parameters that can help a server that receives email from the network smooth out some of the rough spots in message delivery without causing more trouble than it eliminates. Make certain before setting this parameter that it is set to a number that the server cannot handle on a medium-term basis. If it is set too low, ConnectionRateThrottle will restrict the server's ability to handle email.

In conclusion, sendmail supports numerous parameters intended to help it from getting overloaded. On busy, well-tuned email servers, setting many of these parameters is as likely to hurt performance as help, however. In general, I recommend setting very few of these parameters, or at least setting them to values that are known to be absurdly high, unless direct evidence from multiple incidents indicates otherwise.

6.1.3 Queue Operations

Much of this book has focused on tuning the queue, and for good reason. Besides speeding up I/O access by focusing on the filesystem and storage, some parameters within sendmail can speed up queue runs and vary the exertion required to process the queue.

When we start sendmail with a command line such as sendmail -bd -q1h, the first option tells the process to act as the master daemon and to run in the background. The second option tells the master daemon to spawn a process on start-up and again once every hour to run the queue. When a queue runner starts, it first sorts the messages in the queue, then sequentially tries to process and deliver them. Once it has processed all messages, it will exit, unless the system uses the sendmail 8.12 features allowing persistent queue runners. Many strategies are possible that may be employed in running the queue more efficiently.

The default QueueSortOrder strategy is by "priority," which means that the queue runner inspects the contents of every qf file for the P parameter, sorts all messages by this value, and attempts sequential delivery of each queue entry. The "priority" is partly based on a user-settable parameter specifying how important the user thinks the message is and partly based on how long the message has remained undeliverable. In practice, it means that the queue runner will try to deliver messages first that are not marked as "bulk" and those that haven't sat in the queue for a long time. This approach is generally desirable, but it does require that all qf files in the queue be read prior to the attempt to deliver the first message. If the queue is already a resource bottleneck, this effort can take a long time. In the real world, I have seen a sendmail queue runner take more than an hour to scan a queue directory before it opened its first SMTP connection, although these sorts of situations are rare. On email servers where the queue doesn't tend to get very large, "priority" is an excellent queue sorting algorithm.

If all messages in a queue will be processed anyway, one might adopt the strategy of trying to consume the least network overhead in attempting their delivery. In this case, a queue sort order of "host" represents the best strategy. This way, all messages (modulo those with multiple recipients) bound for the same host will be sent at the same time, minimizing the setup and teardown of TCP connections. If network bandwidth is precious, queue I/O bandwidth is relatively plentiful, and the goal is to send messages as quickly as possible, this tactic is an excellent strategy.

If queue I/O bandwidth is expensive compared to network bandwidth, then the queue sort order of "file" might be a good strategy. This method sorts the qf files based on their file name and starts a delivery attempt on the first one. Using this method, the qf files do not need to be opened and read before delivery is attempted; instead, just a listing of the files in the directory is obtained. This method has a much lower I/O overhead, especially on start-up, than the methods previously described. If all messages are of roughly the same priority, and especially if the queued entries are already sorted by host, then a more sophisticated sorting algorithm offers no special advantages. This algorithm has the shortest start time before the first message delivery is attempted in general, and it works very well if the queue is characterized by heavy I/O contention.

Sometimes, perhaps due to network outage or other unfortunate circumstances, we might find ourselves with a deep queue of messages, most of which we expect to be deliverable on the first attempt. If we start several queue runners with any one of the above algorithms, each will proceed to process the queue in the same order. This operation won't be as efficient as we might like, because all queue runners will try to lock the same set of messages at the same time. For this reason, sites often start one queue runner with the sort order of "priority," one with "host," and one with "file," but that number might not be as many queue runners as one would like under some circumstances. Version 8.12 added the "modification" sort order, which sorts by modification time starting with older entries first, but the total number of alternative sorting methods remains rather small. Mitigating this limitation is the reason for the "random" queue sort order. It takes the file names of the qf files, shuffles them, and then processes them. Therefore, any number of queue runners will be statistically likely to be working on different parts of the queue at any one time while starting up almost as fast as the "file" strategy (the computational cost of shuffling file names is negligible on today's computers, even for queues with thousands of entries).

Generally, for single queue runners on small queues, either the "priority" or "host" strategy is probably the most effective. If queues become large or queue I/O bandwidth grows scarce, then "file" or "random" is probably the best strategy to adopt, with "random" being particularly effective if one wants to start several queue runners at the same time.

If a large number of queue runners are trying to process a single queue, and they're reading through each qf file to sort the deliveries, this approach can cause a great deal of I/O contention. Also, sorting the queue might take so much time that another queue runner will be spawned before the previous one starts its first delivery attempt. Under these circumstances, it might be appropriate to limit the number of processes that can run the queue at any time. MaxQueueChildren and the related MaxRunnersPerQueue are both designed to help control this situation. MaxQueueChildren controls the maximum number of queue runners that the master sendmail daemon will allow to exist at any time. Note that it does not track queue runners spawned by hand (sendmail -q or the equivalent), just those that would be periodically spawned by the master process don't expect to see cooper-ation in this regard among several sendmail processes started from the command line at different times. In version 8.12, if queue groups and MaxQueueChildren are defined (by default, the latter is not), then the default value for MaxRunnersPerQueue is set to 1. If one has multiple queues, it's important to set this value appropriately. Typically, one would always want at least one queue runner per queue, and probably allowing for several is appropriate, especially if relatively few queues exist. Until the choice is demonstrated to be either insufficient or detrimental, I'd generally start with a value of 5 for MaxRunnersPerQueue if I had at least five queues, and a much larger value if I had fewer queues, or one primary queue that received more activity than the rest, if I felt the need to set this value at all. If MaxQueueChildren is smaller than the number of queues, then each will be processed in round-robin fashion.

Using the queue groups feature, one can create more complex sorting strategies. For example, to split off all email headed for the busy domain hotmail.com, add the following M4 lines:

 FEATURE('queuegroup')  QUEUE_GROUP('hotmail', 'P=/var/spool/mqueue/hotmail, F=f') 

The following line is then added to the access table:

 QGRP:hotmail.com       hotmail 

Now, messages whose first recipient on the recipient list is bound for this domain will be queued in a special area.

At first glance, even more complex strategies seem appealing. However, further consideration will reveal that moving too far down this path leads to madness. While notions such as one queue per domain or hierarchical queues might seem intriguing, how do they affect the number of queue runners per queue? How does one implement the mailq command? How can a person track what's going on in all parts of the "queue" without sophisticated tools? The use of multiple queues to increase parallelism and queue rotation are excellent strategies that busy sites should adopt. Sites with a disproportionate amount of traffic going to a single domain or sites with a large volume of mailing list and non mailing list traffic may want to use queue groups to help minimize how performance problems in one area affect another area, but increasing complexity in one's queue structure to obtain a marginal gain in performance probably is not advisable. Keep a server's queue structure as simple as possible.

6.1.4 Ruleset Tuning

An unfortunate number of email administrators attempt to speed up sendmail operation by streamlining ruleset processing in the sendmail.cf file. Almost without exception, this tactic proves to be a serious mistake. Although many of the rules in the sendmail.cf file deal with very uncommon situations, some of them can prevent serious problems from happening if these rare events do occur. In any event, the amount of CPU time it takes to process a full-blown set of rules isn't significantly greater than the time needed to process those in a stripped-down sendmail.cf file. Also, as we've already seen, it's usually unlikely that CPU load will cause the system bottleneck in any case. Do not try to streamline the sendmail rulesets, and especially don't do so by hand-editing the sendmail.cf file.

Some configuration variations, however, can be tuned from M4, which may enhance performance, and they certainly would both be more effective and reduce the chance of serious negative side effects. Of course, each of these possibilities has its downsides, which should be carefully considered.

The "nocanonify" feature can be enabled by adding

 FEATURE('nocanonify') 

to the .mc file. If this feature is enabled, sendmail assumes that the email addresses it is passed are already canonical and, therefore, do not need to be expanded. This technique saves some effort, but it means that anyone sending email through that server to or from the address npc@discovery, for example, will not have that host name expanded to include the fully qualified domain name. As a consequence, that address won't be resolvable (at least not correctly) by other hosts on the Internet. For this reason, this feature is a good idea only on those hosts where all host names processed by a server will be canonical, such as on an email gateway where no one sends email from the command line or using on-server MUAs such as elm or mutt.

As has already been discussed, reducing internal computation, unless it's egregious, doesn't generally have a significant performance effect on email server operation. However, reducing unnecessary connections over the network, or even to other processes, can significantly boost performance. To limit spam, on each SMTP connection the envelope sender email address is checked to make sure it is plausible. That is, the addresses must represent valid domains, such that email returned to the sender might actually be delivered to a real mailbox. What would be the point of receiving email from a sender named nobody@bite.me? Performing these checks necessitates a DNS lookup, which requires a network connection to a root name server at least. Adding both FEATURE(accept_unresolvable_domains) and FEATURE(nocanonify) to the .mc file suppresses this check. As we've already seen, it is not appropriate to set FEATURE(nocanonify) on all email servers. If it cannot be set, then the only reason to set FEATURE(accept_unresolvable_domains) is if it is necessary to accept email from domains that aren't in DNS. Frankly, this case means that a site's DNS is broken, and the problem should be fixed there. Of course, identifying spam early and refusing the connection could bring a performance savings. However, largely because of this sanity check that sendmail currently performs by default, it's rare for spammers to send email that lacks a sane return address, even if it's not theirs. I would generally recommend not enabling these two features in concert. On a server that handles email only from trusted hosts, such as a mass-mailing gateway, it probably would be appropriate to do so.

6.1.5 Table Lookups

The sendmail MTA provides an astounding number of options for relaying or rewriting both sender and recipient email addresses, through mechanisms such as the virtusertable, genericstable, and mailertable. The CPU-bound test server introduced in Chapter 1 must be set up in an extreme configuration for these map checks to produce a clearly measurable effect on throughput. If the machine is configured to use both interactive delivery mode and queueing to avoid queue writes, and it is set to deliver each message to /dev/null, then adding the checks associated with each new map will reduce the message discard rate by 1% to 2%. Therefore, we can conclude that most of these table lookups occur rapidly, and working hard to strip them is probably not necessary. If a table lookup isn't used, there's no reason to express that feature in a running configuration file.

An exception involves the access database. Adding

 FEATURE('access_db') 

to the sendmail.cf file and creating an access database causes throughput using the previously described configuration to drop by about 12% due to the number of access database lookups done in the rulesets. Of course, this example involves a CPU-bound server that never performs operations on any files. Something like this almost certainly won't be deployed in any real operation. The access database is invaluable as an anti-spam mechanism, but performance-sensitive systems should use it only if absolutely necessary.

Another configuration recommendation relates to .forward files. On many email servers, users do not have real accounts on the machine, in which case .forward files will never be consulted. Thus the forward file search path, ForwardPath, should be null:

 define('confFORWARD_PATH', '') 

Further, by default, sendmail supports several .forward file naming conventions (and sometimes locations). The range of possibilities should be limited as much as is appropriate for that server.

The hoststat command can provide a lot of useful information about the state of the email system. It includes host-by-host information on volume, connection time, number of messages, and more transferred between those hosts and the local server. To enable the command, add the following line to an .mc file:

 define('confHOST_STATUS_DIRECTORY', '/var/run/.hoststat') 

This information needs to be stored on the server to be accessible to the hoststat program, which means more disk writes. The writes are asynchronous and small, but plentiful at least one write for each delivery attempt. If this information is stored, putting it on a different disk than the queue or spool is probably a good idea, and putting it on a tmpfs or other memory-based filesystem is probably a better one. The downside to the latter approach is that these files aren't persistent across a machine crash, although that possibility may not be a concern. Of course, if this information isn't used at all, then support for this option should be disabled.

In contrast, the amount of data written to the statistics file for retrieval by the mailstats program is very small and shouldn't create a performance concern. The statistics file can be cleaned out by running mailstats -p > /dev/null. Future versions of sendmail will probably support an option to zero out the file without creating output. The output of mailstats is a good quick-and-dirty method for summarizing email throughput over a specified period without having to sort through the log files. Of course, the logs contain a lot of information that isn't available to mailstat. Both are valuable. A busy server might want to periodically (perhaps daily) run a script like the following to keep a running total of the statistics available to mailstat:

 #!/bin/sh  # mailstat_archive.sh  ARCHIVE_DIR=/archive/email/  DATE='date +%Y%m'  mkdir -p $ARCHIVE_DIR/$DATE  /usr/sbin/mailstats > $ARCHIVE_DIR/$DATE/mailstats.'date +%d'  /usr/sbin/mailstats -p > /dev/null 

Many sites now use LDAP (Lightweight Directory Access Protocol) databases as a centralized site-wide information repository for aliases and authentication information, among other data. The appeal of such an approach is very compelling, and the last few releases of sendmail have included ever-increasing support for storing and retrieving LDAP data. However, each query of an LDAP database involves a network round trip. Despite largely living up to the billing of the first letter in the abbreviation, LDAP lookups simply aren't as fast as lookups in a local Berkeley DB hash table. Therefore, if an email system struggles while making these queries over the network, some benefits might be realized by moving the data closer to where it will be accessed.

Of course, centralizing user data has a considerable following for valid reasons: It's easy to maintain, it's convenient, and it minimizes cross-database "drift." These considerations can prove quite beneficial. Ideally, we could maintain this central-ization without the expense of traversing the network every time a request needs to be made. One possibility is to install an LDAP replica, or a server replicating just the necessary subset of information, on a local network to speed up accesses and make them more reliable. Some replication is almost certainly necessary just from a reliability standpoint. If the LDAP directory is fairly small, one might even consider running the replica on the email server itself, much as this book recommends for DNS name servers.

If closeness isn't sufficient, then it may be appropriate to use something like the OpenLDAP [OPE] ldapsearch command to dump out the subset of interesting data and entries. This information may be formed into appropriate sendmail flat files using Perl or UNIX utilities such as sed and awk. We can then run makemap or newaliases to generate fast, locally accessible snapshots of the information in the LDAP database. Of course, this information wouldn't be as current, but if that consideration is not strictly necessary, performing this action in an automated fashion say, once daily might suffice for many organizations and could reduce the load on an email server significantly. Of course, dealing with these issues necessitates a balancing act, and the factors that go into designing a solution for a particular environment are so numerous that all cannot be listed here.

Some organizations have added similar hooks to sendmail to support their own home-grown databases. If these alterations take the form of code changes to sendmail itself, then maintaining these changes against continual releases of sendmail represents a serious endeavor, not to be undertaken lightly. In some situations, this is a requirement doing periodic data dumps of the variety mentioned earlier will not suffice.

Some of these modifications include connections to SQL databases or connections to other data repositories. Regardless of how the data are stored, some general rules apply in designing these performance-enhancing interfaces. Let us assume that the equivalent of an organization's virtusertable resides in a SQL-based RDBMS somewhere, and the high-volume corporate email relay needs to access it. A site might want to provide a nearby replica of those data. If this strategy isn't possible, and the site's security architecture doesn't preclude it, perhaps giving the database server an extra network interface on a network shared by the gateway would be acceptable. It might even be worthwhile to create a network dedicated to this purpose on which only the data repository and email server reside.

SQL connections are not lightweight. If a sendmail process (and the database) must set up and tear down a session each time it wants to do a lookup, the email server will run slowly. A better design would have sendmail contact via IPC one or more dedicated proxy processes running on the email server, then have these processes maintain one or more persistent connections open to the database. This isn't a trivial programming task. Properly done, one has to communicate asynchronously with the database and several sendmail processes and keep track of and schedule all these sets of transactions. Fortunately, this problem has been solved many times before, and coding this sort of proxy should be straightforward. The question of whether to add a caching function to this daemon is left to the discretion of a programmer who has already gotten the proxy to work correctly.



sendmail Performance Tuning
sendmail Performance Tuning
ISBN: 0321115708
EAN: 2147483647
Year: 2005
Pages: 67

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net