11.3 Locating the Remote Mail Host

qmail-remote can identify the remote host for a message in two ways. If the smtproutes control file has an entry for the recipient domain, that entry determines the remote host, and qmail-remote pretends it found a single MX record for that host with distance zero and makes a list of the IP addresses for that host. The list usually has one entry, unless the host has multiple IP addresses.

Failing that, the usual way is through DNS. First, it looks up the hostname and retrieves any MX records, randomizing the order of multiple MX records with the same distance, then finds the IP addresses for each of the MX hosts.

Once it has the list of IP addresses, DNS goes down the list, starting at the lowest distance, trying to contact each host. Once it finds a host that answers, that's the host used for the SMTP delivery. (This description is slightly oversimplified; the omitted details are covered shortly.)

11.3.1 The smtproutes File for Outbound Mail

It's sometimes useful to override MX data with explicit routes for particular domains. The smtproutes control file consists of a list of two- or three-field lines, with the fields separated by colons. The first field is the domain to route, the second is the name or IP address of the host to which to deliver mail for that host, and the optional third field is the port to contact on the delivery host, defaulting to port 25.

The three primary uses for smtproutes are to override MX data that's known to be wrong, or at least suboptimal, to route mail to private pseudo-domains, and to send outgoing mail to a smarthost. The first situation occurs if a domain has several equal-distance MX hosts, one of which accepts SMTP connections but doesn't actually accept mail on those connections. An smtproutes entry forces mail to a host that's working.

Within a local network, it can often be useful to have private pseudo-domains for special applications. For example, I set up a mail-to-news gateway on my news host in the pseudo-domain news so that mail addressed to comp.whatever@news is posted to the appropriate newsgroup. The news gateway isn't accessible to outside users and doesn't appear in the DNS, so I use entries in smtproutes on other hosts to route the news pseudo-domain to the gateway machine. (The gateway's rcpthosts doesn't list news, so gateway mail from outside is automatically rejected.)

If an entry in smarthosts has an empty domain field, that is, it starts with a colon, that entry is taken to be the default route for remote domains. This feature can be useful to send outgoing mail to a gateway host on a local network, or to an ISP's mail server for dialup or consumer broadband users.

When qmail-remote looks up domains in smtproutes, it looks for successive tails of the recipient domain; if the target domain is bad.example.com, an entry for example.com matches it, unless there's also a more specific entry for bad.example.com.

11.3.2 Secondary MX Servers for Inbound Mail

The DNS makes it possible to list multiple MX hosts for a domain. If the hosts don't have the same distance value, the ones with greater distances are known as secondary servers. The server with the smallest distance is the primary server. A large domain can have several primary servers, and the primaries all have the same MX distance.

The SMTP specification requires that senders check closer hosts first, so secondary servers receive mail only if the closer servers are all unavailable. Qmail automatically handles secondary mail service for incoming mail. If a domain lists the qmail server as an MX for a domain but the domain isn't in locals or virtualdomains, qmail assumes it's a secondary server for that domain. When qmail-remote looks for a host to deliver mail to, it always checks servers starting with the ones at the smallest distance, but if it's a secondary for the domain, it ignores any MX with an equal or greater distance than its own, so that it always forwards the mail to a closer server.

If there is no lower distance server for a nonlocal domain, mail to the domain fails with the message Sorry. Although I'm listed as a best-preference MX or A for that host, it isn't in my control/locals file, so I don't treat it as local. This always indicates a configuration error. If there is supposed to be a different primary server, the DNS configuration is wrong. If the qmail system is supposed to be the primary server, the domain should be listed in locals or virtualdomains.

For outbound mail, qmail-remote's strategy, contacting the lowest-distance server for a domain, delivers to a primary server if one is available, otherwise to a secondary.

11.3.3 TCP Time-Outs

Sometimes an MX host is unavailable, either because the host itself isn't working or because there's a network failure somewhere between that host and yours. It can take a while for an attempted TCP connection to time out and tell a calling program that there's no answer at the other end, so an SMTP client should avoid trying to contact remote hosts that don't answer.

qmail-remote uses a simple scheme to track TCP failures. In the queue directory, the file queue/lock/tcpto contains a set of up to 64 entries of 16 bytes, each representing a failed remote host. (This makes the file size 1024 bytes, which generally fits in a single disk block that can be read or written in one operation.) Each entry in the file contains an IP address, the time the host was last tried, and the number of consecutive times it tried and failed to contact the host, capped at 10.

After qmail-remote makes its list of MX records to contact, before it tries to contact each MX, it looks up the host's IP address in tcpto. If the IP is present and has been tried at least twice, and it's been less than about an hour since the last try, it skips the host and goes on to the next. The exact wait time is randomized for each qmail-remote process, in the range of 60 to 90 minutes, to avoid having a whole bunch of simultaneous attempted connections when the time-out is up. Whenever qmail-remote connects successfully to a host, its IP is removed from tcpto. If it tries and fails to contact a host, it updates the host's record, incrementing the retry count and resetting the last contact time (unless the last contact time is less than two minutes old). If the host isn't already in tcpto, and tcpto is full, it discards an old entry, using a heuristic that looks for an entry that was last tried a long time ago and has a low retry count.

The overall effect of this scheme is to track unavailable remote hosts and retry each host no more than about once an hour. Once a host starts responding again, connections are tried whenever needed. If qmail-remote finds that all of the MX hosts for a delivery are unavailable, either because it tried and failed to contact them or because they're listed in tcpto as recently tried, the delivery attempt fails, and qmail-send will reschedule the delivery later using its quadratic backoff rule.