Introduction | sendmail Cookbook

Chapter 7 contains recipes that use the AUTH SMTP extension to provide strong authentication of the end points in a mail exchange. In this chapter, the recipes use the SMTP STARTTLS extension for both strong authentication and encryption.

While encryption is one of the primary benefits of the SMTP STARTTLS extension, it is important to remember that this is not end-to-end encryption. Mail can take multiple hops before it is delivered. Some intervening hops may not use STARTTLS. Additionally, the mail message is still stored as clear text by both the sender and the recipient. STARTTLS only provides encryption for mail passing over a single hop between two sendmail systems configured for STARTTLS.

Chapter 7 explained how sendmail relied on the Simple Authentication and Security Layer (SASL) to provide the security tools necessary for AUTH authentication. STARTTLS relies on the Transport Layer Security (TLS) protocol for both authentication and encryption.

Transport Layer Security

TLS is an Internet standard protocol, defined in RFC 2246, The TLS Protocol Version 1.0 . TLS is based on the Secure Sockets Layer (SSL) protocol that was originally developed by Netscape for web security. The manner in which TLS is used to secure mail transport is defined in RFC 2487, SMTP Service Extension for Secure SMTP over TLS . The SMTP extension that supports TLS is called STARTTLS . The receiving system advertises support for STARTTLS in its response to the client's EHLO command. The connecting system requests TLS security by issuing the STARTTLS command in response to the advertisement.

TLS uses public key encryption , which is also called asymmetric encryption . It is asymmetric because it uses two different keys. A public key that is made available to the world and a private key that is kept secret. Anything encrypted with the public key can only be decrypted with the private key, and anything encrypted with the private key can only be decrypted with the public key. Four keys are needed to authenticate both the TLS client and the TLS server: the client's public and private keys and the server's public and private keys. ^[1]

^[1] For the SMTP connection, the TLS client is the system initiating the connection, and the TLS server is the system to which the connection is being made.

The certificate

A TLS public key is distributed in a file called a certificate . It is called a certificate because it is certified to contain a valid public key by a digital signature . There are a few different techniques used to sign the certificates used by sendmail.

First, there are commercial certificate authorities (CAs). For a fee, a commercial CA will sign your certificate request creating a signed certificate. A CA is trusted to provide reasonable assurance that the certificate is correct and represents the organization it claims to represent. Commercial CAs are well established because they have been signing certificates for web sites for a number of years . Using a commercial CA to sign sendmail certificates is easy, and it has the added benefit of supporting a global customer base ”certificates signed by a commercial CA are accepted around the world. Because many web sites want access to a global customer base, they use commercial CAs. In fact, a list of trusted commercial CAs can be obtained from any web browser. Figure 8-1 shows the list of CAs displayed by Netscape.

Figure 8-1. The Netscape list of recognized CAs

An alternative to using a commercial CA is to create a private CA . Creating a private CA is more complex than using a commercial CA, and it limits the customer base to those sites that are willing to accept certificates from the private CA. While a private CA is inadequate for a web site that must support a global customer base, it is a viable solution for sendmail security because mail security is often used between a limited number of partners . Recipe 8.1 shows you how to build a private CA, and Recipe 8.3 uses that CA to create the certificates used in the other recipes.

Finally, it is possible to create a self-signed certificate , which is not signed by any type of CA. sendmail does not accept self-signed certificates for authentication because, without the certification of a recognized CA, there is no way to verify the certificate's information. sendmail does, however, accept self-signed certificates for encryption. In fact, if you only want encryption, the TLS client doesn't need a certificate at all. However, using a CA increases the utility of the STARTTLS protocol because it adds strong authentication to basic encryption.

If the sendmail system is also a web server, it may already have a commercially signed certificate that can be used for sendmail. If not, use openssl to create certificates. To have a sendmail certificate signed by a commercial CA, create a certificate signing request (CSR) ”also called a certificate request ” and send it to the commercial CA for signing. (See the web site of the CA you select for the exact details.) If you run a private CA, create the CSR and use the private CA to sign it. Recipe 8.2 and Recipe 8.3 show examples of creating a CSR and of signing it to produce a signed certificate.

Certificates contain much more than just a public key. Use openssl to examine the contents of a certificate, as in this example:

 #  openssl x509 -noout -text -in rodent.pem  Certificate:     Data:         Version: 3 (0x2)         Serial Number: 2 (0x2)         Signature Algorithm: md5WithRSAEncryption         Issuer: C=US, ST=Maryland, L=Gaithersburg, O=WroteTheBook,                 CN=chef.wrotethebook.com/Email=craig@chef.wrotethebook.com         Validity             Not Before: Jan 28 16:25:58 2003 GMT             Not After : Jan 28 16:25:58 2004 GMT         Subject: C=US, ST=Maryland, L=Gaithersburg, O=WroteTheBook,                  CN=rodent.wrotethebook.com/Email=alana@rodent.wrotethebook.com         Subject Public Key Info:             Public Key Algorithm: rsaEncryption             RSA Public Key: (1024 bit)                 Modulus (1024 bit):                     00:b0:91:de:51:14:19:91:61:39:ed:23:61:ef:9d:                     e4:44:9f:ff:6a:fd:5c:cc:e8:bd:79:1f:2d:bc:eb:                     ca:66:da:86:fb:99:17:79:07:94:dd:94:90:aa:6a:                     44:55:ff:ec:65:e9:4e:41:b1:e9:e9:c4:09:89:81:                     47:2b:a1:bd:a0:0b:bf:9c:c5:5e:39:de:c5:49:a3:                     e6:5c:af:9e:32:eb:ae:63:80:7f:c6:ad:12:d0:87:                     01:db:06:a3:93:bf:41:9f:51:81:4f:09:fa:da:e3:                     ca:69:dc:35:4f:cd:43:fd:ab:6c:51:3d:24:97:60:                     bd:bf:e8:91:9c:5a:f4:b2:df                 Exponent: 65537 (0x10001)         X509v3 extensions:             X509v3 Basic Constraints:                 CA:FALSE             Netscape Comment:                 OpenSSL Generated Certificate             X509v3 Subject Key Identifier:                 76:6E:60:D0:EE:B4:62:49:84:46:04:7C:44:E6:F8:9E:CF:84:B6:DC             X509v3 Authority Key Identifier:                 keyid:3C:D9:71:02:66:8A:D4:AD:3C:07:FD:57:2A:C5:03:C2:B8:7D:FF:D5                 DirName:/C=US/ST=Maryland/L=Gaithersburg/O=WroteTheBook/                        CN=chef.wrotethebook.com/Email=craig@chef.wrotethebook.com                 serial:00     Signature Algorithm: md5WithRSAEncryption         c4:45:4a:e4:01:eb:0b:83:c4:26:c5:de:9e:06:8f:7a:b6:2d:         f0:f3:63:dd:ba:1c:22:2d:f9:ac:3d:92:04:c4:4f:a8:9c:d1:         7c:de:a7:6d:94:10:62:cb:69:c6:ba:8f:09:e7:a9:49:18:97:         4b:aa:50:b9:0a:c9:49:59:0d:65:b6:44:1a:d6:88:6c:bd:d9:         3e:89:e6:1c:76:c8:54:69:7f:b4:9d:7a:a7:de:92:12:cc:f5:         43:e5:e5:da:b0:34:9d:cf:ee:b9:65:a1:a8:d7:cc:aa:96:12:         10:7f:2b:99:94:79:f3:b4:bf:0a:58:54:65:7f:98:36:be:a5:         22:de

Several important pieces of information can be seen in this display. Foremost, of course, is the public key. It is the 1024-bit RSA public key that appears about halfway through the display. This is the key used to encrypt the session information sent to the system holding the private key ”the session information includes the session key that will be used to encrypt the mail.

The system that owns this certificate is identified by the Subject field. The Subject field contains the distinguished name (DN) of the remote host. The DN is a formal name that can contain several fields. In this example, the DN contains a country code (C), a state code (ST), a locality (L), an organizational name (O), and a common name (CN). Inside the sendmail configuration, the DN of the subject is found in the ${cert_subject} macro, and the CN of the subject is found in the ${cn_subject} macro. In the example, the certificate belongs to the host rodent.wrotethebook.com .

The CA that signed the certificate is identified by the Issuer field, which contains the DN of the CA. Inside the sendmail configuration, the DN of the CA is found in the ${cert_issuer} macro, and the CN of the CA is found in the ${cn_issuer} macro. In the example, chef.wrotethebook.com is the private CA that is created in Recipe 8.1.

At the bottom of the display is the digital signature of the certificate created by the CA. It is an MD5 digest of the certificate, which is stored in the sendmail configuration in the macro ${cert_md5} . The digest is encrypted by the CA's private key. sendmail uses the CA's public key to decrypt the digest. If sendmail successfully decrypts and verifies the digest, it stores OK in the macro ${verify} and considers the certificate valid for authentication.

The ${verify} macro can contain any one of several different values depending on the result of the verification process. The possible values for the ${verify} macro are:

FAIL: The certificate failed the verification process.
NONE: STARTTLS is not being used. Thus no verification was performed.
NOT: A certificate was not requested from the remote host. If the local system does not request a certificate from the remote system, verification is not performed.
NO: No certificate was provided by the remote host. Therefore, there was no certificate to verify.
OK: The certificate was successfully verified .
PROTOCOL: A protocol error occurred.
SOFTWARE: The STARTTLS handshake failed.
TEMP: A temporary error occurred.

Once again, refer back to the certificate displayed by the openssl x509 command. Near the top of the output, the TLS version number is identified in the Version field of the certificate. Inside the sendmail configuration, the TLS version number is stored in the ${tls_version} macro.

As we have seen, several sendmail macros can be traced directly back to the TLS certificate. Two other important TLS related macros get their values from the session encryption used for the email connection. ${cipher} identifies the type of encryption used for the connection, and ${cipher_bits} contains the number of bits of encryption used for the connection. All of the important TLS encryption and authentication data is available for use inside the sendmail configuration.

Using the access database with TLS

The best way to control how TLS security information is applied on your system is through the access database. These are the access database records that provide control over a TLS connection:

CERTISSUER :: Authorizes relaying based on the DN of the CA that signed the certificate. Recipe 8.5 and Recipe 8.6 provide example of using the CERTISSUER : record.
CERTSUBJECT :: Authorizes relaying based on the DN of the host that presented the certificate. Recipe 8.6 covers the use of this record.
TLS_Srv :: Defines the minimum security required for an outbound TLS connection. Recipe 8.7 and Recipe 8.9 show how to use the TLS_SRV : record.
TLS_Clt :: Defines the minimum security required for an inbound TLS connection. Recipe 8.8 and Recipe 8.9 contain examples of the TLS_Clt : record.
TLS_Rcpt :: Defines the minimum security required for mail sent to the specified recipient. Recipe 8.10 uses this record.
Try_TLS :: Controls whether the STARTTLS command is issued to the specified remote TLS server. See Recipe 8.11 for an example of how this record is used.
Srv_Features :: Controls whether STARTTLS is advertised to the specified remote TLS client, and whether the specified remote TLS client is required to present a certificate. See Recipe 8.12 and Recipe 8.13 for examples of this record.

sendmail provides hooks into the rulesets that process the access database entries related to STARTTLS:

LOCAL_TLS_CLIENT is a hook into the tls_clt ruleset that processes TLS_Clt : database records.
LOCAL_TLS_SERVER is a hook into the tls_srv ruleset that processes TLS_Srv : access database records.
LOCAL_TLS_RCPT is a hook into the tls_rcpt ruleset that processes TLS_Rcpt : records.
LOCAL_TRY_TLS is a hook into the try_tls ruleset that processes Try_TLS : records.

These hooks provide access to the full range of sendmail's capabilities, including all of the information that sendmail has at the time that the access database records are being processed . Anything that cannot be done within the confines of the standard database record formats can probably be done through a custom ruleset. The power and simplicity of using the access database to control delivery over a TLS connection make custom TLS rulesets largely unnecessary. However, if you have an application for a custom ruleset, sendmail provides all of the tools you need to roll your own.

OpenSSL

The sendmail implementation of STARTTLS depends on OpenSSL for authentication, encryption, and support tools. Before any of the recipes in this chapter can be used, OpenSSL must be installed.

Modern systems often include OpenSSL in order to provide web security. Our sample Red Hat Linux system includes an OpenSSL RPM file, as this simple command shows:

 $  rpm -q openssl  openssl-0.9.6b-18

If your system includes OpenSSL, you can proceed. If it is not installed, install the version provided by your Unix vendor using the vendor's package management tools. If your Unix vendor does not provide OpenSSL, download the source from http://www.openssl.org/source/or ftp://ftp.openssl.org/source/. The source comes as a gzipped tarball. Unzip and restore the tarball. Go to the newly created directory and run config and then make to compile the source code. Then run make test and make install to install OpenSSL. If you have any problems compiling the source, see the INSTALL file delivered with the source code.

Additionally, sendmail must be specially compiled to support STARTTLS. If your sendmail binary was not compiled with STARTTLS support, recompile sendmail as described in Recipe 1.6 and Recipe 1.7.