Using TLSSSL with SMTP | Secure Messaging with MicrosoftВ® Exchange Server 2003 (Pro-Other)

Using TLS/SSL with SMTP

The first step toward better COMSEC for your Exchange servers is to think about what kind of traffic they re passing. For most Microsoft Exchange Server 2003 servers, the majority protocol will be Simple Mail Transfer Protocol (SMTP). Accordingly, the place to first focus your attention is on SMTP traffic. If you can find some way to secure it, that will go a long way toward your goal of 100 percent COMSEC protection for mail traffic originating or terminating at one of your servers.

One potential way to solve this problem is to require the use of Secure Sockets Layer (SSL) for SMTP connections, but that raises a problem: by default, all SMTP servers use port 25. If you use SSL on port 25, non-SSL servers won t be able to connect; if you use a nonstandard port number, other servers won t be able to find your servers to send mail to them. (You might think that the Domain Name System [DNS] service resource [SRV] record would provide a solution; it could, but for some reason no message transfer agent [MTA] vendors are using SRV records to find mail servers for a domain!)

There s a clever workaround, though: the STARTTLS verb (part of the extended SMTP [ESMTP] command set) allows an SMTP client and server to negotiate the use of Transport Layer Security (TLS) for an SMTP connection. Each end of the connection can choose to authenticate the other, or the TLS connection can be used purely for privacy. Either way, this approach offers three important advantages: First, it doesn t interfere with ordinary servers and clients. Clients that support STARTTLS can use it; those that don t can continue to use unencrypted SMTP. Second, it s opportunistic . When you enable the use of TLS with SMTP, your server automatically requests TLS when communicating with other servers, and it accepts TLS connections when requested . Assuming the other server completes the negotiation process, mail flow is automatically protected. (Note that you ll generally have to tell your users to enable SSL/TLS in their Internet mail clients, though.) Finally, TLS- encrypting the SMTP stream means that message headers are also protected, giving you an additional degree of protection against traffic analysis.

Note	Be careful: SMTP+TLS does not protect messages from end to end ” that is, it doesn t protect the message in storage, or from client to server (unless the client supports TLS, too) ”it only protects the message as it passes between two servers that both support TLS.

Requesting an SSL Certificate

Before you can begin to use TLS with SMTP, you ll have to obtain and install a certificate for your SMTP server. The basic process of requesting an SSL certificate is simple, although there are several subtleties that can make it more difficult. If you ve already set up your own certificate authority (CA), you ll find that the process is quite simple; if not, you ll have to save the certificate request to a file and deliver it to your preferred CA. A more detailed explanation of how to set up and use the Microsoft CA, or how to make use of an external CA for your public-key infrastructure, can be found in Chapter 12, Secure E-Mail ; for now, I assume that you have access to a CA of some sort and want to request and install an SSL certificate for use with Outlook Web Access, IMAP, POP, or SMTP. The basic mechanics of certificate issuance are the same for all of these protocols.

No matter what you want to do with the certificate, you ll initiate the request process from Exchange System Manager on the Access tab of the virtual server properties for whatever protocol you want to secure. On the Access tab is the Certificate button, which is enabled whenever you run Exchange System Manager on an Exchange server. The private key associated with the certificate is generated on the local machine, so it doesn t make sense to generate a certificate from your administrative workstation.

Note	You can request a certificate for use with Outlook Web Access by using Internet Services Manager or Exchange System Manager, but it s easier to request POP and IMAP certificates with Exchange System Manager; for clarity, I stick with Exchange System Manager in this chapter.

Using an Online CA

When you click the Certificate button on a server that doesn t have a certificate for the specified protocol, Exchange System Manager launches the Web Server Certificate Wizard. This will be immediately familiar to you if you ve ever used Internet Information Services (IIS) to request an SSL certificate, for good reason: it s the same code.

Caution

To request a certificate for Exchange to use, you must have administrative privileges on the Exchange server, and the account you use must have permission to request the certificate from the CA.

After the Welcome page, the first step in the wizard is to specify what you want to do. If you re requesting a certificate for a virtual server that doesn t currently have one, your choices are to attach an existing certificate to the virtual server, import a backed - up certificate, or request a new one. If you re modifying an existing certificate, the wizard asks you to choose between renewing the certificate, removing it from the virtual server, or requesting a new one. In either case, let s assume that you want to request a new certificate. Once you ve done so, the process works like this:

The Delayed Or Immediate Request page allows you to choose to send the request directly to an online CA, using the Microsoft Remote Procedure Call (RPC) protocol, or to save the request in PKCS#10 format, the standard format used for requesting certificates. Because you re using your own internal CA, if it s online (see Chapter 12 to find out why it might not be), you ll select the Send The Request Immediately To An Online Certification Authority option. If you re using an external CA, or if your internal CA is offline, you ll select the Prepare The Request Now, But Send It Later option, then click Next .
The wizard prompts you for a name for your certificate and a key length (see Figure 11-1). Although Microsoft Windows 2000 and Windows Server 2003 support 512-bit keys, that s too short; always choose a key length of at least 1024 bits. Click Next to continue.

Figure 11-1: Name your certificate and select a key length of at least 1024 bits.
The wizard asks you to identify your organization and organizational unit (OU). These attributes are encoded into the issued certificate, so it s important that you get them right ”whatever you select here is permanent. Click Next once you re satisfied with your entries.
The wizard next asks for your server s common name (as opposed to the distinguished name, made up of the organization and OU components you just entered and the location components you re about to be asked for). The common name should be the fully qualified domain name (FQDN) of your server, as it will appear on the Internet. For example, if your SMTP server is named exch-smtp-sea-1.fabrikam.corp, but its external DNS name is mail1.fabrikam.com, you d enter the latter in this wizard page. Click Next to continue.

The Geographical Information page asks you to pick the country or region, state or province , and city name you want used in the certificate attributes. No one is going to check these for validity, but it s important to make sure that what you enter here is what you want your certificate to say; users or administrators who look at the certificate properties might be suspicious of your site s integrity if the location encoded in the certificate doesn t match your server s real location. Then click Next.

Note

If your machine s common name changes, you ll have to get a new certificate. If you get it wrong (for example, if the actual FQDN and the one in the certificate don t match), then client applications, including Microsoft Internet Explorer, complain that the certificate is bad. Be careful to enter the correct DNS name. Some certificate issuers allow you to re-request a newly issued certificate within a short period so that you can fix mistakes like this.

Up to this point, it doesn t matter whether you re requesting a certificate from an online or offline CA, because the same data has to go into the request either way. However, after providing your geographic information, an online request prompts you for the CA you want to use, based on the list of CAs visible to the requestor . If the CA you want to use isn t listed here, you cannot submit a certificate to it. If the CA is listed, select it (see Figure 11-2), and then click Next. You ll see a summary screen of the parameters you ve requested for your certificate; clicking Next again actually sends the request. If it succeeds, the certificate is automatically installed, and you can begin configuring TLS immediately.

Figure 11-2: Select an online CA to send your request directly to it.

Creating and Submitting a PKCS#10 Request

If you re submitting a certificate to a CA outside your network, or to a stand- alone CA that isn t integrated with Active Directory, your submission process will be a bit different. After providing the geographic information as described in step 5, the Certificate Wizard lets you choose where to save the request file. This file is a plaintext, base-64-encoded PKCS#10 request ”not much to look at. After saving it, you ll have to submit that file to your CA, normally using a Web-based interface. The details of the interface depend on your CA; for example, Thawte ( http://www.thawte.com ), VeriSign ( http://www.verisign.com ), and InstantSSL ( http://www.instantssl.com ) all have Web interfaces for requesting certificates, but they look and behave somewhat differently. (The good news is that each vendor has a prominently featured set of instructions covering the mechanics of submitting a request.)

Because the Microsoft CA can also accept PKCS#10 requests through its Web interface, let s walk through the process of issuing and installing a certificate using this method.

On your Exchange server, open Internet Explorer and navigate to http://yourCA/certsrv . You should see the CA Welcome page, as shown in Figure 11-3. The obvious choice for you is the Request A Certificate option; select it and click Next.

Figure 11-3: The Welcome page for the Windows Certificate Services CA.
The next CA page asks you to select a request type (Figure 11-4). The User Certificate Request list is what you d use if you were requesting an end user certificate; because you want a certificate for a virtual server (more important, because you already have a saved request), the Advanced Request option is what you need. Select it and click Next.

Figure 11-4: To issue a certificate for a virtual server, select the Advanced Request option.
The Advanced Certificate Requests page gives you three choices: submit a new request using the CA s forms interface, submit a precreated PKCS#10 request, or request a certificate for a smart card user. Because you want to use your existing certificate request, select Submit A Certificate Request Using A Base64 Encoded PKCS #10 File Or A Renewal Request Using A Base64 Encoded PKCS #7 File and click Next.
The Submit A Saved Request page (Figure 11-5) is where you actually submit your request for processing. Open the PKCS#10 file using Notepad, then copy its contents and paste them into the Saved Request text box. (You can also use the Browse link to find and upload the request file if you configure the certificate server as a trusted site.) Click Submit, and the request is sent to the CA for processing.

Figure 11-5: Paste your certificate request file into the Saved Request text box and click Submit.

You re not quite done; as the Submission Confirmation page tells you, you ll still have to come back to the CA to check the status of the request and retrieve the issued certificate. (Depending on the kind of CA you re using, it might automatically issue the certificate, or it might require administrator intervention ”see Chapter 12.) To do so, return to the CA Welcome page at http:// servername /certsrv/ . Select the Check On A Pending Certificate option and click Next. The CA lists all the pending requests it knows of; select yours, then click Next again.

What you see next depends on whether the CA has issued your certificate or not. If not, the CA Web page tells you that the request is still pending; you ll have to approve it (or have it approved). See the sidebar Approving Pending Requests later. If the certificate has been issued, you ll see a page like the one shown in Figure 11-6. This page is a little confusing, because there s no obvious link that says Download my new certificate. Instead, you need to click Download CA Certificate and save the resulting file on your Exchange server. Exchange System Manager expects the certificate to be stored in the Distinguished Encoding Rules (DER) encoded format, so be sure to choose the appropriate option.

Figure 11-6: Download the new certificate to finish installing it.

Once you ve saved the certificate, you re ready to associate it with your virtual server. Go back to Exchange System Manager, find the virtual server you want to attach the certificate to, and open its Properties dialog box. Click the Access tab and click Certificate again. The Web Server Certificate Wizard starts again. This time, it indicates that a certificate request is already pending; select the Process The Pending Request And Install The Certificate option before clicking Next. On the next page, specify the path to the certificate file you just downloaded and click Next. Exchange System Manager decodes the certificate and shows you a summary screen that allows you to double-check that you re installing the right certificate. Once you re satisfied, click Next one more time, then click Finish, and your certificate is installed. You can verify this by checking the status of the Communications button on the Access tab ”it will only be available if you have a valid certificate installed for the virtual server.

Approving Pending Requests

If you check the status of your first certificate request, it might say that the request is being held pending administrator approval. This is common, because the default setup for the stand-alone mode of the Windows CA is to hold certificates until the administrator has approved their issuance. You use the Certification Authority snap-in, which must be run on the CA itself, to approve and check on requests. Here s how to see the pending requests so you can approve or deny them:

Launch the Certificate Services snap-in (Start Program Files Administrative Tools Certificate Authority).
When the snap-in opens, navigate to your CA in the left-hand pane, then expand its node. You ll see five folders beneath it: Revoked Certificates, Issued Certificates, Pending Requests, Failed Requests, and Policy Settings.
Click Pending Requests. The rightmost pane fills with all the pending requests on this CA. Right-click an individual request to use the All Tasks Issue or All Tasks Deny commands ”you can undoubtedly guess what they do. Pay attention to which request you re clicking to ensure that you approve the correct one!

The Certification Authority snap-in can do a lot of other tricks, too, including revoking individual certificates (select a certificate from the Issued Certificates folder, right-click it, and choose All Tasks Revoke Certificate) and setting issuance policies for the CA. For a more complete reference, see the Windows 2000 online help or Chapter 18 of Crawford, Russel, and Gerend s Microsoft Windows 2000 Server Administrator s Companion, Second Edition.

Enabling STARTTLS

Now that you have a certificate in place, you re ready to start using it to protect your SMTP traffic. You have three choices: you can force the use of TLS for all outbound mail, you can use TLS for mail to selected domains, and you can choose to force other servers that contact you to use TLS. These options are independent of one another, so you can use them individually or together.

Forcing TLS for All Mail Traffic

To turn on TLS for all outbound mail on a selected SMTP virtual server, you ll use the Delivery tab of the SMTP virtual server s Properties dialog box. In particular, the Outbound Security button displays the Outbound Security dialog box, shown in Figure 11-7. All you have to do is select the TLS Encryption check box and click OK.

Figure 11-7: Turning on outbound TLS only requires selecting one check box.

However, once you do this, you are effectively restricting Exchange to communicating only with other hosts that support TLS. This will have the unwelcome side effect of making your mail sit in delivery queues forever, or at least until the retry interval expires and the messages are returned, with nondelivery reports (NDRs), to their senders. A better approach is to tailor the use of TLS to specific domains by using SMTP connectors.

Enabling TLS for Specific Domains

You re almost certainly better off restricting your outbound TLS use to domains that you know support TLS (one way to find out: telnet to port 25 on the domain s mail server and see whether sending an EHLO returns a response with STARTTLS listed). The easiest way to accomplish this restriction is by creating SMTP connectors to handle traffic for those specific domains; you do this from within the Routing Groups container of Exchange System Manager. When you create the connector (or modify the properties of an existing one), the two critical steps are to ensure that you ve specified the correct domains in the Address Space tab and that you ve turned on TLS on the Advanced tab. To do so, click the Advanced tab, click Outbound Security, and make sure that the TLS Encryption check box is selected.

Enabling TLS for Inbound Mail

Turning on inbound TLS is almost as easy, but you have to do it from a different location. Click the Access tab of the virtual server Properties dialog box, then click Communication in the Secure Communication command group . You ll see the Security dialog box, shown in Figure 11-8, from which you can turn on TLS at its default 40-bit strength or the full (and highly recommended) 128-bit version. It s important to note that selecting the Require Secure Channel check box does exactly that: servers that don t support TLS, or those that cannot successfully negotiate an agreeable set of TLS parameters with your server, won t be able to deliver mail to you. Be careful when turning this on; if you see a sudden drop in the volume of inbound mail after enabling this option, you might want to consider whether improved privacy is worth losing messages.

Figure 11-8: You can turn on regular or extra-strength TLS for inbound connections.

When Negotiation Fails

After you first turn on outbound TLS, keep an eye on your server queues. Some systems (mostly those running variants of UNIX sendmail) will reject TLS connections from systems with self-signed certificates, whereas others will accept STARTTLS but then fail the security negotiation for some other reason. You can see where I m going with this: if your server cannot successfully negotiate a TLS connection, and if you or the other system require TLS, mail won t flow to those systems.

If you see mail backing up to a particular destination system after you turn on TLS, you ll need to find out whether TLS failure is the cause. If so, you ll need to know why TLS is failing; check the SMTP logs and the system and application event log for clues. If that doesn t help, contact the administrator of the remote system. If you still cannot resolve the problem, you might be able to work around it. The Exchange/IIS stack doesn t support domain-specific TLS settings, but you can work around that limitation by creating a second SMTP virtual server that doesn t use TLS and pairing it with an SMTP connector. Set the connector s properties to include the domains that you re having trouble talking to. Because Exchange always prefers a more specific route to a less specific one, mail bound for the problem destinations will be sent over the connector (provided the address spaces match), and your mail will flow again.