Protocols

Algorithms are building blocks; they specify how to transform a set of data into some other form by following a predefined set of rules. The algorithms described earlier in this chapter transform data by hashing, encrypting, or decrypting it. Those operations are useful on their own, but they are much more useful when combined with other operations to provide higher levels of functionality. This part of the chapter describes a variety of security protocols that use the basic algorithms described earlier (along with some new ones of their own) to provide authentication and confidentiality in various ways.

The Secure Sockets Layer (SSL) and Transport Layer Security (TLS) Protocols

Netscape originally introduced the SSL protocol in an early version of Netscape Navigator and its companion Web server. Its intent was to provide cryptographic authentication and confidentiality protection for Hypertext Transfer Protocol (HTTP) traffic, and it was wildly successful. In fact, the competing Secure HTTP standard never got off the ground because SSL was so widely deployed and worked so well. Early versions of SSL had some security holes, but Netscape fixed them with SSL revision 3.

However, the Internet community doesn t like vendor-owned standards, so in 1996 the Internet Engineering Task Force (IETF) released RFC 2246, a specification for the Transport Layer Security (TLS) protocol. TLS combined the cryptographic security of SSL with a more general implementation model and a better defined set of operations. TLS and SSL aren t directly interoperable, but a client/server pair can begin negotiating a TLS connection and fall back to using SSL if one end of the pair doesn t speak TLS. Because the two are so closely related , most of the time they re lumped together in Microsoft documentation and support materials, so I follow that convention here.

SSL/TLS is primarily used to secure HTTP traffic, although the protocol is generic enough to be used with Internet Message Access Protocol (IMAP), Post Office Protocol (POP), Network News Transfer Protocol (NNTP), and other protocols. Exchange 2000 supports the use of SSL/TLS with these protocols; Windows 2000 and Windows Server 2003 also allow users to accept or require SSL/TLS for LDAP connections. Because SSL/TLS provides authentication and security, this capability is very useful when designing systems that allow access from the Internet.

The Internet Protocol Security Extension (IPSec) Protocols

The IP that we all use and love is actually properly known as IP version 4 (IPv4). IPv4 was never designed to have any particular kind of security applied to it, because at the time of its design no one anticipated the explosive growth of the Internet or the widespread penetration of IPv4-based networks into critical business, government, and infrastructure systems. IPSec was designed to provide robust cryptographic authentication and confidentiality, but as a set of extensions to existing network protocols instead of a brand-new protocol like SSL.

The IPSec approach has some major benefits, the most important being that applications don t have to know, or care, that their traffic is being carried over an IPSec- protected connection. As far as Exchange, IBM Lotus Notes, or Unreal Tournament knows , its connection to the remote server is an ordinary socket connection. Beneath the application s level of awareness, though, the IPSec protocol can be applied to authenticate, encrypt, or authenticate and encrypt connections. IPSec can also act opportunistically ”whenever an IPSec-enabled computer attempts to connect to another machine, it can attempt to negotiate an IPSec session so that all connections that can be protected are. IPSec support is built into the Windows 2000, Windows XP, and Windows Server 2003 families, and it is well-supported by Cisco and other major networking vendors .

A detailed discussion of IPSec is far outside the scope of this book; it s a complicated set of several protocols. However, it s worth mentioning the two most immediately useful protocols because they re commonly used in Exchange deployments.

IPSec Authentication Header (AH)

RFC 2401 defines the Authentication Header (AH) protocol, which (as its name implies) adds a cryptographic authentication header to each IP datagram on a secured connection. The signature on the packet is calculated using all of the packet s data, except for some fields (like the time-to-live and hop count) that are certain to change while the packet is in transit. Once the signature is calculated, it s sandwiched between the original IP datagram header and the packet s payload (see Figure 2-5). This allows the packet to be routed without losing the AH data; non- IPSec-capable devices think that the AH data is just part of the payload.

Figure 2-5: The AH data follows the IPv4 packet header; the AH signature is calculated on the IPv4 header and the datagram s payload.

The exact algorithms used by a particular IPSec AH connection vary, as part of the IPSec negotiation process involves letting each end of the connection specify which algorithms it can accept. However, the most common algorithm used for IPSec AH is called MD5-HMAC. The HMAC stands for hashed message authentication code; it refers to an implementation of MD5 (or another hashing algorithm) that provides some signature functionality without using public-key algorithms.

Hashed Message Authentication Codes

HMACs are wonderfully useful in some circumstances. They depend on the existence of a secret shared between sender and recipient; this secret is combined with the message and the combination is hashed. Recall that one property of a hash algorithm is that it is one-way, so an attacker can t recover the secret. Another is that changing the message input changes the hash. The recipient can verify the signature on an HMAC-protected packet by combining its copy of the shared secret with the received message, rehashing it, and comparing the hashes.

This might seem like the same procedure used in public-key signature algorithms, only without the encryption. Conceptually, it is the same, but there are some important differences that give HMACs their utility. The first is speed; hash algorithms tend to be very fast, and computing an HMAC is as fast as computing an ordinary hash (the shared secret is usually a small fraction of the size of the message being hashed). In addition, HMACs can use any available hashing algorithm, most of which have no royalty or licensing requirements. Their chief drawback is the requirement that sender and recipient share a secret, but because the initial setup phase of IPSec allows the two computers to securely establish a shared secret, this is no problem for IPSec. IPSec can use MD5 or SHA-1 for HMACs.

HMACs have one major drawback: because both ends of an HMAC-protected message exchange have the secret key, the key might be compromised on either end. HMAC signatures don t provide nonrepudiation because neither party has exclusive access to the signing key.

 

IPSec Encapsulating Security Payload

AH provides message integrity (and, optionally , nonrepudiation, assuming that a real signature algorithm is used); its counterpart , Encapsulating Security Payload (ESP), provides message confidentiality and integrity checking. The full ESP definition is found in RFC 2401, but I limit myself to the highlights here. First, it s important to know that ESP can be used in two modes. In transport mode , ESP is used to encrypt the entire datagram contents, excluding the datagram header. The IPSec ESP header follows the IP datagram header, but precedes the Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) content. In tunnel mode , the entire packet, header and all, is protected by ESP. Because tunnel mode protects all contents of all packets on a secured connection, it s the mode used for VPN connections and links between separate networks, so it s much more common than transport mode. Figure 2-6 shows the difference: in the upper half of the figure, the workstation and server are using a transport-mode connection, so traffic between other computers on the network is not protected. In the lower half of the figure, tunnel mode is being used to secure all communications between the two LANs.

Figure 2-6: IPSec transport vs. tunnel mode.

Next, note that ESP and AH can be, and often are, used together. The impact of this capability is that the ESP data might be located at different places in the datagram:

• In the first case, the AH signature is applied to the entire packet; ESP is applied to the packet payload only. This provides authentication of the entire packet s data, but confidentiality protection only applies to the ESP- protected data. In Figure 2-7, the encrypted ESP header and payload are within the range of data signed by AH; the ESP encryption is applied before the packet is signed. The receiving system first verifies the AH packet. If the contents match the AH signature, the ESP portion is decrypted.

  
  Figure 2-7: In one mode of IPSec, AH is used to sign the entire packet, but only a portion of it is protected by ESP.

• In the second case, found only when tunnel mode is used, the AH signature is applied to the original packet before it s encapsulated with ESP. The datagram header for the new packet is not protected.

A third ESP capability involves the wide range of acceptable cryptographic algorithms specified in the standard. All IPSec ESP implementations must provide DES to meet the standard. However, they can optionally support CAST, 3DES, IDEA, Blowfish, RC5, and an RC4 clone called ARCFour. As part of the initial setup of a security association between two endpoints, each endpoint specifies what algorithms it supports, and the two ends agree to use the strongest algorithm that is supported by both. Although supporting multiple algorithms makes things a bit harder for implementers, it provides greater compatibility and flexibility for administrators, and it gives attackers much more work to do because they have to be prepared to attack a range of ciphers, not just one.

Secure/Multipurpose Internet Mail Extensions

There are two basic ways to secure message traffic: you can secure the communications channel used to pass the messages, or you can secure the messages themselves . SSL/TLS, IPSec, and proprietary hardware encryption boxes take the former approach; they provide authentication and encryption of a connection between two computers. Network traffic, including Simple Mail Transfer Protocol (SMTP), Messaging Application Programming Interface (MAPI), and HTTP, are encrypted as they pass over the connection, but only between the two endpoints of the tunnel. If the tunnel terminates at the entrance to another network (as it usually will when IPSec is used in tunnel mode), the traffic can pass unencrypted on the destination network en route to the destination computer. From that network, an attacker can easily sniff the traffic as it passes to an unprotected computer, or he or she can steal the data once it s written ”unencrypted ”on the destination machine. Of course, malicious administrators can read e-mail in transit or in storage, so it would be nice to protect against them, too.

S/MIME solves this problem by providing end-to-end security. Let s say that Alice wants to send Bob a confidential message asking him out on a date. If she s using an S/MIME-capable client like Outlook (all versions since Outlook 2000 SR1 support S/MIME version 3), she can compose a message using all of the mail client s capabilities, including rich text, attachments, and so forth. When she sends the message, it is encrypted with Bob s public key on her local machine before it s transmitted to the mail server. Alice s mail server can then send the message to Bob s server. Even if the message has to transit the Internet, its confidentiality is preserved because the message stays encrypted until Bob opens it. At that time, Bob s local computer uses his private key to decrypt the message; he can choose whether or not to expose the message by saving an unencrypted version on his local computer or some other location under his control.

Because S/MIME is an IETF standard, properly written S/MIME clients can exchange mail with each other. However, there are some obstacles to full interoperability. Chief among them is a simple problem: If Alice and Bob are in different organizations, how do they obtain each other s certificates so they can start a secure message exchange? Do they trust the same CAs? If not, how can they each decide whom to trust? This thorny problem can be resolved, but not without some effort on your part ”details are found in Chapter 12, Secure E-Mail.

There s another, lesser problem: how does Alice s client know which encryption or signature algorithms it can use when sending mail to recipients? The signature algorithm choice is easy: the S/MIME standard requires support for both MD5 and SHA-1, so clients can comfortably use either. Choosing the correct algorithm for encrypted mail is conceptually more difficult, but there s an elegant solution: put an attribute into each X.509 certificate that specifies for which algorithms it can be used. S/MIME specifies a set of symmetric algorithms for message encryption, including RC2, DES, 3DES, and CAST. In addition, the standard calls for support for RSA public-key encryption. Outlook is smart enough to figure out which algorithm should be used, but you can always override its choices. The range of options varies according to which certificate you use to sign and/or encrypt the message (see Figure 2-8).

Figure 2-8: Outlook s Change Security Settings dialog box lets you specify which algorithms should be used for an S/MIME message.

Authentication-Only Protocols

Authentication protocols allow an entity (a user, a computer, or a piece of software) to present some credentials that prove its identity. As you learned in the preceding chapter, authentication is a prerequisite for access control, so these algorithms are important in the context of permitting resource access for legitimate users and denying it to attackers.

LAN Manager and NT LAN Manager Protocols

Once upon a time, LAN Manager was what Microsoft called its MS-DOS-based networking code. That name persists with us today, even though the original codebase for LAN Manager is long gone, because Server Message Block (SMB), the basic networking protocol used in LAN Manager, is still with us. The LAN Manager family of authentication systems lives on too, even in Windows XP and Windows Server 2003. There are three related authentication protocols in the family: the original LAN Manager (LM) version, the NT LAN Manager version that shipped with Windows NT 3.1 (NTLMv1), and an improved, stronger version of NTLM (NTLMv2) that first shipped with Windows NT 4.0 Service Pack 4.

All of these protocols are challenge-response protocols . When a client requests authentication, the server generates a challenge and sends it to the client. The client encrypts it with a secret that only the client and server should know. The server then has to decrypt the secret and ensure that it s correct. If it can do so, that means that the client has the correct shared secret. Challenge-response protocols don t require exposure of the secret itself; in Windows, the key that the client uses is the hash of the user s password, which the domain controller stores in the security accounts manager (for Windows NT) or Active Directory (for Windows 2000). The differences among LM, NTLMv1, and NTLMv2 have to do with how the challenge is hashed. LM uses a method that s trivially easy for attackers to break; NTLMv1 is more difficult, and NTLMv2 is more difficult still.

The original version of LM exposed user password hashes, which were part of the challenge, to network sniffing. These hashes could be attacked , and the corresponding passwords recovered, with a brute force tool like l0phtcrack. This is obviously bad, so Microsoft now recommends that you turn off LM authentication altogether and force the use of NTLMv2. Doing so, however, might require some extra work on your part, because NTLM is natively supported by Windows NT 4 Service Pack 3 and later, but not by Microsoft Windows 95, Windows 98, or Windows Me. You can use the DSClient tool from the Windows 2000 CD to upgrade these Windows clients so that they speak NTLM, but there are also some server-side settings you must apply to force your server never to answer when an LM-only client makes an authentication request.

Kerberos

The Kerberos authentication protocol was named for the three-headed dog that, according to Greek mythology, guarded the underworld. That gives you a pretty good idea of what its designers had in mind when they built Kerberos; it was originally designed to function in the hostile environment of MIT s student computing network, where neither the clients nor the network could be assumed to be trustworthy. As with NTLM, Kerberos is a challenge-response protocol, but the mechanisms it uses to authenticate are significantly stronger. Kerberos also offers mutual authentication, which allows clients to verify the identity of the servers they connect to. These features are implemented by three separate Kerberos components : a client, a server, and a trusted key distribution center (KDC). (Now you know why Kerberos the three-headed dog was such an appropriate namesake!)

Kerberos is a complex protocol, so a complete description is outside the scope of this book. However, a brief description of how it works will help you understand how it s used to authenticate Exchange clients. Let s say Alice, using computer A, wants to log on to DOMAIN; her logon request is going to computer B, a Windows 2000 domain controller for DOMAIN. Here s what happens:

1. Alice enters her credentials on machine A, either using the keyboard or with a smart card or token.

2. The Kerberos client on Alice s workstation hashes her password and uses it to encrypt a request for a session key. The session key request is sent to the KDC.

3. The KDC looks up Alice s password hash and uses it to decrypt the session key request. If the request is valid, the KDC creates a new, random session key and uses it to create a session ticket . This ticket is encrypted with the shared secret for the server that Alice is requesting access to (in this case, the domain controller; it could just as easily be a file or Exchange server). The KDC then encrypts the ticket with Alice s password and sends the ticket back to machine A.

4. The Kerberos client on machine A decrypts the ticket and stores it in a RAM buffer. It then passes the ticket on to computer B, the domain controller.

5. Computer B decrypts the ticket using its local copy of the secret it shares with the KDC. If it can decrypt the ticket, that indicates that the KDC really did issue the ticket, so computer B can assume that Alice authenticated to the KDC and that her credentials are valid.

6. At this point, Alice is authenticated, but not necessarily authorized to access the requested resource. To authorize Alice, computer B checks to ensure that Alice has permission to have the requested resource. If so, she gets access; if not, she doesn t.

This brief description oversimplifies the process somewhat; there are a number of other subtleties, like the fact that tickets have expiration times to prevent them from being recorded and replayed by an attacker. If you re interested in more details of how Kerberos works, see the Additional Reading section at the end of the chapter.

Simple Authentication and Security Layer Protocol

Most communications protocols don t include built-in security, so adding security features to an existing standard like SMTP is fraught with peril. Simple Authentication and Security Layer (SASL) was designed to provide a way to wrap existing protocols with an additional layer of security. RFC 2222, which defines SASL, describes the protocol thusly:

This document describes a method for adding authentication support to connection-based protocols. To use this specification, a protocol includes a command for identifying and authenticating a user to a server and for optionally negotiating protection of subsequent protocol interactions. If its use is negotiated, a security layer is inserted between the protocol and the connection .

This sounds similar to SSL, and in fact it is. However, SASL is a bit more generic; the protocol specification specifies several different mechanisms that can be used with SASL, including mechanisms for unauthenticated access, authentication using plaintext usernames and passwords, and authentication using SSL/TLS or Kerberos. Application developers are free to use whatever combination of SASL mechanisms makes the most sense for them. In the case of messaging systems, SASL is usually used to implement the SMTP authentication extension (also known as SMTP AUTH; see RFC 2554). Exchange supports the use of SASL with three mechanisms: GSSAPI (based on Kerberos), NTLM challenge-response, and LOGIN (which uses the user s Windows logon credentials).

Extensible Authentication Protocol

SASL provides a flexible way for applications to choose the right set of authentication services. Extensible Authentication Protocol (EAP) provides the same service for network connections themselves. The original Point-to-Point Protocol (PPP) didn t require authentication; EAP was originally designed to provide a way for a PPP client and server to negotiate and agree on a mutually acceptable authentication method. EAP is also used in VPN software, including the Windows 2000 RRAS. The EAP definition in RFC 2284 describes some useful authentication types, but the most interesting one for our purposes is EAP-TLS (defined separately in RFC 2716), which allows EAP (and thus PPP, VPN, or even wireless connections) to use TLS. Because TLS supports the use of certificates, EAP-TLS provides a method for performing certificate-based access control, whether with locally stored certificates or smart cards.

Protected EAP

Protected EAP (PEAP) is a new Internet draft standard proposed by Cisco, Microsoft, and RSA Security. The existing EAP is useful for networks where physical interception is unlikely , but for wireless networks a more secure protocol was desirable. PEAP provides better security for the initial client/server authentication; in addition, it supports seamless reauthentication for roaming users and a fast reauthentication method. As an extra bonus, it can be used in a mode that allows clients to be authenticated without revealing their identities. PEAP is supported in Windows Server 2003 and will become more and more common as wireless access devices proliferate.

Remote Authentication Dial-In User Service Protocol

Remote Authentication Dial-In User Service (RADIUS), defined in RFC 2058, allows for distributed authentication among multiple systems. In an ordinary authentication system, the client presents its credentials to some central service that accepts or rejects them. However, for large networks (including those operated by Internet service providers), there might not be a single central service available. RADIUS solves this problem by allowing network access devices (like VPN servers) to accept the credentials and pass them to a RADIUS server. The server can then pass the authentication request on to the user directory. This pass-through makes it much easier to centralize authentication in a single directory. For example, a hardware VPN concentrator isn t likely to understand LDAP or Active Directory; with RADIUS, the concentrator can pass its request on to a RADIUS server, which can authenticate the user against Active Directory and process a set of rules to determine whether the now-authenticated client should be granted access.

The IEEE 802.1x Standard

The Institute of Electrical and Electronics Engineers (IEEE) is responsible for the 802 family of technical standards. IEEE 802.3 defines Ethernet, which you ve probably heard of, and 802.1 defines a variety of network-layer standards and protocols, including media access control (MAC) bridging and the relatively new 802.1x. By design, regular Ethernet networks don t attempt to provide authentication or access control services. Many vendors have implemented their own proprietary ways of providing port-based access control, so that only specified devices can connect to physical network ports. The 802.1x standard defines this process in a way that allows standards-compliant equipment to interoperate ; our primary interest is in the way that 802.1x allows strong authentication and access control for wireless connections, but these benefits apply equally to wired connections.

In an 802.1x connection, the client (or supplicant ) makes a request to an uncontrolled MAC address on the authenticator , which in the case of an 802.11 wireless local area network (WLAN) is normally a wireless access point. The authenticator sends a challenge to the supplicant; if the response to the challenge is valid, the authenticator requests credentials (normally using EAP-TLS) from the supplicant and passes them to an authentication server (normally using RADIUS). If access is granted, the authenticator gives the supplicant an authentication key that allows the supplicant to connect to controlled (for example, access-restricted) ports on the network access device. This combination of access control and authentication services greatly strengthens the security of WLAN connections. On its own internal network, Microsoft only permits wireless access for computers that have valid machine certificates; this effectively prevents attackers from surreptitiously connecting to the Microsoft WLAN.