Lesson 4:Security Protocols | Self-Paced Training Kit Exam 70-642: Configuring Windows Server 2008 Network Infrastructure

In addition to the basic security features implemented by network operating systems, such as passwords, there are also a number of security protocol standards that applications and operating systems use to protect their data as it's transmitted over the network. These protocols generally implement specific types of data encryption and define how the communicating computers are able to exchange the information needed to read each other's encrypted transmissions.

After this lesson, you will be able to

Describe how IPSec secures LAN communications
Understand how Layer 2 Tunneling Protocol (L2TP) is used in virtual private networking
Describe how Secure Sockets Layer (SSL) can secure Web client/server communications
Understand how Kerberos provides secure single network logons

Estimated lesson time: 20 minutes

IPSec

IPSec is the colloquial term used to describe a series of draft standards published by the Internet Engineering Task Force (IETF) that define a methodology for securing data as it is transmitted over a LAN using authentication and encryption. Most of the security protocols that provide encryption of data transmitted over a network are designed for use on the Internet or for specialized traffic between specific types of clients and servers. Until IPSec, there was no standard to protect data as it was transmitted over the LAN. You could control access to LAN resources using permissions and passwords, but the actual data as it traveled over the network medium was open to interception.

IPSec actually consists of two separate protocols that provide different levels of security protection. The IP Authentication Header (AH) protocol provides authentication and guaranteed integrity of IP datagrams, and the IP Encapsulating Security Payload (ESP) protocol provides datagram encryption. Using the two protocols together provides the best possible security IPSec has to offer.

The Authentication Header protocol adds an extra header to the datagrams generated by the transmitting computer right after the IP header. When you use AH, the Protocol field in the IP header identifies to the AH protocol, instead of the transport layer protocol contained in the datagram. The AH header also contains a sequence number that prevents unauthorized computers from replying to a message and an integrity check value (ICV) that the receiving computer uses to verify that incoming packets have not been altered. Encapsulating Security Payload works by encapsulating the transport layer data in each datagram using its own header and trailer, and by encrypting all of the data following the ESP header. The ESP frame also contains a sequence number and an ICV.

To use IPSec on a LAN, both the transmitting and receiving systems must have support for the protocols. However, because all of the information that IPSec adds to packets appears inside the datagram, intermediate systems such as routers do not have to support the protocols. Many of the major network operating systems are implementing IPSec in their latest versions, including Windows 2000 and various forms of UNIX. On a computer running Windows 2000, you configure the TCP/IP client to use IPSec in the Options tab of the Advanced TCP/IP properties dialog box. After selecting IP Security and clicking Properties, you see an IP Security dialog box like the one shown in Figure 13.10. After selecting the Use This IP Security Policy option, you can choose from the following policies:

Figure 13.10 The Windows 2000 IP Security dialog box

Client (Respond Only). This policy configures the computer to use IPSec only when another computer requests it.
Secure Server (Require Security). This policy configures the computer to require IPSec for all communications. Attempts to connect by computers that do not support IPSec are denied.
Server (Request Security). This policy configures the computer to request the use of IPSec for all communications, but to allow connections without IPSec when the other computer doesn't support it.

The IPSec functionality described in the previous section refers to transport mode operation, in which the upper layer data carried inside a datagram is protected by authentication or encryption. IPSec is also capable of operating in tunnel mode, however, which is intended for gateway-to-gateway communications, such as those used in virtual private networks (VPNs).

When two computers establish a VPN link across the Internet, the transmitting computer that originally generated the packet sends a normal datagram to a gateway (or router) that provides access to the Internet. The gateway, operating in tunnel mode, then encapsulates each entire datagram (including the IP header) within another datagram, and the entire construction is encrypted and authenticated using IPSec. This outer datagram functions as an encrypting "tunnel" through which the upper layer data travels in complete safety. After passing through the Internet and on reaching the gateway leading to the destination computer, the outer datagram is stripped away, and the data inside is authenticated and decrypted. The gateway can then forward the original (unencrypted) datagram to the destination end system. Thus, for this type of communication, the end systems involved in the transaction do not even have to support IPSec.

L2TP

IPSec can operate in tunnel mode independently or in cooperation with L2TP. This protocol was derived from the Cisco Systems Layer 2 Forwarding protocol and the Microsoft Point-to-Point Tunneling Protocol (PPTP) and is now defined by an IETF document. Layer 2 Tunneling Protocol creates a tunnel by encapsulating Point-to-Point Protocol (PPP) frames inside UDP packets. Even if the PPP frame contains connection-oriented TCP data, it can be carried inside a connectionless UDP datagram. In fact, the PPP frame can even contain Internetwork Packet Exchange (IPX) or NetBIOS Enhanced User Interface (NetBEUI) data.

Layer 2 Tunneling Protocol has no encryption capabilities of its own. It's possible to create a tunnel without encrypting the data inside it, but this is hardly worth the trouble. This protocol uses the IPSec ESP protocol to encapsulate and encrypt the entire UDP datagram containing the PPP frame. Thus, by the time the data is transmitted over the network, each packet consists of the original upper layer application data encapsulated within a PPP frame, which is in turn encapsulated by an L2TP frame, a UDP datagram, an ESP frame, an IP datagram, and finally another PPP frame, at which point the packet is finally ready for transmission over the network.

SSL

Secure Sockets Layer is a special-purpose security protocol that is designed to protect the data transmitted between Web servers and their client browsers. Virtually all of the Web servers and browsers available today support SSL. When you access a secured site on the Internet, to purchase a product using a credit card, for example, your browser is probably using SSL to communicate with the server. If your browser displays the protocol heading https:// in its address field instead of http://, then you are connecting to a secured site.

Like IPSec, SSL provides authentication and encryption services. Authentication is performed by the SSL Handshake Protocol (SSLHP), which also negotiates the method to be used to encrypt the data. The SSL Record Protocol (SSLRP) packages the data in preparation for its encryption. When a Web browser connects to a secured server, the server transmits a digital certificate to the client that it has obtained from a third-party certificate authority (CA). The client then uses the CA's public key, which is part of its SSL implementation, to extract the server's public key from the certificate. Once the browser is in possession of the server's public key, it can decipher the encrypted data sent to it by that server.

Kerberos

Kerberos is an authentication protocol that is typically used by directory services, such as Active Directory, to provide users with a single network logon capability. Once a server running Kerberos authenticates a client (called an authentication server), that client is granted the credentials needed to access resources anywhere on the network. Kerberos was developed at the Massachusetts Institute of Technology and is now standardized by the IETF. Windows 2000 and other operating systems rely heavily on Kerberos to secure their client/server network exchanges.

When a client logs on to a network that uses Kerberos, it sends a request message to an authentication server, which is already in possession of the account name and password associated with that client. The authentication server responds by sending a ticket-granting ticket (TGT) to the client, which is encrypted using a key based on the client's password. Once the client receives the TGT, it prompts the user for the password and uses it to decrypt the TGT. Because only that user (presumably) has the password, this process serves as an authentication. Now that the client is in possession of the TGT, it can access network resources by sending a request to a ticket-granting server (TGS), which may or may not be the same as the authentication server, containing an encrypted copy of the TGT. The TGS, on decrypting the TGT and verifying the user's status, creates a server ticket and transmits it to the client.

The server ticket enables a specific client to access a specific server for a limited length of time. The ticket also includes a session key, which the client and the server can use to encrypt the data transmitted between them, if necessary. The client transmits the server ticket (which was encrypted by the TGS using a key that the server already possesses) to that server, which, on decrypting it, grants the client access to the desired resource.

Exercise 1: Security Protocols

Match the security protocols in the left column with the correct descriptions in the right column.


Handshake Protocol ESP L2TP AH Kerberos	Provides encryption for IPSec Encapsulates PPP frames in UDP datagrams Provides authentication services to Active Directory Provides authentication services for SSL Provides authentication services for IPsec

Lesson Review

Which of the following protocols does L2TP use to encrypt VPN data?
1. L2TP (itself)
2. AH
3. ESP
4. SSL
What mode does IPSec use to secure LAN communications?
1. AH mode
2. Transport mode
3. ESP mode
4. Tunnel mode
Which of the following is the first element that a client receives as part of the Kerberos authentication process?
1. A TGT
2. A TGS
3. A session key
4. A server ticket

Lesson Summary

IPSec is series of TCP/IP standards that provide authentication and encryption services for LAN communications.
Layer 2 Tunneling Protocol creates tunnels for VPNs that can be encrypted using IPSec.
Secure Sockets Layer is a security protocol used by Web servers and browsers to encrypt and authenticate their transmitted data.
Kerberos is an authentication protocol used by directory services to provide clients with single logon access to network resources.