Introduction to IP Security (IPSec)

IP Security (IPSec) is a framework of open standards for ensuring private, secure communications over IP networks. This protocol is rapidly becoming the underlying framework for secure communications using VPNs and will likely replace Point-to-Point Tunneling Protocol (PPTP) as Microsoft's VPN protocol of choice. IPSec takes advantage of many of the more popular encryption protocols in use today. IPSec is based on an end-to-end security model, which means that the only computers that must know about IPSec are the sending and receiving computers. The packets travel the network without being affected by any of the intervening network devices. Each IPSec device handles its own security and functions with the assumption that the transport medium is not secure. The Internet is an excellent example of a transport medium that is not secure.

The Microsoft Windows Server 2003 implementation of IPSec is based on standards developed by the Internet Engineering Task Force (IETF) IPSec working group. However, it is important to note that Microsoft uses two implementations of IPSec: the IETF version, also known as Pure IPSec Tunnel mode, or just tunnel mode; and the Microsoft variant on IPSec, which Microsoft calls L2TP/IPSec mode, or transport mode. An IPSec VPN configured to use transport mode secures an existing IP packet from source to destination, using the encryption and authentication methods discussed later in this section. Tunnel mode puts an existing IP packet inside a new IP packet that is sent to a tunnel endpoint in the IPSec format.

Both transport and tunnel mode can be encapsulated in Encapsulating Security Protocol (ESP) or Authentication Header (AH) headers. The original IETF Request for Comments (RFC 2401, http://www.ietf.org/rfc/rfc2401.txt) IPSec tunnel protocol specifications did not include mechanisms suitable for remote access VPN clients, instead focusing on site-to-site VPN implementations. For that reason, Microsoft's implementation of tunnel mode relies on the use of the L2TP protocol developed jointly with Cisco to provide this additional packet format. (An introduction to the components of IPSec comes later in this chapter.) It is worth noting that a newer RFC (RFC 3193, http://www.ietf.org/rfc/rfc3193.txt) introduced in late 2001 discusses using L2TP with IPSec.

Note: Protecting VPN Connections

By default, client remote access VPN connections are protected using an automatically generated IPSec policy that uses IPSec transport mode (not tunnel mode) when the L2TP tunnel type is selected. You will probably see this configuration in almost every production implementation of a Windows Server 2003 IPSec solution. To enable this configuration, configure the Routing and Remote Access Service for L2TP VPN connectivity, as described previously in Chapter 6, "Implementing, Managing, and Troubleshooting Network Access."

Exam Alert: Understand IPSec

The exam includes questions and scenarios on IPSec. Although you don't need to memorize the minutiae surrounding the encryption protocols used by IPSec, you should be familiar with what the components of IPSec are, how to implement an IPSec tunnel, and especially how to work with IPSec policies.

Understanding the Architecture and Components of IPSec

Let's look at the underlying architecture and components of the IPSec protocol. IPSec provides data integrity and identity protection services for each IP packet by adding a security protocol header to each IP packet. This header is made up of several components, each with its own function.

Authentication Header (AH)

The IPSec Authentication Header (AH) provides three services as part of the IPSec protocol. First (as its name might suggest), AH authenticates the entire packet. Second, it ensures data integrity. Third, it prevents any replaying of the packet by a third party who might try to penetrate the IPSec tunnel. One service AH doesn't provide is payload encryption. AH protects your data from modification, but an attacker who is snooping the network would still be able to read the data. To prevent the modification of the data, AH uses two hashing algorithms to "sign" the packet for integrity:

• The Message Digest 5 (MD5) algorithm applies the hashing function to the data in four passes.

• The Secure Hash Algorithm (SHA1) is closely modeled after MD5. SHA uses 79 32-bit constants during the computation of the hash value, which results in a 160-bit key. Because SHA has a longer key length, it is considered more secure than MD5.

AH uses an IP protocol decimal ID of 51 to identify itself in the IP header. The AH header contains the following fields:

• Next Header This field identifies the next header that uses the IP protocol ID.

• Length This field indicates the length of the AH header.

• Security Parameters Index (SPI) Used in combination with the destination address and the security protocol (AH or ESP), the SPI is used by the receiver to identify the cryptographic keys and procedures to be used to decode the packet.

• Sequence Number This field provides the anti-replay functionality of AH. The sequence number is an incrementally increasing number (starting from 0) that is never allowed to cycle and that indicates the packet number. The machine receiving the packet checks this field to verify that the packet has not been received already. If a packet with this number has already been received, the packet is rejected.

• Authentication Data This field contains the Integrity Check Value (ICV) used to verify the integrity of the message. (This is the hash value mentioned previously.) The receiver calculates the hash value and checks it against the ICV to verify packet integrity.

An IP packet that has AH applied in transfer mode is modified with the AH header between the IP header and the TCP header, as shown in Figure 8.1.

Figure 8.1. An IP packet with the AH header inserted in transfer mode.

AH can be used alone or in combination with the ESP protocol, which is discussed next.

Encapsulating Security Protocol (ESP)

Encapsulating Security Protocol (ESP) provides confidentiality in addition to authentication, integrity, and anti-replay. This portion of the IPSec protocol encrypts the data contents of the packet. The format of the ESP varies, depending on the type and mode of encryption being utilized. ESP can be used alone, in combination with AH, or using Microsoft's implementation, nested within the L2TP.

ESP appears in the IP header with an IP protocol decimal ID of 50. The ESP header contains the following fields:

• SPI The receiver uses the SPI, in combination with the destination address and the security protocol (AH or ESP), to identify the cryptographic keys and procedures to be used to decode the packet.

• Sequence Number This field provides the anti-replay functionality of ESP. The sequence number is an incrementally increasing number (starting from 0) that is never allowed to cycle and that indicates the packet number. The machine receiving the packet checks this field to verify that the packet has not been received already. If a packet with this number has already been received, the packet is rejected.

Note: Replaying: Why Is It Bad?

You may have noticed that replaying has been included several times as part of the discussion of IPSec. Replaying is a somewhat obscure method for obtaining access to a system. A replay attack occurs when packets are intercepted by an unauthorized party, stored, and later retransmitted in an effort to trick one or both sides of the transmission into thinking that it is a valid communication. A replay attack becomes an issue because TCP/IP protocols such as Network File System (NFS) have no mechanisms to determine whether a packet is being replayedeven after several hours. Fortunately, the anti-replay mechanisms in IPSec make a replay attack a virtual impossibility.

The ESP trailer contains the following fields:

• Padding Specifies 0 to 255 bytes used for 32-bit alignment and with the block size of the block cipher.

• Padding Length Indicates the length of the Padding field in bytes.

• Next Header Identifies the makeup of the payload, such as TCP or UDP.

The ESP authentication trailer contains one field: authentication data. This field contains the Integrity Check Value (ICV) and a media access control (MAC) used to verify the sender's identity and ensure message integrity.

ESP is inserted after the IP header and before an upper-layer protocol, such as TCP, UDP, or ICMP, or before any other IPSec headers (such as AH) that have already been inserted. Everything following ESP (the upper-layer protocol, the data, and the ESP trailer) is encrypted, as shown in Figure 8.2. The IP header is not signed and, therefore, not necessarily protected from modification unless tunneling mode is active, as shown in Figure 8.3.

Figure 8.2. An IP packet with the ESP header inserted in transfer mode.

Figure 8.3. An IP packet with the ESP header inserted in tunnel mode; the original IP header is now encrypted.

The final piece of the IPSec protocol is the authentication and key exchange mechanism. Authentication and key exchange are accomplished using a pair of protocols.

Internet Security Key Association Key Management Protocol (ISAKMP/Oakley)

ISAKMP/Oakley (also known as ISAKMP/IKE, for Internet Key Exchange) provides the mechanism that allows disparate VPN servers to share encryption key information and make the IPSec protocol practical in today's environment. Before secured data can be exchanged between VPN servers, a contract between the two computers must be established. In this contract, called a Security Association (SA), both computers agree on how to exchange and protect information. In other words, the two servers (or the server and client computer) need to agree on how to encrypt and decrypt the data to be sent.

To enable this process, the IPSec protocol uses a standard process to build this contract between the two computers. This process combines the ISAKMP (Internet Security Association and Key Management Protocol) and Oakley key generation protocols. ISAKMP provides the centralized security association management, whereas Oakley actually generates and manages the encryption keys used to secure the information.

IKE actually performs a two-phase operation to establish the secure communication channel. In each phase, confidentiality and authentication are ensured by the use of encryption and authentication algorithms that are agreed upon by the computers negotiating. In the first phase (also known as main mode), the two computers establish a secure, authenticated channel called the Phase I (Main Mode) SA. The IKE provides identity protection during this phase. In the first phase (also known as quick mode), the two computers establish the rules for communication called the Phase II (Quick Mode) SA. During the second phase, a new shared key is created for use. After the Phase II SA is in place, IPSec secured communications can occur.

The final piece of this puzzle that needs to be covered is Microsoft's IPSec/L2TP implementation, which adds an additional tunneling protocol to the IPSec implementation.

L2TP and IPSec

The major difference between the ESP tunnel and L2TP is that the L2TP tunnel performs at Layer 2 of the OSI Model, the data link layer. This way, L2TP can tunnel additional protocols, such as IPX or NetBEUI. IPSec's ESP tunneling protocol tunnels only the TCP/IP protocol, based on the standard. When L2TP and IPSec are used in combination to provide a secured tunnel, the original packet header is used to carry the packet's source and final destination, whereas the tunnel packet's IP header might contain the address of an IPSec gateway. The L2TP header carries the information needed to route the packet over the network. The Point-to-Point Protocol (PPP) header within the encapsulated packet identifies the protocol of the original packet. In other words, when using L2TP to transfer data, IPSec is used to secure the tunnel. L2TP encapsulates the packet in a PPP frame. The PPP frame is then added to a UDP-type frame assigned to port 1701. UDP, which is part of the TCP/IP suite, qualifies for IPSec to secure the contents; thus, the contents of L2TP are secure, regardless of the originating protocol and/or data type.

One additional benefit of the L2TP method is that you have a choice of additional encryption algorithms for securing the data.

What's New with Windows Server 2003 IPSec

The following new features are supported in the Windows Server 2003 IPSec implementation:

• IPSec in Windows Server 2003 now supports the User Datagram Protocol (UDP) encapsulation of IPSec packets to allow Internet Key Exchange (IKE) and Encapsulating Security Protocol (ESP) traffic to pass through a Network Address Translation (NAT) devicesomething not possible in Windows 2000. It is now possible for Windows 2000 and Windows XP clients to establish IPSec connections with a Windows Server 2003 server that is located behind one or more NAT devices.

• The IP Security Monitor is now implemented as an MMC snap-in instead of as a standalone executable, as was the case in Windows 2000. Also, you can now monitor information about local and remote computers as well as several other enhancements.

• IPSec now supports the use of a 2048-bit Diffie-Hellman key exchange. As a result, the secret key resulting from the Diffie-Hellman exchange has greater strength. This results in a longer key length, which increases the difficulty an attacker faces when trying to determine a secret key.

• You can administer and control IPSec from the command line with new extensions to the netsh command. Using the netsh ipsec context, you can configure static or dynamic IPSec main mode settings, quick mode settings, rules, and configuration parameters. The netsh ipsec context replaces the Ipsecpol.exe tool that was provided with the Windows 2000 Server Resource Kit.

• IPSec now provides stateful filtering of network traffic during computer startup. Any outbound traffic initiated by the computer upon startup is permitted, as is any inbound reply traffic. Dynamic Host Configuration Protocol (DHCP) is exempt from this new protection provided by IPSec and is thus allowed during startup. You can also specify other types of traffic you want to exempt. Computer startup security can be configured only by using the netsh command with the ipsec context.

• Windows Server 2003 Enterprise Edition and Windows Server 2003 Datacenter Server Edition provide improved support for integration of IPSec with network load balancing. This allows a group of NLB servers to better provide highly available IPSec-based VPN services to clients. NLB can now accurately track IPSec-secured sessions, and the IPSec Internet Key Exchange (IKE) protocol can now detect when an IPSec-secured session is being established with a cluster server and quickly recover from a failover.

• IPSec provides an extension to the Resultant Set of Policy (RSoP) snap-in (another new addition to Windows Server 2003) that can be used to view IPSec policy assignments of computers or other Active Directory objects.

The following are the standard features of the Windows Server 2003 IPSec implementation:

• IPSec in Windows Server 2003 is policy based. It cannot be configured without an IPSec policy being in place, allowing an administrator to more easily apply settings to groups of objects such as computers or users.

• IPSec on Windows Server 2003 can use Kerberos v5, a digital certificate, or a shared secret (string) for user authentication.

• IPSec mutually authenticates computers prior to any data being exchanged.

• IPSec establishes a security association (SA) between the two host computers involved in the data transfer. An SA is the collection of a policy and keys, which define the rules for security settings.

• IPSec encrypts data using Data Encryption Standard (DES) or Triple DES (3DES).

• IPSec uses the MD5 or SHA1 algorithm for data hashing.

• IPSec is invisible to users. IPSec operates at the network level of the Open System Interface (OSI) model; therefore, users and applications do not directly interact with the protocol. After an IPSec tunnel has been created, users can connect to applications and services as if they were on the local network and not on the other side of a public network.

IPSec operates at the network layer; therefore, it is invisible to applications and computers. An understanding of the following features, however, will help you troubleshoot problems that may arise in connectivity:

• IPSec policies are part of Group Policy, both locally and within Active Directory. This built-in feature allows changes and management to be centralized. Settings for IPSec are enforced on the computer as the policy is enforced.

• The Internet Security Key Association Key Management Protocol (ISAKMP) monitors the negotiations between the hosts and provides the keys to use with security algorithms.

• The installed IPSec driver secures traffic between the two hosts.

Now that you have a basic introduction to the benefits and background of the IPSec protocol, you are ready to look at working with IPSec.