Lesson 1: IPSec Fundamentals

---

IPSec in the Windows Server 2003 operating system protects networks from active and passive attacks by securing IP packets through the use of packet filtering, cryptography, and the enforcement of trusted communication. IPSec is useful for improving the privacy and integrity of host-to-host, host-to-network, and network-to-network communications. IPSec can also be used as a host-based firewall to harden clients and servers by using packet filtering.

This lesson will discuss the universal, fundamental aspects of IPSec. The information in this lesson definitely applies to computers running Windows Server 2003 and other Windows-based computers. However, it also accurately represents how UNIX, Linux, and other computers would implement the IPSec standards.

See Also 

Chapter 9 covers deploying and troubleshooting IPSec.

After this lesson, you will be able to

• Identify common IPSec usage scenarios.

• Describe the IPSec negotiation process, including the differences between Main Mode and Quick Mode communications.

• List the two protocols used for protecting IPSec communications.

• List the improvements in the IPSec implementation included with Windows Server 2003.

Estimated lesson time: 25 minutes

IPSec Overview

IPSec is a framework of open standards for helping to ensure private, secure communications over Internet Protocol (IP) networks through the use of cryptographic security services. IPSec supports network-level data integrity, data confidentiality, data origin authentication, and replay protection. Because IPSec is integrated at the Internet layer (layer 3), it provides security for almost all protocols in the TCP/IP suite, and because IPSec is applied transparently to applications, there is no need to configure separate security for each application that uses TCP/IP.

IPSec can be used to provide packet filtering, to encrypt and authenticate traffic between two hosts, and to create a virtual private network (VPN). Using these capabilities of IPSec helps to provide protection against:

• Network-based denial-of-service attacks from untrusted computers.

• Data corruption.

• Data theft.

• User-credential theft.

• Administrative control of servers, other computers, and the network.

Besides simply improving security, IPSec can be used to save money by enabling communications between remote offices and remote access clients across the public Internet, rather than more costly dedicated circuits that offer privacy at the physical level.

See Also 

This chapter will provide a basic overview of IPSec's functionality. If you want to know the details, and you have the spare time, read the following RFCs: 3457, 3456, 3281, 3193, 2857, 2709, 2451, and approximately 22 more. You can obtain copies at http://www.ietf.org.

Securing Host-to-Host Communications

You can use IPSec to encrypt and validate the integrity of communications between two computers. For example, IPSec can protect traffic between domain controllers in different sites, between Web servers and database servers, or between Web clients and Web servers. When an IPSec client attempts to initiate a connection to an IPSec server, the client and server negotiate IPSec integrity and encryption protocols. After the IPSec connection is established, the application's data is transported within the IPSec connection.

For example, consider the common scenario of a user downloading e-mail from a server using Post Office Protocol version 3 (POP3). If IPSec is not enabled, the e-mail client software initiates a connection directly to the e-mail server software. The user name and password will be transmitted in clear text, so that anyone with a protocol analyzer such as Network Monitor can intercept the user's credentials. An attacker who has control of a router can modify the contents of the user's e-mail messages as they are downloaded without being detected.

Now consider the same scenario with IPSec enabled. In this case, when the server receives the POP3 request from the e-mail client, it will send a message back to the client requesting an IPSec connection. The client will agree, and IPSec will negotiate encryption and integrity protocols. Then IPSec on the client computer will intercept the e-mail client's network traffic, store it within encrypted IPSec packets, and send the data to the server using TCP/IP. IPSec on the server will receive the packets, decrypt the contents, and pass the e-mail client's original communication to the e-mail server software.

In this IPSec-enabled scenario, neither the e-mail client nor the e-mail server software is aware that the communications were protected by IPSec. Similarly, routers and firewalls between the client and server cannot modify the communications or extract the user's credentials. In fact, the routers and firewalls cannot even determine that the user is downloading e-mail, because the POP3 protocol is completely hidden within the IPSec packets.

IPSec can operate in two different modes: transport mode and tunnel mode. Typically, you should use transport mode to protect host-to-host communications. In transport mode, IPSec tunnels traffic starting at the transport layer, also known as layer 4. Therefore, IPSec in transport mode can encrypt the User Datagram Protocol/Transmission Control Protocol (UDP/TCP) protocol header and the original data, but the IP header itself cannot be protected.

IPSec transports an application's data by adding an IPSec header and trailer to outgoing packets. Depending on the IPSec protocol used, the original contents of the outgoing packets will be encrypted. IPSec's position in the packet when functioning in transport mode is shown in Figure 8.1. The diagram shows IPSec using the ESP protocol. ESP is the most common of the two IPSec protocols because it provides both authentication and encryption. IPSec protocols will be described in more detail later in this chapter.

Figure 8.1: Transport mode IPSec

In the past, IPSec traffic could not pass from a privately numbered network to a publicly numbered network through a Network Address Translator (NAT) server, such as a firewall or proxy server. IPSec could not deal with having the NAT server change the source and destination IP address-to IPSec, this translation was tampering with the packet, and the packet would be rejected. IPSec NAT Traversal (NAT-T) allows IPSec traffic to pass through compatible NAT servers. However, both the IPSec hosts and the NAT server must support NAT-T, and the NAT server must be configured to allow traffic on UDP port 4500. Windows Server 2003, Windows XP Professional, and Windows 2000 support NAT-T as IPSec clients. Microsoft Internet Security and Acceleration (ISA) Server and Windows Server 2003 support NAT-T as a firewall.

See Also 

For more information about NAT-T, refer to RFC 3193.

Planning 

Planning to use IPSec to authenticate or encrypt communications between a private and public network? Make sure your NAT server supports NAT-T. If not, factor the cost of upgrading into the cost of deploying IPSec.

Securing Host-to-Network Communications

IPSec is often used to authenticate and encrypt traffic sent directly between two hosts. However, IPSec can also protect traffic traveling from a single host to an entire network, as illustrated in Figure 8.2. This is most commonly used in remote access scenarios. In the past, many organizations required users to dial in to remote access servers connected to the organization's private network. Today, organizations can eliminate the cost of maintaining dial-in servers by using IPSec to allow remote users to connect to an organization's private network across the Internet. Most security experts agree that IPSec provides a level of security similar to that of dial-up remote access.

Figure 8.2: Remote access with IPSec

As you recall, when you protect traffic sent directly between two hosts, you will almost always use IPSec transport mode. When you protect traffic between a host and a network, or between two networks, you must use IPSec tunnel mode. Although transport mode stores the UDP/TCP header and the application data between an IPSec header and trailer, tunnel mode stores the entire original packet, as shown in Figure 8.3. The IP header, including the source and destination addresses, must be stored within the IPSec packet because the traffic is destined for a computer other than the computer to which the IPSec connection was established.

Figure 8.3: Tunnel mode IPSec

Exam Tip 

It's important that you understand when to use tunnel mode and when to use transport mode. It's simple, really. Use transport mode when you communicate with one computer, and use tunnel mode when you communicate with an entire network. If an exam question asks about encapsulating or tunneling the IP header, that's a clue that it's referring to tunnel mode.

Once again, consider the scenario of a remote user retrieving e-mail from a mail server on a private network. When the user's e-mail client attempts to initiate a connection to the mail server's IP address, IPSec on the client computer detects that traffic is being sent to a network that must be accessed by using IPSec tunnel mode. The client's IPSec then establishes an IPSec connection to the IPSec gateway that provides access to the internal network.

IPSec will then encapsulate the entire packet generated by the e-mail client, including the source and destination IP addresses, the TCP header, and the application's data. IPSec adds a new IP header with the destination address of the IPSec gateway. The IPSec gateway will decrypt the packet, restoring the packet to the original condition it was in when sent by the e-mail client. The original IP header is restored too, including the original source and destination IP addresses. Finally, the IPSec gateway forwards the packet to the mail server.

As with transport mode, the e-mail client is not aware that the communications were protected with IPSec. Unlike with transport mode, the mail server's operating system also is unaware that IPSec was in use, because the IPSec gateway removed the IPSec header and trailer before forwarding the packets to the private network.

Real World

If hosts on two networks are communicating across the Internet and all clients are IPSec enabled, transport mode can be used to encrypt traffic between individual hosts, or tunnel mode can be used to encrypt all traffic sent between the two networks. Naturally, tunnel mode is more convenient because it doesn't require every host to have IPSec enabled-but which is more secure?

Tunnel mode is more secure than transport mode, in theory. Remember, VPNs protect against an attacker trying to capture your traffic, analyze it, and use the information gathered to do something malicious. Imagine that an attacker is capturing IPSec-encrypted packets as they travel between the private networks of two competing businesses. If tunnel mode is used, all the attacker can determine is how much traffic is sent between the networks, and when it is being sent. This information might be useful because the attacker might be able to guess that a sudden increase in traffic volume indicates an impending merger between the companies and then use that information to buy some stock and make an illegal profit.

If transport mode is used, attackers can analyze the total volume of traffic being sent, just as they could with tunnel mode. However, they can also analyze the shape of traffic sent between hosts within the network. By analyzing the shape, they might be able to determine the internal IP addresses of Web and e-mail servers and build a partial map of the private network. Even though they can't see the encrypted contents of the packets, they can examine the lengths of the packets and the communications patterns. Web traffic, for example, can be recognized even when encrypted because Web browsers send multiple, short requests to a Web server, which returns multiple, much longer responses containing the files that make up a Web page. E-mail servers, backup servers, and Active Directory directory service domain controllers can also be identified by attackers analyzing the shape of traffic.

Now, even if an attacker does manage to capture and analyze your traffic, would this information really be useful? Probably not, but I've talked to a few organizations that use this possibility as a justification to avoid VPNs, so I think it's important to understand the risk. While we're at it, a tin foil hat reduces the risk of aliens reading my mind, but you won't see one on my head.

Securing Network-to-Network Communications

IPSec can also be used to connect two remote networks. Before Internet connectivity was common, remote offices were connected with private links provided by communications companies. These links would typically consist of a circuit (such as a T1 in the United States or an E1 in Europe) from each of the remote offices that connected to a switched frame relay network that would carry the traffic over long distances.

Today, many organizations still use private links to connect offices. Private links offer some distinct advantages, most notably predictability and stability. Although the Internet continues to become more reliable, performance factors such as usable bandwidth, latency, and jitter fluctuate unpredictably. Private links dedicate bandwidth to a communication link and always follow the same path-guaranteeing that performance will always stay the same.

IPSec can connect two remote offices across the Internet, providing the same connectivity as a private link using an existing Internet connection. IPSec uses authentication and encryption to reduce the risk of traffic being intercepted; a private link relies on physical security to reduce the risk of eavesdropping. For many, the security provided by IPSec and private links is similar enough that the additional cost of a private link cannot be justified.

However, IPSec does nothing to stabilize the Internet's available bandwidth or latency, nor to improve the reliability. When IPSec is used to connect two offices across the Internet, both offices lease links to a local Internet service provider (ISP). Then administrators at both offices establish an encrypted IPSec tunnel between an IPSec gateway located at each office, as shown in Figure 8.4. To the clients on both networks, the gateways act like standard routers. The clients do not need to support IPSec to make use of the tunnel.

Figure 8.4: A site-to-site IPSec tunnel

Real World
Comparing Security Levels

During the late 90s, when use of the Internet was skyrocketing, I was working at an ISP that offered a service to connect remote offices using VPNs. At the time, most organizations connected remote offices using leased lines, such as T1s, and frame relay connections. They were comfortable with these 'private' connections and extremely wary of allowing their traffic (encrypted or not) to travel across the 'public' Internet.

In my opinion, though, private connections are no more secure than public connections. You still rely on a third party to carry traffic between locations. For private connections, you use one or more communications companies. Any of these communications companies could, in theory, tap into your network and watch the unencrypted traffic pass through. For public connections, one or more ISPs carry your traffic. The primary disadvantage of using the public connection is that you don't know who exactly is carrying your traffic. The dynamic nature of the Internet could cause your traffic to be routed to a third-party ISP. In theory, you might not trust this third-party ISP.

However, whether you use public or private networks as the underlying transport, there is probably nobody who has the access, time, and inclination to sniff your traffic. If your traffic does get re-routed to a third-party ISP, they're not going to realize that interesting traffic is passing through their network. Even if they did, they probably don't have the equipment necessary to intercept your encrypted data stream and piece together enough of your data to crack the encryption and find anything useful.

In a nutshell, there is a small theoretical risk that your data could be compromised whether it crosses a public or private network. I think that risk is about the same either way. However, VPNs are much less expensive than private links, which means you have some money left over to spend on other security priorities, like staffing.

Although both ends of the tunnel can be servers running Windows Server 2003, they do not have to be. IPSec is a set of Internet standards that is supported by a wide variety of operating systems and network devices. One or both of the IPSec tunnel endpoints can be non-Microsoft firewalls, network devices, Windows 2000-based servers, or UNIX/Linux servers.

Negotiating IPSec Connections

Unfortunately, IP was not originally designed with authentication or encryption in mind. As the Internet grew and TCP/IP became the network protocol of choice, this unsecured form of communication became the standard. IPSec allows computers to continue using IP, while adding authentication and encryption.

However, most computers on IP networks today do not have IPSec enabled. As a result, computers with IPSec enabled are usually configured to politely ask remote computers to use IPSec to improve the security of the connection. If the two computers determine that they both have IPSec configured, and can agree upon a set of security standards, they can begin to use IPSec. This process is known as IPSec negotiation.

Not all IPSec negotiations are successful. Often the negotiations will fail because one of the two computers is not capable of using IPSec. Alternatively, the computers might not have the same security protocols enabled, which would mean that they wouldn't be able to agree on a set of standards. In these cases, the computers will either revert to unprotected IP communications or determine that they will not communicate at all if they cannot use IPSec.

Internet Key Exchange (IKE) is the algorithm by which the first secure Security Association, or SA (a secure channel), is negotiated. IKE is a combination of the Internet Security Association Key Management Protocol (ISAKMP) and the Oakley Key Determination protocol and performs a two-phase negotiation: Main Mode and Quick Mode.

See Also 

You can read more about IKE negotiation and this process at RFC 2409: http://www.ietf.org/rfc/rfc2409.txt.

Main Mode

The initial long form of the IKE negotiation (Main Mode or Phase 1) performs the authentication and generates the master key material to establish an ISAKMP SA between machines. The result is referred to as an ISAKMP SA or an IKE SA. After the ISAKMP SA is established, it will remain in place for the period of time defined on the host computers-by default, it will last for 8 hours on computers running Windows. If data is actively being transferred at the end of the 8 hours, the Main Mode security association (SA) will be renegotiated automatically.

Main Mode negotiation occurs in three parts:

1. Negotiation of protection suites

2. Diffie-Hellman exchange

3. Authentication

Part 1 of the Main Mode negotiation uses unencrypted communications to identify the protection suites that are available and determines which algorithms will be used during the session. The IPSec client will send the IPSec server a list of protection suites that the client supports. Each proposed protection suite includes attributes for encryption algorithms, hash algorithms, authentication methods, and Diffie-Hellman Oakley groups. The IPSec server then responds to the client with the chosen protection suite. Generally, this will be the first compatible protection suite.

Security Alert 

A Windows IPSec client will propose protection suites in the order in which they are listed in a filter action. A Windows IPSec server will use the first suitable protection suite listed by the client. Therefore, the Windows client determines the priorities of the protection suites, not the server. You should place this sequence in order from most to least secure. Lesson 3 describes how to configure filter actions.

After a protection suite has been negotiated, part 2 of the Main Mode negotiation generates a Diffie-Hellman public and private key pair based on the negotiated Diffie- Hellman Oakley group. The IPSec hosts exchange public keys and then separately generate the Main Mode master key keying material. This keying material will be used to encrypt the traffic sent between the two hosts, enabling all future communications to be considered private.

Part 3 of the Main Mode negotiation performs authentication. The authentication that occurs for Main Mode negotiation is a computer-based authentication (also known as machine-based authentication). The authentication process verifies only the identity of the computers, not the individuals using the computers when the authentication process occurs. The exact messages exchanged during this phase depend on which authentication method is used. Lesson 2 provides more information about the available authentication modes.

Quick Mode

Quick Mode (also known as Phase 2) IKE negotiation establishes a secure channel between two computers to protect data. Because this phase involves the establishment of SAs that are negotiated on behalf of the IPSec service, the SAs created during Quick Mode are called the IPSec SAs. Two SAs are established, each with its own Security Parameter Index (SPI) label. One IPSec SA is used for inbound traffic, and the other is used for outbound traffic. During Quick Mode, keying material is refreshed or, if necessary, new keys are generated. A protection suite that protects specific IP traffic is also selected.

IPSec hosts will perform IKE Quick Mode negotiation on a regular basis to reduce the risk of an attacker using brute force methods to determine the keys used in the communications. Each renegotiation re-establishes two new IPSec security associations with new keys and SPIs. By default, computers running Windows will perform Quick Mode negotiation every hour (3600 seconds) or after 100 megabytes have been transferred. Either side of the connection can start the renegotiation process. Therefore, the site that first reaches the defined session key limit will initiate renegotiation. Lesson 3 describes how to specify session key limits.

Establishing a handful of IPSec connections is not going to bog your computer down. However, the negotiation process does use a significant amount of processing time. If you have a server that is receiving anywhere from dozens to thousands of incoming IPSec connections per second, monitor the server's processor utilization. If the processor utilization is consistently over 30 percent during peak hours, consider adding a network interface card (NIC) that offloads the IPSec processing. This enables the NIC to handle the negotiations while the server's processor deals with more interesting tasks.

Establishing the IPSec connection is processor-intensive because it uses asymmetric public key cryptography. The data transmitted after the connection is established is encrypted by using symmetric shared key cryptography and does not use a significant (by modern standards) amount of processing capacity. So offloading processor utilization is only going to benefit you if the server is constantly receiving a large number of new connections.

Authentication Header and ESP

IPSec can use two protocols: Authentication Header (AH) and ESP. The protocols can be used either separately or together. AH provides data origin authentication, data integrity, and anti-replay protection for the entire packet, including the IP header and the data payload carried in the packet. Naturally, AH does not provide protection for the fields in the IP header that are allowed to change in transit, such as the hop count. AH does not encrypt data, which means it does not provide privacy. Attackers can read the contents of packets if they can intercept them, but the packets cannot be modified.

ESP is more commonly used than AH because it provides data origin authentication, data integrity, anti-replay protection, and the option of privacy. While AH and ESP can be used together, you will use ESP alone in most circumstances. You should choose AH over ESP only when the data and header in the packet need to be protected from modification and authentication but not encrypted. You might do this if you have an intrusion detection system, firewall, or quality of service (QoS) router that needs to inspect the contents of the packet. Otherwise, take advantage of the privacy provided by encryption, and use ESP. If IPSec traffic must traverse a NAT server, you must use ESP, because ESP is the only IPSec protocol that supports NAT-T.

IPSec in Windows

IPSec is natively available and can be used to protect network communications for Windows 2000, Windows XP Professional, and Windows Server 2003. Additionally, a legacy client is available for Microsoft Windows NT 4.0, Windows 98, and Windows Millennium Edition (ME). You can download the legacy client from http://www.microsoft.com/windows2000/server/evaluation/news/bulletins/l2tpclient.asp.

Windows Server 2003 improvements

Windows Server 2003 includes several IPSec manageability improvements. The IP Security Monitor snap-in, first introduced with Windows XP Professional, replaces the Ipsecmon.exe tool included with Windows 2000 Server. Now, you can use the Netsh tool to manage IPSec configuration from the command line just as you would other network configuration settings, instead of being required to use dedicated command- line tools. Troubleshooting is easier, too, because you can use the Resultant Set Of Policy (RSoP) snap-in to view existing IPSec policy assignments.

IPSec security is improved, too. Windows 2000 Server was vulnerable to network attacks during startup, even if IPSec had been configured. Now, you can configure computer startup security to manage network traffic during startup. This allows only the outbound traffic that the computer initiates during startup, inbound traffic sent in response to the outbound traffic, and Dynamic Host Configuration Protocol (DHCP) traffic. Previous versions of Windows automatically allowed several types of traffic through IPSec filters. Windows Server 2003, by default, only allows IKE traffic.

Previous versions of Windows could reveal more information than necessary to an attacker when public keys were used to authenticate IPSec connections, but Windows Server 2003 can be configured to exclude the name of the certification authority (CA) to prevent exposure of trust relationships. Now, you can use dynamic addressing to specify the addresses of DHCP servers, DNS servers, WINS servers, and default gateways in IP filters. This allows you to create more restrictive filters than you could with Windows XP or Windows 2000 computers, because you do not have to deploy different filters to different locations just because clients used different network configurations.

Packet filtering

Although the primary purpose of IPSec is to ensure the integrity of hosts and to encrypt traffic, the Windows Server 2003 IPSec implementation also provides limited firewall capabilities for end systems. This was extremely important with versions of Windows released prior to Windows XP. However, Windows XP and Windows Server 2003 include Internet Connection Firewall (ICF), which provides more powerful stateful packet filtering than IPSec.

Although IPSec and ICF functionality overlap, they both have unique features. ICF is stateful, and IPSec provides filtering based on source and destination IP addresses. Fortunately, there's nothing to stop you from using both together on computers running Windows XP Professional and Windows Server 2003.

See Also 

For more information about ICF and stateful packet filtering, refer to Chapter 4.

You should enable ICF on computers running Windows XP Professional and Windows Server 2003 regardless of whether you use IPSec. However, to ensure proper IKE management of IPSec SAs, you must configure ICF to permit ISAKMP for UDP port 500. If you are using NAT-T, you must also allow traffic on UDP port 4500. ISAKMP is not one of ICF's pre-configured services, however, so you will need to add it. To add ISAKMP, click the Advanced tab in the filtered network interface's properties dialog box. Then click the Settings button. Click the Services tab, and then click Add. Enter settings in the dialog box as shown in Figure 8.5, and then click OK and repeat the process for the second port number.

Figure 8.5: Allowing the ISAKMP service through ICF

Lesson 3 provides information on configuring packet filtering by using IPSec.

Lesson Review

The following questions are intended to reinforce key information presented in this lesson. If you are unable to answer a question, review the lesson materials and try the question again. You can find answers to the questions in the 'Questions and Answers' section at the end of this chapter.

1. By default, how often does IPSec regenerate Main Mode keys?

   1. Every hour

   2. Every 4 hours

   3. Every 8 hours

   4. Every 24 hours

   5. Weekly

2. Which mode would you use to protect communications between two private networks connected by the Internet?

   1. Transport mode

   2. Tunnel mode

3. Which mode would you use to protect communications between an IPSec- enabled e-mail client and an e-mail server on a private network?

   1. Transport mode

   2. Tunnel mode

4. Which of the following IPSec protocols provides encryption for network communications?

   1. AH

   2. ESP

   3. SA

   4. IKE

   5. ISAKMP

Lesson Summary

• Use IPSec in transport mode to protect communications between two IPSec- enabled computers. Use IPSec in tunnel mode when protecting communications to an entire network.

• Main Mode IKE negotiations occur at the beginning of a session. Quick Mode negotiations occur immediately after Main Mode negotiations complete, and then recur on a regular basis while the session is active.

• You can choose to use either the AH or ESP protocol with IPSec. You will usually use ESP, because ESP provides encryption and is compatible with NAT-T. Use AH only when you specifically do not want to encrypt traffic.

• IPSec can be used to provide packet filtering for computers running Windows, and it compliments ICF by providing filtering based on source or destination IP address.

---