Designing an IPSec Policy

You need to create a set of IPSec policies that matches the needs of your environment. Organizations that have consistent security needs and a more simple network structure can create fewer and less complex policies to meet their goals. Other organizations with more stringent security needs and more complex environments require a greater number of policies and potentially more rules in their policies.

Use the steps shown in Figure 6.9 to help you decide how restrictively secure you want to make your IPSec policies.

Figure 6.9: Designing IPSec Policies

When designing an IPSec policy, you need to decide whether to apply an IPSec policy to protect traffic just for specific paths (between specific IP addresses) or to provide general protection for all traffic sent to and from a specific computer. If you want to use IPSec to provide general protection for all traffic sent to and from a specific computer, configure a policy that is based on blocking all traffic and then permitting exceptions, or permitting all traffic and then blocking particular ports. Blocking all traffic provides greater security, but this approach also requires a detailed analysis of the computer's communications, to ensure that required traffic is not blocked.

Consider a simple scenario that requires a simple IPSec policy strategy. All computers belong to a single Active Directory domain. The environment is composed entirely of servers running Windows Server 2003 and clients running Windows XP. All servers (except domain controllers) request confidentiality by using ESP, and all clients support IPSec. In such a case, only two policies are required: one for clients, and one for member servers. These policies can be applied by using Group Policy at the OU level in Active Directory. Because a different IPSec policy is applied to servers, the server computer accounts are grouped into their own OU. A GPO is created for the OU, and then the IPSec server policy is assigned to that GPO.

Because of the operating systems that the computers run and because all of the computers are domain members, both the IPSec server and client policies require that Kerberos be used for mutual authentication. To ensure that domain controllers do not receive any IPSec policy, the predefined domain security group for domain controllers is denied Read access to the GPO that assigns IPSec client policy at the domain level.

Conversely, many more policies can be required in a more complicated scenario. A single IPSec policy can contain many rules, each rule tailored for a specific type of traffic. Many different IPSec policies might need to be designed, to meet the security requirements different computers. Factors that can increase the number of policies required in your environment include:

• Computer roles. Different servers performing in different roles require different types of security, including some combination of packet filtering, confidentiality, integrity, and tunneling. Meeting the security needs of an environment that includes many servers with different roles and requirements can require hundreds of different configurations per server. Furthermore, servers might serve multiple roles, which complicates role-based configurations.

• Sensitivity of data sent over the network. Different levels of encryption are required to accommodate different security needs. For example, 3DES, which provides greater security than DES, can be used to protect transmission of personnel files stored on a file server.

• Computer operating systems. Some operating systems automatically support IPSec transport mode, but some require special client software to be installed. Depending on their operating system, computers might receive IPSec policy through different means: some by Group Policy, others only through locally installed policies. Additionally, Windows Server 2003 IPSec includes several new features that are not supported by Windows 2000 or Windows XP.

• Domain memberships. Some clients and servers belong to domains in different forests, some to UNIX realms, and some are members of workgroups. The trusts and authentication methods in place between such forests, realms, and workgroups affect which IPSec configurations are available.

• Domain relationships. As above, what IPSec configurations are available and what authentication is possible depends on whether domains are in the same forest or different forests, and on whether those forests trust each other.

If you have a complex environment, use IPSec only where it is truly needed. Although you want to provide an appropriate level of security, use as few policies as possible to minimize the complexity of your system. A simpler system is less likely to produce problems and is also easier to troubleshoot if it does.

After identifying what network segments, communications, and computers you want to secure by using IPSec, you need to specify the settings and rules that make up an IPSec policy. The settings and rules you specify also determine how strictly security is enforced by your IPSec policies. Not only must you put policies in place to meet your security requirements, but you must also ensure that computers on your network have compatible policies that can allow them to negotiate SAs.

General IPSec Policy Settings

General IPSec policy settings must be specified whether you want the policy to provide packet filtering or end-to-end security. Make sure to manage all IPSec policies in a controlled way from design, through testing, and into production. To ensure proper management, include version numbers in policy names. General IPSec policy settings are shown in Table 6.4.

Table 6.4: General IPSec Policy Settings

Setting	Description	Example	For More Information
Name	The name and version number of the policy. If the policy will be imported into a Windows 2000 Active Directory store or a local policy store, limit the name to 62 characters.	ContosoDefDomain v1.23	"Add, edit, or remove IPSec policies" in Help and Support Center for Windows Server 2003
Description	Optional text that describes the policy, includes the name of its administrative owner, and that can aid in management of this system.	Administrative owner_name. This policy blocks certain ports and requests integrity to specific server IP addresses.	"Add, edit, or remove IPSec policies" in Help and Support Center for Windows Server 2003
Policy change poll interval	Specifies the period of time in minutes between polling by the IPSec Policy Agent for changes to existing applied policies. IPSec polling does not detect changes made to domain or OU membership or the assigning or unassigning of policies in a GPO. These changes are detected by the Winlogon service every 90 minutes, by default.	Default of 180 minutes accepted	"Add, edit, or remove IPSec policies" in Help and Support Center for Windows Server 2003
Key exchange settings	Determines how keys are created, and how frequently (in seconds) IKE negotiates an ISAKMP SA. In the IP Security Monitor snap-in, relevant statistics are displayed under Main Mode\IKE Policies.	Default settings accepted	"Configure key exchange settings" in Help and Support Center for Windows Server 2003
Key exchange methods	Determines how identities are protected when keys are exchanged. You can specify which algorithms are used, including Message Digest 5 (MD5) and Secure Hash Algorithm (SHA1) for integrity, DES and 3DES for confidentiality, and the length of the master key. An ordered list of security settings is also specified, so that several settings can be offered during negotiation with the IPSec peer.	Default methods accepted	"Create key exchange security methods" in Help and Support Center for Windows Server 2003

Note

Computers running Windows Server 2003 and Windows XP support the 3DES and DES algorithms and do not require installation of any additional components. However, computers running Windows 2000 must have the High Encryption Pack or Service Pack 2 (or later) installed in order to use 3DES. If a computer running Windows 2000 is assigned a policy that uses 3DES encryption, but does not have the High Encryption Pack or Service Pack 2 (or later) installed, the security method defaults to the weaker DES algorithm. To ensure at least some level of privacy for communication, make sure to allow DES as a fallback option whenever a 3DES setting is applied to a group of computers in case some of them cannot support 3DES.

IPSec Rules

IPSec rules determine which traffic is affected by an IPSec policy and which actions take place when that type of traffic is encountered. Table 6.5 describes the contents of IPSec rules that two computers use to establish a secure, authenticated channel.

Table 6.5: Settings of IPSec Rules

Setting	Description	Setting Example	For More Information
Filter list	Specifies a named list of filters. Each filter in the filter list specifies the types of traffic to which the filter action is applied. Filters can be defined to match specific IP protocols, source and destination TCP and UDP ports, and source and destination IP addresses. The filter list name might include the version number, the last update time, and the administrative owner. Each computer discards the filter list name during policy processing. The description or name for each filter is maintained on each computer during policy processing. Make sure to name each filter.	Source Address: My IP Address Destination Address: 172.16.0.4 Protocol: TCP Source Port: Any Dest Port: 1434 Mirrored: Yes Name: Me to sqlsvr3 TCP* 1434	"Filter list" in Help and Support Center for Windows Server 2003
Filter action	Specifies whether a packet is permitted, blocked, or secured. If packets are to be secured, specifies how they are secured. A list of security methods specifies the security protocol, cryptographic algorithm, and session key regeneration frequency.	Request Security	"Filter action" in Help and Support Center for Windows Server 2003
Authentication methods	One or more authentication methods, which are specified in order of preference. Available options are Kerberos V5, certificate, or preshared key.	Kerberos V5	"Authentication methods" in Help and Support Center for Windows Server 2003
Tunnel endpoint	Specifies whether to use tunnel mode and, if so, the tunnel's endpoint.	172.16.0.5	"Tunnel endpoint" in Help and Support Center for Windows Server 2003
Connection type	Specifies whether the rule applies to LAN connections, remote access connections, or both.	LAN	"Connection type" in Help and Support Center for Windows Server 2003

For more information about IPSec policy rules, see "IPSec policy rules" in Help and Support Center for Windows Server 2003.

Understanding Default IPSec Policies

Windows Server 2003 includes three default IPSec policies that are provided as examples only. Do not use any part of the examples as templates to edit or change when creating your own IPSec policies. Instead, design new custom IPSec policies for operational use. The example policies will be overwritten during operating system upgrades and when IPSec policies are imported (when the import files contain other definitions of the same example policies). The three default IPSec policies are as follows:

• Client (Respond Only). This default policy contains one rule, the default response rule. The default response rule secures communication only upon request by another computer. This policy does not attempt to negotiate security for any other traffic.

• Server (Request Security). This default policy contains two rules: the default response rule and a second rule that allows initial incoming communication to be unsecured. The second rule then negotiates security for all outbound unicast IP traffic (security is not negotiated for multicast or broadcast traffic). The filter action for the second rule allows IKE to fall back to unsecured communication when required. This policy can be combined with the Client (Respond Only) policy when you want traffic secured by IPSec when possible, yet allow unsecured communication with computers that are not IPSec-enabled. If IKE receives a response from an IPSec-enabled client, but the IKE security negotiation fails, the communication is blocked. In this case, IKE cannot fall back to unsecured communication.

• Secure Server (Require Security). This default policy has two rules: the default response rule and a rule that allows the initial inbound communication request to be unsecured, but requires that all outbound communication be secured. The filter action for the second rule does not allow IKE to fall back to unsecured communication. If the IKE security negotiation fails, the outbound traffic is discarded and the communication is blocked. This policy requires that all connections be secured with IPSec. Any clients that are not IPSec-enabled cannot establish connections.

Both the Server (Request Security) and Secure Server (Require Security) default policies allow inbound unsecured communication, so that they can be used with the Client (Respond Only) policy. IPSec policies that are designed for operational use are typically customized to combine these behaviors as needed for a specific computer or network environment. The most secure IPSec configuration is not be represented by the default policies.

The most secure configuration is one that negotiates security and neither receives unsecured traffic nor allows communication with non-IPSec-aware computers. In such a configuration, the client computer policy must have a filter in a rule to initiate secured communication when applications on the client computer attempt outbound connections to the server.

Predefined Filter Lists

There are two predefined example filter lists included with Windows Server 2003:

• All ICMP Traffic. This filter affects any ICMP traffic (IP protocol 1) that is sent and received by using the unicast IP address of any network interface on a computer and all other computers. In Windows Server 2003, this filter matches outbound IP broadcast and multicast traffic when that traffic also uses the ICMP protocol.

• All IP Traffic. This filter affects all IP traffic that is sent and received using the unicast IP address of any network interface on the computer to any destination IP address. Because inbound broadcast and multicast traffic use a multicast or broadcast type for the destination address, the inbound traffic is exempt. All ISAKMP traffic sent over UDP port 500, which is required for establishing IPSec-secured communication, is also exempted from IPSec filtering. For more information, see "Understanding Filter Capabilities" later in this chapter.

Predefined Filter Actions

There are three predefined example filter actions included with Windows Server 2003:

• Permit. All traffic that matches the associated filters in the filter list is permitted. This permit behavior is static, not stateful.

• Request Security. All inbound traffic that matches the associated filters in the filter list is allowed unsecured. All outbound traffic that matches the associated filters in the filter list negotiates security. Traffic is secured if the computer at the other end of the connection supports IPSec with a complementary filter action and filter in its policy. If security negotiation fails to elicit a response from the peer within three seconds, a security association for plaintext traffic is created (this association is known as a soft SA). If the peer responds within three seconds and the security negotiation fails, the communication is blocked. After IPSec SAs are established, all traffic that matches the associated filters is secured in both directions between the two computer IP addresses. This behavior is designed so that servers can request security for all clients, but fall back to unsecured communication with computers that are not IPSec-enabled.

• Require Security. As with the Request Security filter action, all inbound traffic that matches the associated filters in the filter list is allowed unsecured, and all outbound traffic that matches the associated filters in the filter list negotiates security. Unlike Request Security, if IKE fails to receive a response and the security negotiation fails, the communication is blocked. After IPSec SAs are established, all traffic that matches the associated filters is secured in both directions between the two computer IP addresses.

Each of these three filter actions allows inbound unsecured communication. This is not the most secure behavior, nor can it be used to initiate secured communication in some trust environments. These filter actions allow the Server (Request Security) and Secure Server (Require Security) example policies to be used with the Client (Respond Only) example policy. If Kerberos authentication is used, this behavior is appropriate for servers that only communicate with clients in mutually trusted domains. For servers that are in one-way domain trust environments, the client computer policy must have a filter in a rule to initiate secured communication with the server.

It is not appropriate to allow inbound unsecured communication for servers that have an Internet interface, because it can make the servers vulnerable to a denial-of-service attack. Internet attackers can easily spoof IP packets to flood the server, causing a denial of service as the server attempts many IKE negotiations with non-existent source IP addresses. The most secure filter action is one that requires security for all inbound and outbound traffic. To create this filter action, verify that the Accept unsecured communication, but always respond using IPsec and Allow unsecured communication with non-IPsec aware computers check boxes are cleared.

Understanding IP Filters, Filter Actions, and Filter Lists

IP filters are used to specify the type of traffic that is permitted, blocked, or secured by an IPSec policy. For more information about specifying IP filter lists, see "Add, edit, or remove IP filter lists" in Help and Support Center for Windows Server 2003.

Specifying Default Exemptions to IPSec Filtering

By default in Windows 2000 and Windows XP, broadcast, multicast, Kerberos, RSVP, and ISAKMP traffic is exempt from IPSec filtering. In Windows Server 2003, the default filtering exemptions have been removed for Kerberos, RSVP, and multicast and broadcast traffic, but remain for ISAKMP traffic, and inbound multicast and broadcast traffic.

To modify the default filtering behavior for Windows Server 2003 IPSec, you can use the Netsh IPSec context or modify the registry.

To modify the default filtering behavior by using the Netsh IPSec context, use the following command:

 netsh ipsec dynamic set config ipsecexempt value={ 0 |  1 |  2 |  3} 

Depending on which exemptions you want, specify the values as follows:

• A value of 0 specifies that multicast, broadcast, RSVP, Kerberos, and ISAKMP traffic are exempt from IPSec filtering. This is the default filtering behavior for Windows 2000 and Windows XP.

  Use this setting only if required for compatibility with Windows 2000 and Windows XP. However, if Kerberos traffic is exempted from filtering, an attacker can bypass other IPSec filters by using either UDP or TCP source port 88 to access any open port. Many port scan tools will not detect this because these tools do not allow setting the source port to 88 when checking for open ports.

• A value of 1 specifies that Kerberos and RSVP traffic are not exempt from IPSec filtering (multicast, broadcast, and ISAKMP traffic are exempt).

• A value of 2 specifies that multicast and broadcast traffic are not exempt from IPSec filtering (RSVP, Kerberos, and ISAKMP traffic are exempt).

• A value of 3 specifies that only ISAKMP traffic is exempt from IPSec filtering. This is the default filtering behavior for Windows Server 2003.

If you change the value for this setting, you must restart the computer for the new value to take effect.

To modify the default filtering behavior by using the registry, do the following:

1. Under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC, add a new DWORD entry named NoDefaultExempt.

2. Assign this entry any value from 0 through 3.

3. Restart the computer.

The filtering behaviors for each value are equivalent to those noted above for the netsh ipsec dynamic set config ipsecexempt value=x command.

Caution

Do not edit the registry unless you have no alternative. The registry editor bypasses standard safeguards, allowing settings that can damage your system, or even require you to reinstall Windows. If you must edit the registry, back it up first and see the Registry Reference on the Windows Server 2003 Deployment Kit companion CD or at http://www.microsoft.com/reskit.

The following table summarizes the equivalent filters that are implemented if all default exemptions to IPSec filtering are enabled (that is, if NoDefaultExempt is 0). When the IP address is specified, the subnet mask is 255.255.255.255. When the IP address is Any, the subnet mask is 0.0.0.0.

Table 6.6: Equivalent Filters for NoDefaultExempt=0

Source Address	Destination Address	Protocol	Source Port	Destination Port	Filter Action
My IP Address	Any IP Address	UDP	Any	88	Permit
Any IP Address	My IP Address	UDP	88	Any	Permit
Any IP Address	My IP Address	UDP	Any	88	Permit
My IP Address	Any IP Address	UDP	88	Any	Permit
My IP Address	Any IP Address	TCP	Any	88	Permit
Any IP Address	My IP Address	TCP	88	Any	Permit
Any IP Address	My IP Address	TCP	Any	88	Permit
My IP Address	Any IP Address	TCP	88	Any	Permit
My IP Address	Any IP Address	UDP	500	500 [1]	Permit
Any IP Address	My IP Address	UDP	500	500	Permit
My IP Address	Peer IP Address	UDP	4500	4500 [2]	Permit
Peer IP Address	My IP Address	UDP	4500	4500	Permit
My IP Address	Any	46 (RSVP)			Permit
Any IP Address	My IP Address	46 (RSVP)			Permit
Any IP Address	<multicast> [3]				Permit
My IP Address	<multicast>				Permit
Any IP Address	<broadcast> [4]				Permit
My IP Address	<broadcast>				Permit
<AII IPv6 protocol traffic [5]					Permit
[1]In order for IPSec transport mode to be negotiated through an IPSec tunnel mode SA, ISAKMP traffic is not exempted if it needs to pass through the IPSec tunnel first. [2]When IPSec NAT-T is performed, the filter exemption for UDP port 4500 is automatically generated, based on the source and destination IP addresses used during the initial part of the IKE negotiation on UDP port 500. This dynamic permit filter for port 4500 is displayed in the IP Security Monitor snap-in, under Quick Mode\Specific Filters, and in the output for the netsh ipsec dynamic show qmfilter command. [3]Multicast traffic is defined as the class D range, with a destination address range of 224.0.0.0 with a 240.0.0.0 subnet mask, which corresponds to the range of addresses from 224.0.0.0 to 239.255.255.255. [4]Broadcast traffic is defined as a destination address of 255.255.255.255 (the limited broadcast address), or as having the host ID portion of the IP address set to all 1's (the subnet broadcast address). [5]IPSec does not support filtering for IP version 6 (IPv6) packets, except when IPv6 packets are encapsulated with an IPv4 header.

Windows Server 2003 IPSec does not support specific filters for broadcast protocols or ports, nor does it support multicast groups, protocols, or ports. Because IPSec does not negotiate security for multicast and broadcast traffic, these types of traffic are dropped if they match a filter with a corresponding filter action to negotiate security. A filter with a source address of Any IP Address and a destination address of Any IP Address can block or permit all multicast and broadcast traffic. By default (and if the NoDefaultExempt registry key is set to a value of 2 or 3), outbound multicast or broadcast traffic will be matched against a filter with a source address of My IP Address and a destination address of Any IP Address. More specific unicast IP address filters that block, permit, or negotiate security for unicast IP traffic should be configured in the same IPSec policy to achieve appropriate security.

For more information about viewing or modifying filter settings, see "Add, edit, or remove filter actions" and "Select a filter action for a rule" in Help and Support Center for Windows Server 2003.

Linking Filter Actions and Filter Lists

Filter lists and filter actions are linked together to form a rule in an IPSec policy. Although filter lists can be reused in different policies, they cannot be reused in the same policy. Filter actions can be reused by rules in the same policy, and they can be shared among different policies.

You can manage IPSec policy in one of two ways:

• Create a new policy and define the set of rules for the policy, adding filter lists and filter actions as required.

  In this method, you create an IPSec policy first and then add and configure rules. Filter lists (specifying traffic types) and filter actions (specifying how the traffic is treated) are added during rule creation.

• Create the set of filter lists and filter actions, and then create the policies and add rules that combine the filter lists with filter actions.

  In this method, you configure the filter lists and the filter actions. Next, you create IPSec policies and add rules that combine the appropriate filter list with the appropriate filter action. Additionally, you specify authentication methods, connection types, and tunnel settings.

Configuring Firewalls

The most secure firewall configuration is one in which the firewall permits only IKE and IPSec traffic to flow between the specific IP addresses of the peers. However, if these addresses are not static, or if there are many addresses, a less secure configuration might be required to permit IPSec and IKE traffic to flow between subnets.

When a firewall or filtering router exists between IPSec peers, it must be configured to forward IPSec traffic on UDP source and destination port 500, IP protocol 50 (ESP), or IP protocol 51 (AH). If you are using IPSec NAT-T, the firewall or filtering router must also be configured to forward IPSec traffic on UDP source and destination port 4500.

First, to permit IPSec traffic on UDP source and destination port 500, use the following settings to create a firewall filter called Permit ISAKMP traffic on UDP port 500:

• Source address = Specific_IP_address

• Destination address = Specific_IP_address

• Protocol = UDP

• Source port = 500

• Destination port = 500

If you are using IPSec NAT-T, to permit traffic on UDP source and destination port 4500, use the following settings to create a firewall filter called Permit ISAKMP traffic on UDP port 4500:

• Source address = Specific_IP_address

• Destination address = Specific_IP_address

• Protocol = UDP

• Source port = 4500

• Destination port = 4500

The firewall filter must also permit or track fragments for ISAKMP. IKE with certificate or Kerberos authentication requires ISAKMP packets to be fragmented because the IKE protocol uses UDP. ISAKMP messages that are larger than the local interface MTU are automatically fragmented by IP. If only certificate authentication is used, Windows Server 2003 implements a method to avoid IKE message fragmentation. When certificate authentication is used for communication between computers running Windows Server 2003 IPSec and Windows XP IPSec or Windows 2000 IPSec, fragmentation is required.

You must also allow IKE to be initiated from either a source or destination IP address. RFC 2408, Internet Security Association and Key Management Protocol (ISAKMP), specifies that the IKE protocol must be able to negotiate security in either direction. Stateful filtering that allows only one computer to initiate IKE to a responder typically times out and deletes the stateful inbound filter in the firewall. As a result, IKE cannot rekey IPSec security associations, and IPSec connectivity is lost.

During the security negotiation, IKE detects whether IPSec NAT-T is required, and, if so, IKE automatically uses UDP port 4500 as the source and destination port. However, the exact firewall configuration that is required to permit this traffic depends on the location of the network address translator relative to the firewall. Typically, the network address translator modifies the source address and the source port of the IKE UDP header.

To permit IPSec traffic on IP protocol 50 (ESP) or IP protocol 51 (AH), use the following setting to create a firewall filter called Permit IPSec traffic on ESP or AH protocol (50 or 51):

• Source address = Specific_IP_address

• Destination address = Specific_IP_address

• Protocol = 50 or 51

When TCP traffic is protected by IPSec transport mode or tunnel mode, it should not require fragmentation. But UDP traffic that is protected by IPSec might require fragmentation. Therefore, you might need to enable fragment tracking for protocol 50 (ESP) or protocol 51 (AH).

ESP and AH traffic should be allowed to flow in either direction. The upper layer protocols such as TCP will determine when IPSec packets are generated. However, ESP fully encapsulates the TCP header, so stateful filters might not be able to inspect the TCP header to track the TCP connection state and determine when to close the hole. IPSec SA state is controlled by IKE in the encrypted quick mode negotiation. Accordingly, IPSec SA creation and deletion can occur in either direction and cannot be detected by inspection of the IKE negotiation packets. Stateful filtering on the IPSec protocols might impact connectivity. To assess the possible impact, ensure that you test how effectively IPSec protects your application traffic with the appropriate firewall or filtering router.

After the firewall is opened to allow the IKE and IPSec protocols, the firewall might not be able to inspect packets to control which traffic is secured by IPSec. IPSec policy filters determine which traffic IPSec can secure, so if you want only a specific protocol to flow between two peers, you must create IPSec filters that enforce this behavior. Port-specific filters can control the direction in which connections are made. Therefore, if a computer is in a more trusted network (inside the firewall) and you want IPSec to secure traffic only over certain protocols and ports on that computer, do not enable the default response rule in the IPSec policy for that computer. If the default response rule is enabled and an attacker compromises the remote computer, the attacker might be able to modify the IPSec policy to negotiate security for all traffic through the firewall.

Choosing the IPSec Protocol

Windows Server 2003 IPSec protects data sent across the network by encapsulating the original payload with either an IPSec header (ESP or AH) for transport mode or an IPSec header and an additional IP header for tunnel mode. An IPSec protocol is often used to convert many different TCP and UDP ports used by applications and system services into one stream of secure traffic for the purpose of traversing an unsecured network. IPSec policy filters can control what protocols and ports are protected by IPSec.

Depending on which protocol is used, the entire original packet can be encrypted, encapsulated, or both. There are two IPSec protocols, Authentication Header (AH) and Encapsulating Security Payload (ESP).

AH

AH provides data origin authentication, data integrity, and anti-replay protection for the entire packet (both the IP header and the data payload carried in the packet), except for the fields in the IP header that are allowed to change in transit. It does not provide data confidentiality, which means that it does not encrypt the data. The data is readable, but protected from modification.

ESP

ESP provides data origin authentication, data integrity, anti-replay protection, and the option of confidentiality for the IP payload only. ESP in transport mode does not protect the entire packet with a cryptographic checksum. The IP header is not protected. ESP in tunnel mode encapsulates and secures the entire original packet, including the IP header (but not the outer IP header). Windows Server 2003 supports IPSec NAT-T, and ESP is the only protocol that can be used to traverse network address translators. The IKE protocol automatically detects the presence of a network address translator and proposes UDP-ESP encapsulation, rather than ESP, to allow IPSec traffic to pass through the network address translator. If only AH is specified as a security method in the filter action, the security negotiation will fail. ESP supports the option of null encryption when network address translator traversal, and not encryption, is needed.

Use Table 6.7 to choose which protocol to use.

Table 6.7: Choosing IPSec Protocol Types

Protocol	Requirement	Usage
AH	The data and the header need to be protected from modification and authenticated, but remain readable.	Use for data integrity in situations where data is not secret but must be authenticated — for example, where access is enforced by IPSec to trusted computers only, or where network intrusion detection, QoS, or firewall filtering requires traffic inspection.
ESP	Only the data needs to be protected by encryption so it is unreadable, but the IP addressing can be left unprotected.	Use when data must be kept secret, such as file sharing, database traffic, RADIUS protocol data, or internal Web applications that have not been adequately secured by SSL.
Both AH and ESP	The header and data, respectively, need to be protected while data is encrypted.	Use for the highest security. However, there are very few circumstances in which the packet must be so strongly protected. When possible, use ESP alone instead.

Using both AH and ESP is the only way to both protect the IP header and encrypt the data. However, this level of protection is rarely used because of the increased overhead that AH would incur for packets that are already adequately protected by ESP. ESP protects everything but the IP header, and modifying the IP header does not provide a valuable target for attackers. Generally, the only valuable information in the header is the addresses, and these cannot be spoofed effectively because ESP guarantees data origin authentication for the packets. In addition, some IPSec hardware offload adapters do not support the use of AH and ESP on the same packet. Hardware offload network adapters can accelerate IPSec processing by performing hardware offload of IPSec cryptographic functions. If you are using such offload adapters, determine the protocol support that they provide before selecting an IPSec protocol to use.

Selecting IPSec Authentication Methods

Peer authentication is the process of ensuring that an IPSec peer is who it claims to be. By using peer authentication, IPSec can determine whether or not to communicate with another computer before the communication begins. Windows Server 2003 IPSec performs peer authentication, but requires only mutual trust of the identities exchanged. It does not verify that the identity that is received is authorized to use a particular IP address.

Table 6.8 describes the uses of the authentication methods.

Table 6.8: Choosing Authentication Methods

Security Requirement	Authentication Method	Examples
Communication within a Windows Server 2003 or Windows 2000 domain, or between trusted Windows Server 2003 or Windows 2000 domains.	Kerberos V5	Clients accessing a Human Resources database A Web server in a perimeter network connecting to a computer running SQL Server on an internal network
Communication outside of your domain or across the Internet where Kerberos V5 is not available but access to a CA is available.	Public key certificate	Partner organizations using a Web-based CA to access resources on your private network
Communication with systems that do not support the Kerberos V5 protocol and do not have access to a CA.	Preshared key	Windows 98 or Macintosh clients using third-party lPSec implementations UNIX servers on a Windows network

Windows Server 2003 IPSec supports three methods of peer authentication so that computers running different operating systems or that exist in different environments can find a common method when negotiating communication with a peer. An IPSec policy rule associates each IP address in a filter with an authentication method list, so that IKE can determine which authentication method list to use with each IP address. During IKE negotiation, the IKE initiator proposes a list of authentication methods to the IKE responder. The responder must use the source IP address of the initiator to identify which filter controls the IKE negotiation. The authentication method list that corresponds to the filter in the responder's IPSec policy is used to select one authentication method from the initiator's list. The responder then replies to inform the initiator of the agreed upon authentication method. If the selected authentication method fails, IKE does not provide a method for retrying a different authentication method.

Kerberos V5 Authentication

Kerberos V5 is the default authentication standard in Windows Server 2003 and Windows 2000 domains. This method of authentication can be used by any computer in the domain or a trusted domain. Also, because the Kerberos protocol is no longer a default exemption, if you want to enable Kerberos authentication, you must create filters in the IPSec policy that explicitly allow all traffic to your domain controllers.

For IKE Kerberos authentication to be used successfully in Windows 2000 and Windows Server 2003 cross-forest trust configurations, you must use FQDNs when configuring external trusts. In addition, you must configure the IPSec policy on both computers to allow an IKE initiator to communicate to any domain controller in the forest domain hierarchy, so that the initiator can obtain a ticket from a domain controller in the responder's domain.

It is recommended that you use Kerberos authentication in a two-way (mutual) domain and forest trust environment. Although one-way domain and forest trusts are supported, it is not recommended to use Kerberos in this type of trust environment. If you do, you must design the IPSec policy to ensure that the IKE initiator can obtain a ticket from a domain controller in the responder's domain. Also, traffic might be lost if IKE fails to rekey main mode negotiations in the opposite direction. Because an IKE main mode negotiation is performed on demand when quick mode negotiations are rekeyed, when configuring lifetimes in kilobytes (KBs) for the associated filter action, use as high a value as possible to minimize rekey authentication failures. If you are using ESP DES or 3DES for encryption, you can set the lifetime value to as high as 200,000 KB (200 megabytes). If you set a higher value, however, the risk of a sophisticated attacker gaining knowledge of the 56-bit encryption keys increases. If you are using ESP or AH for integrity and authenticity, you can set the value to as high as 2 gigabytes because the key sizes are much larger for MD5 (128-bit) and SHA1 (160-bit).

By default, when Kerberos authentication is used, the Access this computer from the network or Deny this computer access from the networklogon right (defined in Group Policy) is evaluated. This evaluation is only performed by the IKE main mode responder (the computer that receives the ticket and must determine whether accept it). Typically, clients obtain Kerberos tickets to access a server. Likewise, the IPSec policy must contain filters that will trigger the IKE negotiation from the IKE initiator, the client that is requesting IPSec access to the server. The server can then use these logon rights to restrict access to certain client computer security groups. However, if IKE rekeys main mode negotiations, the server does not evaluate these rights.

Public Key Certificate

In Windows 2000 and Windows Server 2003, you can use Certificate Services to automatically manage computer certificates for IPSec throughout the certificate lifecycle. Certificate Services is integrated with Active Directory and Group Policy, and it simplifies certificate deployment by enabling certificate auto-enrollment and renewal and by providing configurable certificate templates. In addition, by publishing the computer certificate as an attribute of the domain computer account, Certificate Services allows you to use IPSec to restrict network access to a server.

Use a public key certificate in situations that include access to corporate resources, external business partner communications, or computers that do not run the Kerberos V5 authentication protocol. This requires that at least one trusted root CA is configured on your network and that client computers have an associated computer certificate.

IPSec supports the use of a variety of third-party X.509 public key infrastructure (PKI) systems, in addition to Windows 2000 Server or Windows Server 2003 Certificate Services. Windows Server 2003 IKE has basic compatibility with several certificate systems, including those offered by Microsoft, Entrust, VeriSign, and Netscape. If you are using a third-party PKI system, the PKI system must be able to issue certificates to computers and store their certificates in the Windows Cryptographic Application Programming Interface (CryptoAPI) computer certificate store.

Note

Certificates obtained from Certificate Services with the advanced option set for Enable strong private key protection do not work for IKE authentication because a personal identification number (PIN) cannot be entered to access the private key during IKE negotiation.

General certificate authentication considerations

IKE's use of certificate authentication is compatible with many different PKI architectures, and IKE places relatively few requirements on the contents of a certificate. Typically, computers that have a common trusted root, or whose certificates can chain through a cross-certification trust relationship, can successfully use IKE authentication. To use certificates for IKE authentication, you define an ordered list of acceptable root CA names in the authentication method. This list controls the certificates that IKE can select and the certificates that IKE will select. If IKE authentication fails, you cannot retry the authentication using a different method. For this reason, before you apply an IPSec policy that can use certificates for authentication, make sure that all target computers have the correct root CA certificates and relevant cross-certificates in the CryptoAPI PKI stores, as well as valid computer certificates. Additionally, to ensure that certificate authentication works as intended, test your PKI infrastructure with various IPSec policy configurations before deployment.

The following sections describe the IKE certificate selection and acceptance process. If you decide to use certificates for IKE authentication, understanding this process and its requirements is integral to ensuring proper deployment.

IKE certificate selection process

When IKE negotiates to use certificates for authentication, the following process is used to select a computer certificate:

1. The list of trusted roots is prepared. This list is the list of the CA root names provided by the peer in the Certificate Request Payloads (CRPs), and it matches the CA root names configured in the list of trusted roots in the appropriate authentication method of the IPSec policy. If there are no matching CA root names, all trusted CA root names from the appropriate authentication method are used.

2. IKE searches the computer store for an IPSec certificate that chains to any of the trusted CA roots identified in step 1. An IPSec certificate contains an Enhanced Key Usage (EKU) attribute with a value equal to the IP security IKE intermediate object identifier (OID) 1.3.6.1.5.5.8.2.2.

3. For each certificate chain found, the following checks are performed to verify that:

   • The certificate chain does not have any trust errors.

   • The certificate chain is not a root-only chain.

   • The computer certificate has a private key.

   • The computer certificate has an RSA type public/private key pair.

   • The computer certificate has a public key length that is greater than 512 bits.

   • The computer certificate has a Digital Signature key usage.

   • The computer certificate is not a CA signing certificate that is used to issue certificates.

   • The certificate chain passes certificate revocation list (CRL) checking, which is performed by default or if the value of the StrongCrlCheck registry key is set to 1 or 2. For more information about CRL checking, see "CRL Checking" later in this chapter.

   If all of these checks succeed, IKE selects the certificate chain to be sent to the IPSec peer. If any of these checks fails, IKE continues to search for another IPSec type certificate, using the same list of root CA names.

4. If a valid computer certificate chain is not located, IKE retries the process, from step 2. Although the same list of root CA names is used, IKE does not search for an IPSec type certificate.

5. If a valid computer certificate chain is still not found and if the list of root CA names in step 1 is a subset of the names allowed by the local IPSec policy, IKE retries, from step 2. This time, IKE uses the entire list of root CA names allowed by the local authentication method.

   This step is required for successful authentication when cross-certificates are used to establish trust relationships.

6. After IKE selects a computer certificate, it includes all intermediate certificates in the chain up to the root, except for the root CA certificate. A certificate chain in PKCS#7 format is then sent to the IPSec peer. If there are no intermediate CAs, only the computer certificate is sent.

   If a computer certificate cannot be selected, the authentication fails.

Note

If IKE negotiates with another computer running Window Server 2003, or with other Microsoft IKE implementations that use IPSec NAT-T (such as Microsoft L2TP/IPsec VPN Client), a special method to avoid fragmentation of ISAKMP UDP packets might be implemented. Otherwise, the ISAKMP message that contains the certificate chain will likely be fragmented as the packet is transmitted.

IKE certificate acceptance process

1. IKE receives the peer's certificates or certificate chains and verifies that the peer's certificates chain up to any of the root CAs in the appropriate authentication method of the local IPSec policy.

2. For each peer certificate chain, the following checks are performed to verify that:

   • The computer certificate Subject Name or Subject AltName is consistent with the peer's ID field passed in the IKE negotiation.

   • The computer certificate chain does not have any trust errors. If there is a trust error, the peer authentication fails.

3. If the two checks in step 2 succeed, the following checks are performed for the peer certificate chain to verify that:

   • The certificate chain passes CRL checking, which is performed by default or if the value of the StrongCrlCheck registry key is set to 1 or 2.

   • The computer certificate has an RSA type public/private key pair.

   • The computer certificate has a public key length that is greater than 512 bits.

   • The computer certificate has a Digital Signature key usage.

   If any of these checks fails, the peer authentication fails.

4. If certificate-to-account mapping is enabled in the IPSec policy for the certificate root CA of the peer, IKE calls the Windows secure channel (Schannel) APIs to perform the mapping. Schannel completes the mapping and builds an access token for the computer account. This access token is automatically evaluated against the Access this computer from the network or the Deny this computer access from the network logon right defined in Group Policy Security settings.

   If the logon right evaluation fails, the peer authentication fails.

Editing Default Certificate Templates to Include the Subject Name

The IP Security Monitor snap-in, Event Viewer, and most IPSec diagnostics report the Subject Name attribute of the IPSec peer certificate. In Windows Server 2003 Certificate Services, the default version 2 certificate template for IPSec certificates does not include the computer name in the Subject Name field (the version 1 certificate templates cannot be modified). To aid in deployment and troubleshooting, edit the default IPSec certificate template to include either the FQDN or the common name of the computer in the Subject Name field. To do this, use the following procedure (note that you must be logged on as a member of the Enterprise Admins group or the root domain's Domain Admins group in Active Directory):

• To edit the default IPSec certificate template

  1. Open the Certificate Templates snap-in (Certtmp.msc).

  2. Right-click the certificate template that you want to modify, and then click Properties.

  3. On the Subject Name tab, click Build from this Active Directory Information, and then click Common Name or Fully Distinguished Name.

  4. Enter the common name or FQDN as required.

If you choose to enter the common name of the computer, the name in the Subject Name field of the certificate will appear as CN=< FQDN>. If any FQDN is greater than 64 characters, select Fully Distinguished Name, not Common Name. After you complete this procedure, all subsequently issued computer certificates will contain the FQDN in the Subject Name field of the certificate. If you have already completed enrollment for computer certificates by using the default certificate template, create a new certificate template and configure it to supersede the existing template. Doing this will automatically update all computer certificates so that the blank name in the Subject Name field is replaced with the FQDN of the computer. If the existing certificate template is not superseded, each computer will have two computer certificates that can be used for IKE authentication. IKE can choose either one.

Auditing IKE Negotiation Successes and Failures

You can view the success or failure of IKE negotiations in the Event Viewer security log. To view these events, enable success or failure auditing for the Audit logon events audit policy for your domain or local computer. If auditing is enabled for IKE events, and IKE authentication fails, event 547 is recorded. If IKE authentication succeeds, event 541 is recorded.

When you enable success or failure auditing for the Audit logon events audit policy, IPSec records the success or failure of each main mode and quick mode negotiation and the establishment and termination of each negotiation as separate events. In many cases, it might be necessary to enable success auditing for the Audit logon events audit policy, to track user activity. Keep in mind, however, that enabling this type of auditing can cause the security log to fill with IKE events. You can disable auditing of IKE events by modifying the registry.

To disable auditing of IKE events in the security log, do the following:

1. Set the HKEY LOCAL MACHINE\System\CurrentControlSet\Control\Lsa\Audit\DisableIK EAudits registry setting to a value of 1.

   The DisableIKEAudits key does not exist by default and must be created.

2. Do one of the following:

   • Restart the computer

   • Stop, and then restart the IPSec service by running the net stop policyagent and net start policyagent commands at the command prompt.

   Note

   Stopping and restarting the IPSec service can disconnect all of the computers that are using IPSec from the computer on which the IPSec service is stopped, and it can prevent further communication with that computer. If you restart the IPSec service immediately, TCP-based communication might resume, due to the retransmit behavior of TCP, after new IKE and IPSec SAs are established.

CRL Checking

By default, in Windows XP and Windows Server 2003, IPSec CRLs are automatically checked during IKE certificate authentication, but a fully successful CRL check is not required for the certificate to be accepted. However, if enhanced security is required, a fully successful CRL check is also required. CRL checking can cause delays in authentication or unnecessary failures, and some third-party PKI systems might not support it. You can disable IPSec CRL checking or specify a stronger level of IPSec CRL checking by using the Netsh IPSec context or by modifying the registry.

To disable IPSec CRL checking or specify a different level of IPSec CRL checking by using the Netsh IPSec context, use the following command:

 netsh ipsec dynamic set config strongcrlcheck value={0 | 1 |  2} 

• To disable IPSec CRL checking or specify a different level of IPSec CRL checking

  1. Under HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\PolicyAgent\, add a new Oakley key, with a DWORD entry named StrongCRLCheck.

  2. Assign this entry any value from 0 through 2, where:

     • A value of 0 disables CRL checking.

     • A value of 1 enables the default level of CRL checking for Windows Server 2003 IKE. When this level of CRL checking is performed, certificate validation fails only if the certificate is revoked.

     • A value of 2 enables strong CRL checking, which means that certification validation fails if any error is encountered during CRL processing.

  3. Do one of the following:

     • Restart the computer.

     • Stop, and then restart the IPSec service by running the net stop policyagent and net start policyagent commands at the command prompt.

Caution

Do not edit the registry unless you have no alternative. The registry editor bypasses standard safeguards, allowing settings that can damage your system, or even require you to reinstall Windows. If you must edit the registry, back it up first and see the Registry Reference on the Windows Server 2003 Deployment Kit companion CD or at http://www.microsoft.com/reskit.

Note that IPSec CRL checking does not guarantee that certificate validation fails immediately when a certificate is revoked. There is a delay between the time that the revoked certificate is placed on an updated and published CRL and the time when the computer that performs the IPSec CRL checking retrieves this CRL. The computer does not retrieve a new CRL until the current CRL has expired or until the next time the CRL is scheduled to be published. By default, IKE requests that Crypto API (CAPI) waits 15 seconds to complete the CRL retrieval. If the CRL cannot be retrieved at that time, IKE either ignores the error (if the value of the StrongCRLCheck registry key is set to 1, or it causes authentication to fail (if the value of StrongCRLCheck is set to 2). CRLs are cached in memory and in \Documents and Settings\UserName\Local Settings\Temporary Internet Files by CAPI. Because CRLs persist across computer restarts, if a CRL cache problem occurs, restarting the computer does not resolve the problem. For more information about CRLs, see the Troubleshooting Certificate Status and Revocation link on the Web Resources page at http://www.microsoft.com/windows/reskits/webresources.

Certificate-to-Account Mapping

In Windows Server 2003, a specific group of computers can be authorized to use IPSec when either Kerberos V5 or certificates are used for IKE authentication. This capability enables much stronger peer authentication and IPSec to be used to restrict network access to a server. When you enable IPSec certificate-to-account mapping, the IKE protocol associates (that is, maps) a computer certificate to a computer account in an Active Directory domain or forest, and then retrieves an access token, which includes the list of computer security groups. This process ensures that the certificate offered by the IPSec peer corresponds to an active computer account in the domain, and that the certificate is one that should be used by that computer.

Certificate-to-account mapping can only be used for computer accounts that are in the same forest as the computer performing the mapping. This provides much stronger authentication than simply accepting any valid certificate chain. For example, you can use this capability to restrict access to computers that are only within the same forest. Certificate-to-account mapping, however, does not ensure that a specific trusted computer is being allowed IPSec access.

Certificate-to-account mapping is especially useful if the certificates come from a PKI that is not integrated with your Active Directory deployment, such as if business partners obtain their certificates from third-party providers. You can configure the IPSec policy authentication method to map certificates to a domain computer account for a specific root CA. You can also map all certificates from an issuing CA to one computer account. This allows IKE certificate authentication to be used to limit which forests are allowed IPsec access in an environment where many forests exist and each perform autoenrollment under a single internal root CA. If the certificate-to-account mapping process is not completed properly, authentication will fail and IPSec-protected connections will be blocked. For more information about establishing certificate-to-account mapping, the Distributed Services Guide of the Windows Server 2003 Resource Kit (or the Distributed Services Guide on the Web at http://www.microsoft.com/reskit).

Excluding the CA Name from Certificate Requests

If you use certificate authentication to establish trust between IPSec peers, you can also use Windows Server 2003 to exclude CA names from certificate requests. Excluding the CA name prevents a malicious user from learning sensitive information about the trust relationships of a computer, such as the name of the company that owns the computer and the domain membership of the computer (if an internal PKI is being used). Although excluding the CA name from certificate requests enhances security, computers with multiple certificates from different roots might require the CA root names to select the correct certificate. Also, some third-party IKE implementations might not respond to a certificate request that does not include a CA name. For these reasons, excluding the CA name from certificate requests might cause IKE certificate authentication to fail in certain cases.

Preshared Key

If you are not using Kerberos V5 authentication and do not have access to a CA, a preshared key can be used. For example, a stand-alone computer on a network that does not connect to the Internet might need to use a preshared key, because neither Kerberos authentication through the computer's domain account nor access to a CA on the Internet are available.

A preshared key is a shared secret key that has been agreed upon by administrators who wish to secure the computer's communications by using IPSec. Administrators must manually configure their systems to use the same preshared key.

Important

Microsoft does not recommend the use of preshared key authentication, because the authentication key is stored in plaintext format in the system registry and hex-encoded in Active Directory-based IPSec policy. Well-known methods can enable attackers with access to these data stores to discover weak preshared key values.

Use preshared key authentication only where no stronger method can be used. Using Kerberos or certificate-based authentication is recommended to avoid security risks associated with preshared key authentication.

If you must use preshared key authentication, use only local or persistent IPSec policy, a 25-character or longer random key value, and a different preshared key for each IP address pair. These practices result in different security rules for each destination, and ensure that a compromised preshared key compromises only those computers that share the key. For more information about local or persistent IPSec policies, see "Assigning IPSec Policies Locally" later in this chapter.

IKE Authentication Security

You can use Windows Server 2003 IPSec policies and IKE authentication to limit network access to trusted computers. In most scenarios, successful IKE authentication results in successful network access to a computer. IKE is not aware of the identity or the public key that is expected from the peer. Therefore, if the certificate private key or domain password for a computer were compromised, an attacker might be able to use that computer to successfully authenticate and gain access to another IPSec-protected computer or to conduct trusted man-in-the-middle attacks on IPSec communication. To ensure that IPSec provides the appropriate network access controls, carefully consider the following:

• How computers are joined to an Active Directory security domain.

• How trusts between domains and forests are controlled.

• How computers obtain a certificate from a trusted root CA (or an issuing CA).

• How PKI trust is controlled (for example, how cross certificates are handled).

• How to reduce the number of trusted computers.