Lesson 1: Designing IPSec Policies | MCSE Training Kit (Exam 70-220): Designing Microsoft Windows 2000 Network Security: Designing Microsoft(r) Windows(r) 2000 Network Security (IT-Training Kits)

An understanding of the IPSec process will help you make the following IPSec policy design decisions:

Decide which IPSec protocols to use.
Decide whether to implement IPSec transport mode or IPSec tunnel mode.
Design IPSec filters that identify which packets to protect with IPSec.
Determine which actions will take place if the packets meet the IPSec filter criteria.
Determine which encryption levels will be used if packets meet the IPSec filter criteria.
Design how computers using IPSec protection authenticate each other.

After this lesson, you will be able to

Design IPSec policies to protect data transmitted between Windows 2000–based computers

Estimated lesson time: 75 minutes

Before discussing the design of IPSec policies, let's examine how IPSec communications take place between two IPSec-enabled computers.

Describing IPSec Communications

IPSec implements encryption and authenticity at a lower level in the Transmission Control Protocol/Internet Protocol (TCP/IP) stack than application-layer protocols such as Secure Socket Layer (SSL) and Transport Layer Security (TLS). Because the protection process takes place lower in the TCP/IP stack, IPSec protection is transparent to applications. The applications require only a recognized port to be protected by IPSec. If the application uses random ports for all connections, it's almost impossible to define an IPSec filter that identifies the application's network data streams.

The application doesn't have to be IPSec-aware because the data transferred between the client and the server is normally transmitted in plaintext. The IPSec process encrypts the payload after it leaves the application at the client and then decrypts the payload before it reaches the application at the server.

For example, Telnet protocol transmits authentication and application data in plaintext. Figure 12.3 illustrates a scenario where both the client and server are configured to negotiate IPSec security whenever transmissions are sent using the Telnet protocol. At no point in this entire sequence of events is the Telnet protocol aware of the encryption process. The process is as follows:

click to view at full size.

Figure 12.3 The IPSec process

The Telnet client sends a packet destined to the Telnet server. The packet is sent from a random source port at the client, but the data is always sent to TCP port 23 on the Telnet server.
The IPSec driver at the client computer intercepts the packet when it reaches the IP layer and compares the packet to the list of IPSec filters configured at the client. In this case, the filter matches TCP 23. This match initiates a negotiation between the client and the server that will establish a security association (SA) between the client and server. A security association defines the details of the IPSec session between the client and the server. The SA defines the IPSec protocol, encryption and integrity algorithms, and the authentication protocol for the connection.
The IPSec driver forwards the packet to the Internet Security Association and Key Management Protocol (ISAKMP) to negotiate a SA between the client and server. ISAKMP is used to determine the definition of the SA that's established between the client and the server.
The client and server proceed with the ISAKMP process using the Internet Key Exchange (IKE) protocol by connecting using User Datagram Protocol (UDP) 500. The ISAKMP process establishes the necessary SAs between the client and server. The SA includes the protocol and the encryption algorithms used to protect the data transmitted between the client and the server.
The results of the SA are returned to the IPSec driver so that the IPSec driver can perform all necessary tasks and make any security modifications to the data before it's transmitted from the client to the server.
The IPSec driver applies the required encryption or integrity algorithm or both to the data and then sends the data to the network interface card (NIC) for transmission to the client.

NOTE
The process of encrypting and decrypting IPSec-protected packets can be processor intensive. To increase performance, consider purchasing IPSec-aware network cards (NICs) that will offload the IPSec encryption process from the computer's processor. Currently, IPSec-aware NICs are available from Intel and 3Com.

Planning IPSec Protocols

IPSec provides two protocols for protection of transmitted data, Authentication Headers (AH) and Encapsulating Security Payloads (ESP). In their simplest form, AH provides authentication and integrity services to transmitted data while ESP provides encryption services. Be aware that AH and ESP are separate protocols. You can use them individually or you can combine them to provide both integrity and inspection protection.

Assessing AH

AH provides authentication, integrity, and antireplay protection to transmitted data on the network. AH doesn't protect transmitted data from being read, but it does eliminate the possibility of the data being modified during transmission. Figure 12.4 shows the composition of an AH packet. It's important to know the fields of an AH packet to understand how the AH packet protects transmitted data.

click to view at full size.

Figure 12.4 IPSec AH header fields

The fields in an AH packet include

Next Header. This field indicates the protocol ID for the header that follows the AH header. For example, if the encrypted data is transmitted using TCP, the next header value would be 6, the protocol ID for TCP.
NOTE
You can find a partial list of protocol IDs in the systemroot\system32\drivers\etc\protocol configuration file, where systemroot is the folder where Windows 2000 is locally installed on your computer.
Length. This field contains the total length of the Authentication Header.
Security Parameters Index (SPI). This field identifies the security association that was negotiated in the ISAKMP protocol exchange between the source computer and the destination computer.
Sequence Number. This field protects the AH-protected packet from antireplay attacks. For each packet issued for a specific SA, the sequence number is incremented by one to ensure that each packet is assigned a unique sequence number. The recipient computer verifies each packet to ensure that a sequence number hasn't been reused. The sequence number prevents an attacker from capturing packets, modifying them, and then retransmitting the packets later.
Authentication Data. This field contains the hash created against the signed portion of the AH packet. The hash is the result of the integrity algorithm applied to the AH packet to determine if the packet has been modified. The recipient performs the same integrity algorithm and compares the result of the hash algorithm with the result stored within the authentication data field to ensure that the signed portion of the AH packet hasn't been altered in transit.

Deploying AH

AH packets are used to authenticate computers involved in data transmissions and to provide integrity to the transmitted packets so an attacker can't modify or replay the transmitted data.

Use AH when communications must be restricted to specific computers in a workgroup or project. AH ensures that mutual authentication takes place between the participating computers so that only authenticated computers can participate in data communications.

The advantage of AH is that it allows mutual authentication capabilities to protocols that don't support mutual authentication. If the authentication process is moved lower in the network protocol stack, all applications can support IPSec.

IMPORTANT
AH is supported only by Windows 2000 clients in a Microsoft networking environment. If you want to deploy mutual authentication in a mixed network containing Windows 98, Windows NT 4.0, and Windows 2000–based clients, consider using SMB signing as an alternative.

Assessing Encapsulating Security Payloads (ESP)

ESP packets are used to provide encryption services to transmitted data. In addition, ESP provides authentication, integrity, and antireplay services.

NOTE
When designing an IPSec solution, you can combine AH and ESP protocols in a single IPSec SA. While both AH and ESP provide integrity protection to transmitted data, AH protects the entire packet from modification while ESP protects only the TCP/UDP header and the data payload from inspection.

The encryption provided by ESP encrypts the TCP or UDP header and the application data included within an IP packet. The encryption normally doesn't include the original IP header unless IPSec tunnel mode is used (discussed in the section "Planning IPSecModes," later in this chapter).

As with the composition of an AH header, knowing the composition of an ESP packet helps you understand how ESP protects the contents of a transmitted data packet. An ESP packet contains an ESP header, an ESP trailer, and an ESP authentication trailer, as shown in Figure 12.5.

click to view at full size.

Figure 12.5 IPSec ESP fields

The ESP header has two fields that are inserted between the original IP header and the TCP or UDP header from the original packet:

Security Parameters Index (SPI). This field identifies the SA that was negotiated between the source computer and the destination computer for IPSec communication. It's the combination of the SPI, the IPSec protocol (AH or ESP), and the source and destination IP addresses that identifies the SA used for the IPSec transmission within the ESP packet.
Sequence Number. This field protects the SA from replay attacks. This field is increased incrementally to ensure that packets are never received more than once. If a packet is received with a previous sequence number, the packet is dropped.

The ESP trailer is inserted after the application data from the original packet and includes the following fields:

Padding. This field is a variable length between 0 and 255 bytes that brings the length of the application data and ESP trailer to a length divisible by 32 bits so that they match the required size for the cipher algorithm.
Padding Length. This field indicates the length of the padding field. After the packet is decrypted, this field is used to determine the length of the padding field.
Next Header. This field identifies the protocol used for the transmission of the data, such as TCP or UDP.

Following the ESP trailer, the ESP protocol adds an ESP authentication trailer to the end of the packet. The ESP authentication trailer contains a single field:

Authentication Data. This field contains the Integrity Check Value (ICV) and a message authentication code. The two components function as a digital signature that verifies the originating host that sent the message and ensures that the packet wasn't modified in transit. The ICV uses the defined integrity algorithm to digitally sign the packet. The integrity algorithm is applied to the ESP header, the TCP/UDP header, the application data, and the ESP trailer.

IMPORTANT
The ICV isn't applied to any mutable fields in the ESP header, TCP/UDP header, application data, or ESP trailer. A mutable field is any field that changes value during transmission. For example, the value of the Time To Live (TTL) field decreases by one for every router that it crosses. If this field were included in the ICV, the ICV would be invalidated every time an ESP packet crossed a router.

ESP provides integrity protection by signing the ESP header, the TCP/UDP header, the application data, and the ESP trailer. ESP also provides inspection protection by encrypting the TCP/UDP header, the application data, and the ESP trailer.

Deploying ESP

ESP provides encryption services for transmitted data. ESP is necessary when the application doesn't recognize application-layer security, such as SSL. Because the IPSec encryption and decryption process takes place at the IP/IPSec layer, the application doesn't have to support IPSec. In fact, the application is unaware that IPSec protection is taking place, as illustrated in Figure 12.6.

Only operating systems and network devices that support IPSec can apply ESP encryption. If an operating system or network device doesn't support IPSec, either the IPSec SA must allow plaintext or you must implement an alternative encryption process.

click to view at full size.

Figure 12.6 IPSec encryption not evident in applications

In addition to encryption support, ESP provides digital signing of the transmitted data. The only difference between AH and ESP is the portion of the data packet that's protected against modification. While AH protects the entire packet, ESP signing doesn't include protection for the IP header used to route the packet through the network. If you want to encrypt the data transmissions and ensure that all fields in the packet are protected, you must configure the SA to implement both IPSec AH and ESP protocols.

NOTE
To allow IPSec traffic to pass through a firewall, you must allow packets using UDP 500 and a protocol identifier (ID) of 51 for AH or a protocol ID of 50 for ESP to pass through the firewall. In addition, the firewall must not be performing Network Address Translation (NAT). IPSec packets can't pass through a NAT because the fields modified by the NAT process are protected by IPSec and can't be modified without invalidating the packet.

Making the Decision

The decision to use AH, ESP, or a combination of AH and ESP in your security design depends on the requirements for IPSec protection. Include AH in your IPSec design to meet the following security objectives:

To protect the entire packets against modification. AH digitally signs the entire packet, including the original IP header, to ensure that the packet isn't modified during transmission. Use AH to sign the packet when your business requirements require protection of the entire packet to prevent attacks that attempt to modify the IP header and reroute packets on the network.
To provide mutual authentication of both client and server. IPSec using AH requires that both hosts involved in the data transmission authenticate with each other. Authentication takes place between the hosts and not between the users working at the hosts.
To limit communications to authorized computers for a project. AH requires mutual authentication of the hosts involved in a data transmission. If the hosts can't negotiate a SA, communications won't take place. You can use AH to ensure that only authorized hosts can communicate with each other.

Use ESP in your IPSec security design to meet the following security objectives:

To protect the application payload from being observed during transmission. ESP packets encrypt the original data payload to ensure that the packet's contents aren't visible when transmitted across the network.
To protect the TCP/UDP header and application data from modification during transmission. ESP can apply a digital signature to the data packet, but it doesn't protect the entire packet from modification. Only the ESP header, the TCP/UDP header, the application data, and the ESP trailer are protected from modification.

WARNING
IPSec using ESP may lead to a firewall losing the ability to inspect data as it's transmitted through the firewall. To pass ESP-protected traffic, you must configure a firewall to allow connections to UDP port 500 and protocol ID 50. Unfortunately, this allows all ESP-protected data to pass. There's no way to determine which protocol is encrypted within the packet. The inability to determine which protocols are encrypted can lead to unauthorized traffic passing through the firewall if both hosts can establish an IPSec connection.

Finally, apply both AH and ESP when you require encryption of transmitted data and protection of the entire packet against modification. To ensure total protection of transmitted data, you can negotiate a SA that requires both AH and ESP.

Applying the Decision

Fabrikam must design IPSec protocols for two distinct IPSec connections: the data collection software and the network link to A. Datum Corporation.

For the data collection software, the security specifications include requirements for protecting the data from inspection, proving the authenticity of the sender, and ensuring that data packets aren't modified during transmission. To meet these requirements, you must apply both AH and ESP protection to each packet. While ESP provides the ability to sign the data portion of a packet, it doesn't protect the entire packet against modification. Only AH protects the entire packet in this manner. Likewise, you must configure ESP to allow the data payload to be encrypted as it's transmitted from the client to the server.

For the network link between A. Datum Corporation, the security requirements specify only protection from inspection. Therefore, you only need to use ESP to encrypt all data transmitted over the Internet between the two networks.

Planning IPSec Modes

You can configure IPSec to use one of two modes: transport mode or tunnel mode. In some cases you require IPSec protection from the issuing client all the way to the destination server, as illustrated in Figure 12.7. This mode is referred to as IPSec transport mode.

click to view at full size.

Figure 12.7 IPSec transport mode providing end-to-end protection

You protect the data sent between the two hosts by using AH, ESP, or both. The transmitted data is protected for the entire path between the two hosts.

In IPSec tunnel mode, shown in Figure 12.8, data is protected only between the two defined tunnel points or gateways. IPSec tunnel mode provides what is known as gateway-to-gateway protection of transmitted data.

click to view at full size.

Figure 12.8 IPSec tunnel mode providing gateway-to-gateway protection

When the data is transmitted between the client and the server, it's sent in an unprotected state until it reaches the initial gateway. Then the protection specified in the SA, AH, or ESP is applied to the packets as they're transmitted to the destination network. The decryption and verification process takes place at the receiving gateway. Finally, the data is transmitted in plaintext to the destination host.

You commonly use tunnel mode when packets must traverse a public or unsecured network between two hosts. Tunnel mode is appropriate when it's unnecessary to protect the data at the local networks.

How Does Network Address Translation (NAT) Affect IPSec Configuration?
IPSec protected data can't pass through a firewall or perimeter server that performs NAT. This is because of the NAT process and its effect on fields protected by the AH and ESP protocols.

When a packet passes through a NAT service, the original source IP address is translated into a common IP address and the source port is translated to the actual port established by the server performing the translation, as illustrated in Figure 12.9.

Figure 12.9 The NAT process

The client sends a packet with the source address of the client computer (192.168.5.2) and the source port used at the client computer (TCP port 2086). When the packet passes through the NAT service, the source IP address and port are translated to 131.107.2.5 port TCP 1067.

If the packets are protected with IPSec, the TCP/UDP header field is always protected by AH and ESP packets. The port information that's translated can't be changed without invalidating the packet in the case of AH-protected data and can't be read when implementing ESP-protected data.

You can identify NAT-protected networks by recognizing the use of private network addressing as defined in Request for Comment (RFC) 1918. In RFC 1918 the following IP address ranges have been reserved for private network usage and aren't in use on the Internet:

10.0.0.0 – 10.255.255.255

172.16.0.0 – 172.31.255.255

192.168.0.0 – 192.168.255.255

If you see these addresses in use on a private network, some form of NAT is probably being used to protect the data that is transmitted to the Internet. NAT only takes place when a private network address space is connected to a public network address space. NAT isn't performed if the two network segments attached to the NAT server or firewall both use private network addressing.

Examining Tunnel Mode Packets

IPSec tunnel mode packets differ from transport mode packets in that a new IP header is added to the packet as it's transmitted between gateways. In Figure 12.10, the AH tunnel mode packet inserts the AH between the new IP header and the original IP header. There's no difference in the fields included in the Authentication Header.

click to view at full size.

Figure 12.10 AH tunnel mode packet fields

Likewise, an ESP tunnel mode packet, shown in Figure 12.11, varies from an ESP transport mode packet in the location of the ESP header. As with the AH tunnel mode packet, the ESP header is placed between the new IP header and the old IP header.

click to view at full size.

Figure 12.11 ESP tunnel mode packet fields

The fields included in the ESP header don't vary between transport mode and tunnel mode. The only difference is the location of the ESP header in the tunnel mode packet and the protection of the original IP header information.

Making the Decision

Table 12.1 shows the design decisions you need to make when choosing between IPSec transport mode and IPSec tunnel mode.

Table 12.1 Determining When to Use IPSec Transport Mode or Tunnel Mode

Use	Under These Circumstances
IPSec transport mode	Communications are taking place within a private network or on a single LAN segment where inspection of transmitted data must be prevented. NAT isn't being performed on the packets as they're transmitted from the source computer to the destination computer. Data must be encrypted over the entire path from the source computer to the destination computer. The connection will be between only two computers.
IPSec tunnel mode	Data must be protected when being transmitted over a public portion of the network. There's no risk of data inspection in the private segments of the network. Encryption can only take place between perimeter servers to avoid passing through a firewall or perimeter server implementing NAT.

Use

Under These Circumstances

IPSec transport mode

Communications are taking place within a private network or on a single LAN segment where inspection of transmitted data must be prevented.

NAT isn't being performed on the packets as they're transmitted from the source computer to the destination computer.

Data must be encrypted over the entire path from the source computer to the destination computer.

The connection will be between only two computers.

IPSec tunnel mode

Data must be protected when being transmitted over a public portion of the network. There's no risk of data inspection in the private segments of the network.

Encryption can only take place between perimeter servers to avoid passing through a firewall or perimeter server implementing NAT.

Applying the Decision

Fabrikam requires the use of IPSec transport mode for the data collection software and IPSec tunnel mode for the network link between the Fabrikam and A. Datum Corporation.

For the data collection application, all data is being transmitted between the Windows 2000–based laptops and the server at the Washington office. The data must be encrypted as it passes across the network to ensure that no one can read it. The data must be signed to prove its authenticity.

Make sure that an IPSec tunnel can be established for the connection to the A. Datum Corporation office. Based on the IP addressing information provided in the scenario, the firewalls at the two organizations are performing NAT. Note that the IP address ranges used at each site are private network address ranges defined in RFC 1918.

Because NAT is being performed, IPSec tunnel mode is unable to pass through the firewall. To allow IPSec tunnel mode to connect the two networks, deploy a separate dual-homed server at each network to allow you to establish an IPSec tunnel that bypasses the firewall, as shown in Figure 12.12.

click to view at full size.

Figure 12.12 Implementing IPSec tunnel mode between the two organizations

To ensure that only IPSec tunnel mode packets are received at the tunnel servers, configure the IPSec filters to encrypt all traffic that is transferred between the 172.16.0.0/16 and the 192.168.9.0/24 networks using the two tunnel servers as tunnel endpoints.

Designing IPSec Filters

To identify the protocols that are to be protected with AH or ESP protocols, you must define IPSec filters that identify known characteristics of the protocols. For example, Figure 12.13 shows the IPSec filter properties required for protecting Telnet transmissions to a Telnet server.

click to view at full size.

Figure 12.13 An IPSec filter for the Telnet protocol

The IPSec filter is defined to match data transmission from any IP address to the Telnet server's IP address where the source port is any port and the destination port is TCP port 23.

The characteristics that you can use to identify a protocol include

Source address information. The source IP address can be a specific IP address, a specific IP subnet address, or any address.
Destination address information. The destination IP address can be a specific IP address, a specific IP subnet address, or any address.
Protocol type. The protocol ID or transport protocol used by the protocol. For example, Point to Point Tunneling Protocol (PPTP) uses Generic Routing Encapsulation (GRE) packets. GRE packets are identified by their protocol ID, which is Protocol ID 47. Telnet, on the other hand, uses TCP as its transport protocol, so an IPSec filter for Telnet would only define the protocol type as TCP.
Source port. If the protocol were to use TCP or UDP, the source port could be defined for the protected connection. The source port is set to a specific port or to a random port depending on the protocol being defined. Most protocols use a random port for the source port.
Destination port. If the protocol uses TCP or UDP, the protocol uses a specific port at the server to accept transmissions. For example, Telnet configures the server to listen for connections on TCP port 23.

NOTE
A listing of the ports used by common services can be found by going to www.isi.edu/in-notes/iana/assignments/port-numbers/.

To ensure that response packets are also protected by IPSec, you should configure all defined IPSec filters as mirrored filters. A mirrored filter reverses the source and destination information so that response packets that originate at the server are also protected by IPSec when they're sent back to the client.

IMPORTANT
The only time you don't use mirrored rules is when you define filters for IPSec tunnel mode. In this case, you must design separate filters to reflect the tunnel endpoint that is used at each end of the tunnel.

Defining When IPSec Filters Aren't Required
There is only one circumstance in which IPSec filters don't have to be defined to enable IPSec protection in a Windows 2000 network. Whenever the Layer Two Tunneling Protocol (L2TP) is used to establish a virtual private network (VPN), Windows 2000 automatically enables IPSec protection for the L2TP tunnel. You don't have to define IPSec filters in this case because Windows 2000 automatically protects the data transmitted through the L2TP tunnel by enabling IPSec ESP protection.

You can view the negotiated IPSec SA by running the Netdiagcommand line utility from the Windows 2000 Support Tools with the following parameters:
 netdiag /test:ipsec /debug /v 
The command's output shows all existing IPSec SAs when the L2TP tunnel is established including the applied IPSec filter. If other IPSec filters are defined, the output also includes any manually configured SAs and the filters used by those associations. For more information on planning and designing L2TP, see Chapter 13, "Securing Access for Remote Users and Networks."

Determining IPSec Exclusions

While you can use IPSec to protect most protocols, it can't protect some protocols due to the nature or use of the protocols. When designing your IPSec solution, you should be aware of the protocols that can't be protected and why you can't use IPSec to protect them. The protocols include

IP broadcast addresses. You can define IPSec filters only for single recipients of packets. You can't define IPSec filters for packets destined to multiple recipients because SAs must be established between pairs of computers.
Multicast addresses. As with broadcast messages, you can't secure packets sent to multiple recipients. Multicast addresses include all class D IP addresses (224.0.0.0 through 237.255.255.255).
Resource ReSerVation Protocol (RSVP). A host uses the RSVP protocol (protocol ID 46) to request a specific Quality of Service (QoS) from the network. IPSec can protect the protocol for which RSVP is requesting the QoS, but not the RSVP packets used to request QoS settings.
Kerberos. You can use Kerberos to authenticate two hosts participating in an IPSec exchange. Kerberos authentication is protected within the Kerberos protocol and doesn't require IPSec protection.
Internet Key Exchange (IKE). IKE is used to negotiate the SA between two hosts participating in an IPSec transmission. You can't encrypt the negotiation process of IPSec. The negotiation must take place by using plaintext packets that define how future packets will be protected.

Making the Decision

To ensure that the IPSec filters you define meet all of your business needs, consider the following when you design them:

You can assign only one IPSec policy per computer. If you're going to filter several different protocols, create a filter list of all the protocols and input them into a filter listing.
You define policies for computers, not for users. You can define IPSec filters only for computers on the network. It doesn't matter which users are sitting at the computers.
When possible, define the protocol requirements so that explicit filters can be defined. Determine the following attributes for each required filter:
- Source IP address
- Source port
- Destination IP address
- Destination port
- Protocol type

NOTE
In some cases, you only require the protocol type for an IPSec filter. Only when you define the protocol type as TCP or UDP do you need to define the source and destination ports. If the IPSec filter is based on a protocol other than TCP or UDP, you don't have to define the source port and destination port for the filter.

You can't identify IPSec encrypted traffic if it's passed through a firewall. Once ESP is applied to a packet, you can't determine which protocol is encrypted within the packet. This can lead to unwanted traffic being allowed to pass through the firewall if the firewall is configured to pass IKE packets (UDP port 500) and ESP packets (protocol ID 50). Because the data within the packet is encrypted, the firewall can't determine which original protocol was protected by IPSec.
If multiple filters are defined, the most specific filters are evaluated first and the least specific filters are evaluated last. The display order of the IPSec doesn't matter.
Always mirror defined packet filters when using IPSec transport mode. Mirroring enables response packets to be encrypted when they're transmitted back to the source computer. The response packets will have the source and destination information reversed. Mirroring the rules ensures that the responses are also encrypted.
Define an IPSec filter for each direction when defining IPSec tunnel mode connections. You can't use mirrored packet filters for IPSec tunnel mode because a different tunnel endpoint must be defined for traffic heading in each direction.

Applying the Decision

Fabrikam must define two IPSec filter listings, one for each IPSec policy. The first IPSec filter list they must define is for the data collection software. The data collection software requires separate IPSec policies to be implemented for the Windows 2000–based laptops running the client software and for the Windows 2000–based server hosting the server-side application.

Table 12.2 lists the necessary filters for the Windows 2000–based laptops to allow the filters to identify transmissions to the data collection software application running on the server in Washington.

Table 12.2 Client-Side IPSec Filters for the Data Collection Software

Source IP	Source Port	Target IP	Target Port	Transport Protocol
My IP Address	Any	172.16.100.123	5555	TCP

Table 12.3 lists the necessary filters for the Windows 2000–based server to have the filters identify transmissions destined to the server-side of the data collection software running on the server in Washington.

Table 12.3 Server-Side IPSec Filters for the Data Collection Software

Source IP	Source Port	Target IP	Target Port	Transport Protocol
Any IP Address	Any	My IP Address	5555	TCP

Alternatively, if the laptop IP address remains constant, you could set the source IP address to be the IP address of the laptop or limit the connections to the Albuquerque office by accepting only packets with a source IP address in the 172.17.0.0/16 network address range.

You must establish filters at the two tunnel servers to ensure that IPSec ESP tunnel mode is used for all communications between the A. Datum Corporation office and Fabrikam's Washington office. Because tunnel mode is being used, you must define two filters at each of the tunnel servers. The first filter will encrypt traffic sent from the A. Datum Corporation network (192.168.9.0/24) to Fabrikam's Washington office (172.16.0.0/16) with the tunnel server at the Fabrikam office (131.107.2.2) as the tunnel endpoint. The second filter will encrypt traffic sent from Fabrikam's Washington office to the A. Datum Cor-poration network with the tunnel server at the A. Datum Corporation office (131.107.100.2) as the tunnel endpoint. To block any other transmissions, you can configure a third filter at each tunnel server to block all other traffic. Table 12.4 shows the filters that you must define at the Fabrikam tunnel server.

Table 12.4 IPSec Filters Required for the Fabrikam Tunnel Server

Source IP	Destination IP	Protocol	Tunnel Endpoint
192.168.9.0/24	172.16.0.0/16	Any	131.107.2.2
172.16.0.0/16	192.168.9.0/24	Any	131.107.100.2

Finally, you must establish filters at the A. Datum Corporation tunnel server to establish an IPSec tunnel mode connection with the Fabrikam tunnel server. Table 12.5 shows the required IPSec filters.

Table 12.5 IPSec Filters Required for the A. Datum Corporation Tunnel Server

Source IP	Destination IP	Protocol	Tunnel Endpoint
192.168.9.0/24	172.16.0.0/16	Any	131.107.2.2
172.16.0.0/16	192.168.9.0/24	Any	131.107.100.2

IMPORTANT
To ensure that all packets destined between the two networks are exchanged through the IPSec tunnel, you must add static routes manually at each tunnel server that defines the route to the other office's network through the IPSec tunnel. The tunnel server at Fabrikam's Washington office must add a static route that defines that the 192.168.9.0/24 network is reachable by the gateway at IP address 131.107.100.2. The tunnel server at the A. Datum Corporation office must add a static route that defines that the 172.16.0.0/16 network is reachable by the gateway at IP address 131.107.2.2.

Designing IPSec Filter Actions

Once you've defined which protocols will be protected with IPSec, you must then define what action is taken if the host sends or receives packets that match an IPSec filter, as shown in Figure 12.14. The IPSec actions that you can take include

Permit. The permit action allows packets to be transmitted without IPSec protection. For example, Simple Network Management Protocol (SNMP) includes support for devices that may not be IPSec-aware. Enabling IPSec for SNMP would cause loss of network management capabilities for these devices. In a highly secure network, you could create an IPSec filter for SNMP actions and set the IPSec action to Permit to allow SNMP packets to be transmitted without IPSec protection.
Block. You use the block action when the protocol that matches the associated IPSec filter should never be allowed to exist on the network. If the associated IPSec filter is matched, all packets with the block action defined are discarded. You can use this filter for Internet-accessible Windows 2000–based computers to prevent common attacks and probes, such as the Finger protocol.
Negotiate Security. The negotiate security action allows an administrator to define the desired encryption and hash algorithms that must be used to secure data transmissions if an IPSec filter is matched. The Negotiate Security action allows an administrator to define the desired encryption and integrity algorithms to secure data transmissions if an IPSec filter is matched.

Figure 12.14 Defining IPSec filter actions

In addition to the three basic actions, you can define settings that define how the Windows 2000–based computer will react if non-IPSec protected data is received and how frequently new session keys are defined to protect the IPSec data. Options include the following:

Accept Unsecured Communication, But Always Respond Using IPSec. You use this option when the IPSec protection is enforced only at the servers, not at the clients. In a typical IPSec deployment, clients are configured to use IPSec if requested by the server but never initiate an IPSec SA. This setting allows the initial packet to be received by the server, which then starts the IKE process to negotiate a SA between the client and the server. While it's riskier to have the initial packet of a data transmission accepted using plaintext, the response packet sent from the server won't be transmitted until a SA is established.
Allow Unsecured Communication With Non-IPSec-Aware Computers. In a mixed network, this option allows non-IPSec aware clients to connect to the server. Windows 2000 clients, if configured to do so, will connect to the server and negotiate an IPSec SA. Non-IPSec aware clients will still be allowed to connect using unprotected data streams.
Session Key Perfect Forward Secrecy. Data encryption for IPSec packets is provided by using session keys. The Perfect Forward Secrecy option ensures that an existing key is never used as the foundation of a new key. All keys must be generated without using existing keys. This reduces the risk of key compromise since previous keys can't be used to determine future keys.

Making the Decision

Table 12.6 shows the design decisions for selecting the IPSec actions to take when an IPSec filter is matched by an incoming data transmission.

Table 12.6 Designing IPSec Actions

Use	Under These Circumstances
Block	You're required to drop all connection attempts using the protocol that matches the IPSec filter assigned the block action. You want to prevent the visibility of commonly attacked ports for an Internet-accessible computer.
Permit	You want to enable unencrypted data transmissions to be accepted by a client or server. You use this action when a protocol must not be encrypted. For example, in mixed environments you might require that all SNMP packets aren't encrypted. By assigning the Permit action to the IPSec filter that defines SNMP packets, you ensure that encryption isn't applied. The protocol allows application-layer protection. For example, you don't need to apply IPSec encryption to HTTP transmissions because HTTP can be encrypted using SSL.
Negotiate	You want to provide IPSec-protected access to authorized computers while blocking unauthorized access if the connecting host fails to authenticate. For example, you might place a server in a Demilitarized Zone (DMZ) and want to access the server using NetBIOS from the internal network but block all NetBIOS connection attempts from the Internet. When you define the Negotiate action, only computers from the internal network can negotiate a SA and use IPsec. You want to define the exact encryption and integrity algorithms that will be used for a SA between two network hosts.
Enable Fallback To No Security	The network contains non-IPSec-aware computers that will require access to a server. You want to use IPSec if possible, but will accept unprotected sessions if the connecting computer doesn't support IPSec.
Accept Unsecured Communication, But Always Respond Using IPSec	The server, rather than the client, is configured to initiate the use of IPSec. This setting allows the client to initially attempt to connect using unprotected transmissions, but allows the server to require IPSec protection.
Session Key Perfect Forward Secrecy	You require maximum security for IPSec session keys. Your security policy requires that previous session keys aren't used to generate new session keys.

Applying the Decision

Fabrikam must now match the IPSec actions to the filters that were previously defined. For the data collection software, the filter action would be set to Negotiate Security so that the client and the server can negotiate the encryption algorithm that will be used to secure the data transmission.

The scenario doesn't state whether the client or server will be used for other purposes. To allow or disallow other protocols, define another filter that's set to be any protocol. If the computers will be used for other purposes, you should set the additional filter action to Permit, so that all other traffic will be used without IPSec protection. If the client or server won't be used for any other purposes, you should set the filter action to Block or Negotiate Security. Negotiate Security is probably a better choice because it allows connections as long as IPSec is negotiated between the client and the server.

The tunnel servers used to create the IPSec tunnel between Fabrikam's Washington office and the A. Datum Corporation office will require two different IPSec actions for the list of filters. The filters that define the traffic that's transmitted between Fabrikam's Washington office and the A. Datum Corporation office require the Negotiate Security action so that ESP tunnel mode is used to encrypt the transmitted data.

In addition, you must establish a separate IPSec filter so that only authorized data received by the external interface card of the tunnel server at each office is allowed. This IPSec filter ensures that the IPSec tunnel mode transmissions—and other IPSec transmissions that are negotiated by the tunnel server—are accepted and all other connections are rejected. Table 12.7 shows the IPSec filters and actions that are required for the tunnel server at Fabrikam's Washington office.

Table 12.7 IPSec Filters and Actions for the Fabrikam Tunnel Server

Source IP	Destination IP	Protocol	Tunnel Endpoint	IPSec Action
192.168.9.0/24	172.16.0.0/16	Any	131.107.2.2	Negotiate Security
172.16.0.0/16	192.168.9.0/24	Any	131.107.100.2	Negotiate Security
Any	131.107.2.2	Any	N/A	Negotiate Security

Finally, you must establish filters and actions at the A. Datum Corporation tunnel server to establish an IPSec tunnel mode connection with the Fabrikam tunnel server. Table 12.8 shows the required IPSec filters.

Table 12.8 IPSec Filters and Actions for the A. Datum Corporation Tunnel Server

Source IP	Destination IP	Protocol	Tunnel Endpoint	IPSec Action
192.168.9.0/24	172.16.0.0/16	Any	131.107.2.2	Negotiate Security
172.16.0.0/16	192.168.9.0/24	Any	131.107.100.2	Negotiate Security
Any	131.107.100.2	Any	N/A	Negotiate Security

Designing IPSec Encryption and Integrity Algorithms

You configure IPSec filter properties to specifically define which algorithms IPSec uses when negotiating security. You can define separate algorithms for AH and ESP-protected data streams, as shown in Figure 12.15.

Figure 12.15 Defining encryption and integrity algorithms

Defining custom settings for IPSec protection enables you to define how IPSec protects transmitted data. If AH protection is required, define Message Digest v5 (MD5) or Secure Hash Algorithm v1 (SHA1) as the integrity algorithm.

If ESP encryption is required, set the digital signing algorithm to be MD5 or SHA1 and the encryption algorithm to be Data Encryption Standard (DES) or Triple DES (3DES).

You can define multiple algorithms for the Negotiate Security action. The listing of algorithms can be ordered so that the security administrator can define the desired IPSec protection while allowing less secure variations that are used only if negotiation fails for the higher-level encryption. For example, the security administrator may desire 3DES encryption of ESP packets. If either the client or server doesn't have the Windows 2000 High Encryption Pack installed, you can define a secondary IPSec security method that allows DES protection. The DES protection will be used only if 3DES protection isn't supported by one of the two hosts involved in the IPSec transmission.

In addition to defining the integrity and encryption algorithms, you can also define when new keys are generated to protect the data. You can define key generation based on the amount of data that's transmitted (in kilobytes) and the lifetime of the key (in seconds). Configuring these options can protect the key from compromise.

Making the Decision

Consider the following design factors when deciding on encryption and integrity algorithms for a SA:

If configuring support for multiple algorithm support, sort the algorithms from strongest to weakest. The listing will be processed sequentially from top to bottom. By ordering the list in this manner, you ensure that the strongest mutual algorithms are always used.
IMPORTANT
For integrity algorithms, SHA1 is considered stronger than MD5, and for encryption algorithms, 3DES is considered stronger than DES.
Include security methods only for the required algorithms. If your IPSec design requires only AH protection, don't include ESP algorithms in the listing of supported security methods. This could result in the establishment of an unsupported IPSec SA.
Use of strong encryption protocols such as 3DES requires the installation of the Windows 2000 High Encryption Pack. Consider including an additional security method to support DES encryption in case a connecting client doesn't have the High Encryption Pack installed. Alternatively, ensure that all authorized client computers have the Windows 2000 High Encryption Pack installed.
Modify the default key generation settings in higher-security networks. You can configure the generation of new session keys based on the amount of data transmitted and the elapsed time in the transmission. The default settings are 100,000 kilobytes (KB) and every 3600 seconds.

Applying the Decision

In both scenarios Fabrikam will use ESP to protect their transmitted data. The security requirements call for authenticity for the data payload but not for the entire packet. To accomplish this, configure the ESP packets to enable both encryption and integrity algorithms.

Because the scenario doesn't mention whether the Windows 2000 High Encryption Pack is applied, you must make provisions to allow the clients to connect without it. To meet these requirements, establish the negotiation list shown in Table 12.9 for both scenarios to ensure that the highest form of encryption and integrity is achieved during the negotiation process.

Table 12.9 Defining Security Methods for Fabrikam

ESP Encryption Algorithm	ESP Integrity Algorithm
3DES	SHA1
3DES	MD5
DES	SHA1
3DES	MD5

Designing IPSec Authentication

IPSec requires that the two network hosts using IPSec authenticate with each other before entering into SA negotiations. IPSec allows three methods for authenticating the two hosts involved in the SA:

Kerberos. Kerberos authentication leverages the default Windows 2000 authentication mechanism. Kerberos provides a strong form of authentication that's easy to configure because all Windows 2000–based computers are able to authenticate with other Windows 2000–based computers in the forest with no further configuration. You can't use Kerberos as an authentication mechanism between forests.
Certificates. You can use certificate-based authentication to authenticate network hosts using IPSec. To use IPSec, the certificates must be issued from a Certification Authority (CA) that's trusted by the two hosts using IPSec. Certificates provide strong authentication options for network hosts that don't belong to the same enterprise network. Certificate-based authentication requires both hosts using IPSec to have certificates for authentication before entering into an IPSec SA.
Preshared keys. Preshared keys are text strings entered at the two hosts to prove their identities. You should use preshared keys only in circumstances where you can't use either Kerberos or certificates or when you're testing the IPSec filters before deploying the IPSec settings on the network.

Making the Decision

Use Table 12.10 to determine which authentication method to use for a specific IPSec scenario.

Table 12.10 Planning IPSec Authentication Protocols

Use	Under the Following Circumstances
Kerberos authentication	All computers using IPSec are members of the same Active Directory directory service forest. You want to minimize the amount of configuration involved in authenticating hosts yet maintain security for authentication.
Public key authentication	You require strong authentication between hosts not in the same forest. A common root CA exists for the two hosts using IPSec. Each host has a valid machine certificate installed that can be used to authenticate the host. You want to use L2TP/IPSec for a VPN solution. L2TP/IPSec requires a machine certificate to be used for authentication.
Preshared keys	You can't use Kerberos or public key authentication. Some third-party IPSec solutions may not be able to use Kerberos or public key authentication, and the only resort is to use preshared keys. You're testing a new IPSec filter and want to make sure that authentication problems aren't causing the SA's failure. You're establishing an IPSec SA between two hosts and the association will only be between the two hosts. The preshared key is set to be complex and access to the IPSec configuration interface is secured to prevent inspection of the preshared key established between the two hosts.

Use

Under the Following Circumstances

Kerberos authentication

All computers using IPSec are members of the same Active Directory directory service forest.

You want to minimize the amount of configuration involved in authenticating hosts yet maintain security for authentication.

Public key authentication

You require strong authentication between hosts not in the same forest.

A common root CA exists for the two hosts using IPSec.

Each host has a valid machine certificate installed that can be used to authenticate the host.

You want to use L2TP/IPSec for a VPN solution. L2TP/IPSec requires a machine certificate to be used for authentication.

Preshared keys

You can't use Kerberos or public key authentication. Some third-party IPSec solutions may not be able to use Kerberos or public key authentication, and the only resort is to use preshared keys.

You're testing a new IPSec filter and want to make sure that authentication problems aren't causing the SA's failure.

You're establishing an IPSec SA between two hosts and the association will only be between the two hosts.

The preshared key is set to be complex and access to the IPSec configuration interface is secured to prevent inspection of the preshared key established between the two hosts.

NOTE
The choice between using Kerberos or public key authentication generally comes down to ease of configuration. In a Windows 2000 forest, Kerberos is the easiest authentication mechanism to use because it's already implemented at all computers in the forest. You more often use public key authentication when a Public Key Infrastructure (PKI) exists or when IPSec is required between organizations.

Applying the Decision

Fabrikam requires different authentication mechanisms for the two scenarios. For the data collection software, both the client laptop and the server are members of the corp.fabrikam.tld forest. The easiest authentication method that doesn't lessen security would be Kerberos because both computers have accounts in the corp.fabrikam.tld domain.

For the tunnel servers between the two organizations, the securest authentication method to use would be public key authentication. You can't use Kerberos authentication because the two tunnel servers are members of different forests and Kerberos isn't supported in this scenario. To ensure that the certificates for each tunnel server are recognized and trusted by the other organization, you must either acquire the certificate from a public CA such as Verisign or establish cross-certification between the two organization's CA hierarchies.

Lesson Summary

In most cases you have to define custom IPSec policies to meet your business needs. The process of designing the IPSec policy includes defining filters that identify the protocol, defining the IPSec mode and protocol that's to be used, identifying the authentication mechanisms to use, and defining the security algorithms.

You must test the IPSec policies to ensure that the computers continue to function and provide the necessary level of security for the transmitted data.