Using IPSec

Using IPSec

By its design, TCP/IP is an open protocol created to connect heterogeneous computing environments with the least amount of overhead possible. As is often the case, interoperability and performance design goals do not generally result in security and TCP/IP is no exception to this. TCP/IP provides no native mechanism for the confidentiality or integrity of packets. To secure TCP/IP, you can implement IP Security. IPSec implements encryption and authenticity at a lower level in the TCP/IP stack than application-layer protocols such as Secure Sockets Layer (SSL) and Transport Layer Security (TLS). Because the protection process takes place lower in the TCP/IP stack, IPSec protection is transparent to applications. IPSec is a well-defined, standards-driven technology.

The IPSec process encrypts the payload after it leaves the application at the client and then decrypts the payload before it reaches the application at the server. An application does not have to be IPSec aware because the data transferred between the client and the server is normally transmitted in plaintext.

IPSec is comprised of two protocols that operate in two modes with three different authentication methods. IPSec is policy driven and can be deployed centrally by using Group Policy. To deploy IPSec, you must determine the

  • Protocol

  • Mode

  • Authentication methods

  • Policies

Securing Data Transmission with IPSec Protocols

As mentioned, IPSec is comprised of two protocols: IPSec Authentication Header (AH) and IPSec Encapsulating Security Payload (ESP). Each protocol provides different services; AH primarily provides packet integrity services, while ESP provides packet confidentiality services. IPSec provides mutual authentication services between clients and hosts, regardless of whether AH or ESP is being used.

Using AH

IPSec AH provides authentication, integrity, and anti-replay protection for the entire packet, including the IP header and the payload. AH does not provide confidentiality. When packets are secured with AH, the IPSec driver computes an Integrity Check Value (ICV) after the packet has been constructed but before it is sent to the computer. With Windows 2000 and Windows XP, you can use either the HMAC SHA1 or HMAC MD5 algorithm to compute the ICV. Figure 9-3 shows how AH modifies an IP packet.

figure 9-3 ah modifications to an ip packet

Figure 9-3. AH modifications to an IP packet

The fields in an AH packet include these:

  • Next Header

    Indicates the protocol ID for the header that follows the AH header. For example, if the encrypted data is transmitted using TCP, the next header value would be 6, which is the protocol ID for TCP.

  • Length

    Contains the total length of the AH.

  • Security Parameters Index (SPI)

    Identifies the security association (the IPSec agreement between two computers) that was negotiated in the Internet Key Exchange (IKE) protocol exchange between the source computer and the destination computer.

  • Sequence Number

    Protects the AH-protected packet from replay attacks in which an attacker attempts to resend a packet that he has previously intercepted, such as an authentication packet, to another computer. For each packet issued for a specific security association (SA), the sequence number is incremented by 1 to ensure that each packet is assigned a unique sequence number. The recipient computer verifies each packet to ensure that a sequence number has not been reused. The sequence number prevents an attacker from capturing packets, modifying them, and then retransmitting them later.

  • Authentication Data

    Contains the ICV created against the signed portion of the AH packet by using either HMAC SHA1 or HMAC MD5. The recipient performs the same integrity algorithm and compares the result of the hash algorithm with the result stored within the Authentication Data field to ensure that the signed portion of the AH packet has not been altered in transit. Because the TTL, Type of Service (TOS), Flags, Fragment Offset, and Header Checksum fields are not used in the ICV, packets secured with IPSec AH can cross routers, which can change these fields.

Using ESP

ESP packets are used to provide encryption services to transmitted data. In addition, ESP provides authentication, integrity, and antireplay services. When packets are sent using ESP, the payload of the packet is encrypted and authenticated. In Windows 2000 and Windows XP, the encryption is done with either Data Encryption Standard (DES) or 3DES, and the ICV calculation is done with either HMAC SHA1 or HMAC MD5.

When designing an IPSec solution, you can combine AH and ESP protocols in a single IPSec SA. Although both AH and ESP provide integrity protection to transmitted data, AH protects the entire packet from modification, while ESP protects only the IP payload from modification.

ESP encrypts the TCP or UDP header and the application data included within an IP packet. It does not include the original IP header unless IPSec tunnel mode is used. Figure 9-4 shows how ESP modifies an IP packet.

figure 9-4 esp modifications to an ip packet

Figure 9-4. ESP modifications to an IP packet

The ESP header has two fields that are inserted between the original IP header and the TCP or UDP header from the original packet:

  • Security Parameters Index (SPI)

    Identifies the SA that was negotiated between the source computer and the destination computer for IPSec communication. The combination of the SPI, the IPSec protocol (AH or ESP), and the source and destination IP addresses identifies the SA used for the IPSec transmission within the ESP packet.

  • Sequence Number

    Protects the SA from replay attacks. This field is incremented by 1 to ensure that packets are never received more than once. If a packet is received with a previous sequence number, that packet is dropped.

The ESP trailer is inserted after the application data from the original packet and includes the following fields:

  • Padding

    A variable length from 0 255 bytes that brings the length of the application data and ESP trailer to a length divisible by 32 bits so that they match the required size for the cipher algorithm.

  • Padding Length

    Indicates the length of the Padding field. After the packet is decrypted, this field is used to determine the length of the Padding field.

  • Next Header

    Identifies the protocol used for the transmission of the data, such as TCP or UDP.

Following the ESP trailer, the ESP protocol adds an ESP authentication trailer to the end of the packet. The ESP authentication trailer contains a single field:

  • Authentication Data

    Contains the ICV, which verifies the originating host that sent the message and ensures that the packet was not modified in transit. The ICV uses the defined integrity algorithm to calculate the ICV. The integrity algorithm is applied to the ESP header, the TCP/UDP header, the application data, and the ESP trailer.

ESP provides integrity protection for the ESP header, the TCP/UDP header, the application data, and the ESP trailer. ESP also provides inspection protection by encrypting the TCP/UDP header, the application data, and the ESP trailer.

Choosing Between IPSec Modes

IPSec operates in two modes: transport mode and tunnel mode. IPSec transport mode is used for host-to-host connections, and IPSec tunnel mode is used for network-to-network or host-to-network connections.

Using IPSec Transport Mode

IPSec transport mode is fully routable, as long as the connection does not cross a network address translation (NAT) interface, which would invalidate the ICV. Used this way, IPSec must be supported on both hosts, and each host must support the same authentication protocols and have compatible IPSec filters configured and assigned. IPSec transport mode is used to secure traffic from clients to hosts for connections where sensitive data is passed.

Using IPSec Tunnel Mode

IPSec tunnel mode is used for network-to-network connections (IPSec tunnels between routers) or host-to-network connections (IPSec tunnels between a host and a router). Used this way, IPSec must be supported on both endpoints, and each endpoint must support the same authentication protocols and have compatible IPSec filters configured and assigned. IPSec tunnel mode is commonly used for site-to-site connections that cross public networks, such as the Internet.

Selecting an IPSec Authentication Method

During the initial construction of the IPSec session also known as the Internet Key Exchange, or IKE each host or endpoint authenticates the other host or endpoint. When configuring IPSec, you must ensure that each host or endpoint supports the same authentication methods. IPSec supports three authentication methods:

  • Kerberos

  • X.509 certificates

  • Preshared key

Authenticating with Kerberos

In Windows 2000 and Windows XP, Kerberos is used for the IPSec mutual authentication by default. For Kerberos to be used as the authentication protocol, both hosts or endpoints must receive Kerberos tickets from the same Active Directory directory service forest. Thus, you should choose Kerberos for IPSec authentication only when both hosts or endpoints are within you own organization. Kerberos is an excellent authentication method for IPSec because it requires no additional configuration or network infrastructure.

Some types of traffic are exempted by default from being secured by IPSec, even when the IPSec policy specifies that all IP traffic should be secured. The IPSec exemptions apply to Broadcast, Multicast, Resource Reservation Setup Protocol (RSVP), IKE, and Kerberos traffic. Kerberos is a security protocol itself, can be used by IPSec for IKE authentication, and was not originally designed to be secured by IPSec. Therefore, Kerberos is exempt from IPSec filtering.

To remove the exemption for Kerberos and RSVP, set the value NoDefaultExempt to 1 in the registry key HKEY_LOCAL_ MACHINE\SYSTEM\CurrentControlSet\Services\IPSEC, or use the Nodefaultexempt.vbs script located in the Tools\Scripts folder on the CD included with this book.

Authenticating with X.509 Certificates

You can use X.509 certificates for IPSec mutual authentication of hosts or endpoints. Certificates allow you to create IPSec secured sessions with hosts or endpoints outside your Active Directory forests, such as business partners in extranet scenarios. You also must use certificates when using IPSec to secure VPN connections made by using Layer Two Tunneling Protocol (L2TP). To use certificates, the hosts must be able to validate that the other s certificate is valid.

Authenticating with Preshared Key

You can use a preshared key, which is a simple, case-sensitive text string, to authenticate hosts or endpoints. Preshared key authentication should be used only when testing or troubleshooting IPSec connectivity because the preshared key is not stored in a secure fashion by hosts or endpoints.

Creating IPSec Policies

IPSec is a policy-driven technology. In Windows 2000 and Windows XP, you can have only one IPSec policy assigned at a time. IPSec policies are dynamic, meaning that you do not have to stop and start the IPSec service or restart the computer when assigning or unassigning IPSec policies. You can also use Group Policy to deploy IPSec policies to Windows 2000 and Windows XP clients. Windows 2000 and Windows XP include three precreated IPSec policies:

  • Client (Respond Only)

    A computer configured with the Client policy will use IPSec if the host it is communicating with requests using IPSec and supports Kerberos authentication.

  • Server (Request Security)

    A computer configured with the Server policy will always attempt to negotiate IPSec but will permit unsecured communication with hosts that do not support IPSec. The Server policy permits unsecured ICMP traffic.

  • Secure Server (Require Security)

    A computer configured with the Secure Server policy will request that IPSec be used for all inbound and outbound connections. The computer will accept unencrypted packets but will always respond by using IPSec secured packets. The Secure Server policy permits unsecured ICMP traffic.

In addition to the built-in policies, you can create custom IPSec policies. When creating your own IPSec policies, you must configure rules that include the following settings:

  • IP Filter List

  • Tunnel Settings

  • Filter Actions

  • Authentication Methods

  • Connection Types

IPSec rules determine what types of network traffic will initiate IPSec between the computer and the host or endpoint it is communicating with. A computer can have any number of IPSec filters. You should ensure that only one rule is created for each type of traffic. If multiple filters apply to a given type of traffic, the most specific filter will be processed first.

IP Filter List

The IP filter list defines the types of network traffic that the IPSec rule applies to. You must define the following details for each entry in the filter list:

  • Source address

    Can be a specific IP address, a specific IP subnet address, or any address.

  • Destination address

    Can be a specific IP address, a specific IP subnet address, or any address.

  • Protocol

    The protocol ID or transport protocol used by the protocol. For example, Point-to-Point Tunneling Protocol (PPTP) uses Generic Routing Encapsulation (GRE) packets. GRE packets are identified by their protocol ID, which is protocol ID 47. Telnet, on the other hand, uses TCP as its transport protocol, so an IPSec filter for Telnet would only define the protocol type as TCP.

  • Source port

    If the protocol were to use TCP or UDP, the source port could be defined for the protected connection. The source port is set to a specific port or to a random port, depending on the protocol being defined. Most protocols use a random port for the source port.

  • Destination port

    If the protocol uses TCP or UDP, the protocol uses a specific port at the server to accept transmissions. For example, Telnet configures the server to listen for connections on TCP port 23.

When configuring IP filter lists for transport mode connections, you should always choose to have the IPSec rule mirrored to secure the return communication defined in the rule. For tunnel mode connections, you must manually specify both the inbound and outbound filter list.

Tunnel Settings

The tunnel setting determines whether IPSec operates in transport or tunnel mode. If you want IPSec to operate in transport mode, select This Rule Does Not Specify A Tunnel when creating an IPSec rule using the Security Rule Wizard. If you want the filter to operate in tunnel mode, you must specify the IP address of the endpoint of the tunnel.

Filter Actions

For each filter rule, you must choose a filter action. The filter action defines how the traffic defined in the IP filter will be handled by the filter rule. The three filter actions are listed here and shown in Figure 9-5.

  • Permit

    Allows packets to be transmitted without IPSec protection. For example, Simple Network Management Protocol (SNMP) includes support for devices that might not be IPSec aware. Enabling IPSec for SNMP would cause a loss of network management capabilities for these devices. In a highly secure network, you could create an IPSec filter for SNMP and set the IPSec action to Permit to allow SNMP packets to be transmitted without IPSec protection.

  • Block

    Discards packets. If the associated IPSec filter is matched, all packets with the block action defined are discarded.

  • Negotiate Security

    Allows an administrator to define the desired encryption and integrity algorithms to secure data transmissions if an IPSec filter is matched.

    figure 9-5 ipsec filter actions

    Figure 9-5. IPSec filter actions

In addition to these three basic actions, you can define settings that indicate how the Windows 2000 based computer will react if non-IPSec protected data is received and how frequently new session keys are defined to protect the IPSec data. Options include the following:

  • Accept Unsecured Communication, But Always Respond Using IPSec

    You use this option when the IPSec protection is enforced only at the servers, not at the clients. In a typical IPSec deployment, clients are configured to use IPSec if requested by the server but to never initiate an IPSec SA. This setting allows the initial packet to be received by the server, which then starts the IKE process to negotiate an SA between the client and the server. Although it is riskier to have the initial packet of a data transmission accepted by using plaintext, the response packet sent from the server will not be transmitted until an SA is established.

  • Allow Unsecured Communication With Non-IPSec-Aware Computers

    In a mixed network, this option allows non-IPSec aware clients to connect to the server. Windows 2000 clients, if configured to do so, will connect to the server and negotiate IPSec protection. Non-IPSec-aware clients will still be allowed to connect by using unprotected data streams.

  • Session Key Perfect Forward Secrecy

    Using Perfect Forward Secrecy will ensure that an existing key is never used as the foundation of a new key. When you use Perfect Forward Secrecy, all keys will be generated without using existing keys. This reduces the risk of continual data exposure should a key be compromised because previous keys cannot be used to determine future keys.

Authentication Methods

For each filter rule, you must choose an authentication method. You can enable multiple authentication methods for each rule and determine their order of precedence by editing the filter rule after it has been created.

Connection Types

You must specify what type of interfaces each filter rule applies to. In Windows 2000 and Windows XP, you can choose to have the rule apply to the following:

  • All network connections

  • Local area network (LAN) connections

  • Remote access connections

    You can create IPSec policies by using Ipsecpol.exe from the command line or from batch files and scripts, in addition to using the user interface.

How IPSec Works

IPSec can be initiated by either the sending host or the receiving host. The two hosts or endpoints enter into a negotiation that will determine how the communication will be protected. The negotiation is completed in the IKE, and the resulting agreement is a set of security associations, or SAs.

IKE has two modes of operation, main mode and quick mode. We will examine each mode momentarily. IKE also serves two functions:

  • Centralizes SA management, reducing connection time

  • Generates and manages the authenticated keys used to secure the information

The SA is used until the two hosts or endpoints cease communication, even though the keys used might change. A computer can have many SAs. The SA for each packet is tracked using the SPI.

Main Mode

During the main mode negotiation, the two computers establish a secure, authenticated channel the main mode SA. IKE automatically provides the necessary identity protection during this exchange. This ensures no identity information is sent without encryption between the communicating computers, thus enabling total privacy. Following are the steps in a main mode negotiation:

  1. Policy negotiation

    These four mandatory parameters are negotiated as part of the main mode SA:

    • The encryption algorithm (DES or 3DES)

    • The hash algorithm (MD5 or SHA1)

    • The authentication method (certificate, preshared key, or Kerberos v5 authentication)

    • The Diffie-Hellman (DH) group to be used for the base keying material

      If certificates or preshared keys are used for authentication, the computer identity is protected. However, if Kerberos v5 authentication is used, the computer identity is unencrypted until encryption of the entire identity payload takes place during authentication.

  2. DH exchange (of public values)

    At no time are actual keys exchanged; only the base information needed by DH to generate the shared, secret key is exchanged. After this exchange, the IKE service on each computer generates the master key used to protect the final step: authentication.

  3. Authentication

    The computers attempt to authenticate the DH exchange. Without successful authentication, communication cannot proceed. The master key is used, in conjunction with the negotiation algorithms and methods, to authenticate identities. The entire identity payload including the identity type, port, and protocol is hashed and encrypted by using the keys generated from the DH exchange in the second step. The identity payload, regardless of which authentication method is used, is protected from both modification and interpretation.

    After the hosts have mutually authenticated each other, the host that initiated the negotiation presents an offer for a potential SA to the receiving host. The responder cannot modify the offer. Should the offer be modified, the initiator rejects the responder s message. The responder sends either a reply accepting the offer or a reply with alternatives. After the hosts agree on an SA, quick mode negotiation begins.

Quick Mode

In this mode, SAs are negotiated on behalf of the IPSec service. The following are the steps in quick mode negotiation:

  1. Policy negotiation

    The IPSec computers exchange their requirements for securing the data transfer:

    • The hash algorithm for integrity and authentication (MD5 or SHA1)

    • The algorithm for encryption, if requested (3DES or DES)

    • A description of the traffic to protect

  2. Session key material refresh or exchange

    IKE refreshes the keying material, and new, shared, or secret keys are generated for authentication and encryption (if negotiated) of the packets. If a rekey is required, a second DH exchange takes place or a refresh of the original DH exchange occurs.

  3. SA exchange

    The SAs and keys are passed to the IPSec driver, along with the SPI.

    A common agreement is reached, and two SAs are established: one for inbound communication, and one for outbound communication.

During the quick mode negotiation of shared policy and keying material, the information is protected by the SA negotiated during main mode. As mentioned in step 3, quick mode results in a pair of SAs: one for inbound communication and one for outbound communication, each having its own SPI and key. Figure 9-6 shows a summary of what is negotiated during main mode and quick mode.

figure 9-6 main mode and quick mode negotiation

Figure 9-6. Main mode and quick mode negotiation

IPSec, Routers, and NAT

IPSec creates a new IP header for a packet that can be routed as normal IP traffic. Routers and switches in the data path between the communicating hosts simply forward the packets to their destination. However, when a firewall or gateway lies in the data path, you must enable IP forwarding at the firewall for the following IP protocols and UDP ports:
  • IP protocol ID 50

    Create inbound and outbound filters to allow ESP traffic to be forwarded.

  • IP protocol ID 51

    Create inbound and outbound filters to allow AH traffic to be forwarded.

  • UDP port 500

    Create inbound and outbound filters to allow IKE traffic to be forwarded.

Because of the nature of the NAT and port address translation (PAT) technologies, which require that packets be altered to change IP address and port information, IPSec is not compatible with NAT. IPSec does not allow manipulation of packets during transfer. The IPSec endpoint will discard packets that have been altered by NAT because the ICVs will not match. At the time of the printing of this book, research into encapsulating IPSec packets in UDP packets so that they can pass through NAT devices is under way.

Monitoring IPSec

You can monitor IPSec in Windows 2000 with IPSecmon.exe and in Windows XP the IP Security Monitor Microsoft Management Console (MMC) snap-in. In addition, you can create log files in both Windows 2000 and Windows XP to view IPSec negotiations.

Using IPSecmon in Windows 2000

In Windows 2000, you can view the status of IPSec SAs and basic information on IPSec sessions by running IPSecmon from the Run prompt. IPSecmon displays information about each SA and the overall statistics of IPSec and IKE sessions. Figure 9-7 shows IPSecmon in Windows 2000. The built-in Server IPSec policy is applied to the Windows 2000 computer named SFOFS001. The SFOFS001 computer has attempted to negotiate IPSec with three other computers: SEADC001, SFODC001, and SFOXP001. However, SFOFS001 has successfully negotiated an SA with SFOXP001 only. The IPSec session with SFPXP001 uses the IPSec protocol ESP with 3DES as the encrypting algorithm and HMAC SHA1 as the authentication algorithm.

figure 9-7 using ipsecmon in windows 2000

Figure 9-7. Using IPSecmon in Windows 2000

Using the IP Security Monitor MMC Snap-In

In Windows XP, IPSecmon has been replaced with an MMC snap-in that provides all the information that IPSecmon did in Windows 2000, only in much greater detail. You can use the IP Security Monitor MMC snap-in to view details of each SA, whereas in Windows 2000, you could view only the basic details of an SA. Figure 9-8 shows the IP Security Monitor MMC snap-in in Windows XP, which enables you to view the exact SA details negotiated during both main mode and quick mode.

figure 9-8 using ip security monitor mmc snap-in in windows xp

Figure 9-8. Using IP Security Monitor MMC snap-in in Windows XP

Using IPSec Logs in Windows 2000 and Windows XP

In both Windows 2000 and Windows XP, you can have IPSec log the IKE exchanges to a log file on the hard drive for troubleshooting or monitoring needs. To have your computer log IKE exchanges, you must create a registry value named EnableLogging in the registry key HKLM\System\CurrentControlSet\Services\PolicyAgent\Oakley. To enable logging, set the value to 1 and restart the IPSec services. The log file will be written to the file %systemroot%\ debug\oakley.log. Ipseclog.vbs automatically configures the registry in Windows 2000 and Windows XP to enable IPSec logging. This file is located in the Tools\Scripts folder on the CD included with this book.

Although the IPSec log file will contain more detailed information than a network capture made with Network Monitor, you can also use Network Monitor to determine how IPSec SA negotiations function in relation to the other traffic on the network.



Microsoft Windows Security Resource Kit
Microsoft Windows Security Resource Kit
ISBN: 0735621748
EAN: 2147483647
Year: 2003
Pages: 189

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net