Introduction to IPSec


The IP Security (IPSec) protocol is a framework for authenticating and encrypting Internet Protocol (IP) traffic. Working at the third layer of the OSI model, IPSec is a series of multi-vendor open standards and protocols. It is proving to be one of the most common methods for securing endpoints and network traffic. IPSec can be used in the following ways:

  • To create a virtual private network (VPN) from end to end

  • To enforce network isolation

  • As a host-based firewall, allowing and dropping filtered traffic

  • To secure less secure protocols (such as RDP)

Although IPSec is optional for IPv4 traffic, it is mandatory for IPv6 traffic. Along with other popular open-source VPN solutions, like Secure Session Layer (SSL) and Transport Layer Security (TLS), IPSec will be a long-term security solution for network communications. Windows 2000 and later can natively participate in Windows IPSec.

Note 

Windows IPSec cannot be used to secure multicast or broadcast network traffic, but it can be used to block it.

An Open Standard

IPSec became an open standard through the efforts of the Internet Engineering Task Force (IETF) Internet Protocol Security working group (www.ietf.org/html.charters/OLD/ipseccharter.html). Led by many companies and technologists, including Microsoft and Cisco, the IPSec working group's efforts resulted in dozens of Request for Comments (RFCs). RFCs 2401 (www.ietf.org/rfc/rfc2401.txt) to 2412 (www.ietf.org/rfc/rfc2412.txt) are considered the IPSec protocol standards. The IPSec standards cover how network packets should be secured, cryptographic key exchange, and the related protocol implementations. The IPSec working group officially disbanded in April 2005, although IPSec standards work still continues in several official sectors.

IPSec Basics

IPSec is an open standard involving numerous pieces, parts, and new terminology. In a nutshell, IPSec uses cryptographic hashes to establish packet and data integrity, and cryptographic ciphers to ensure confidentiality. Per RFC 2401, the security services offered include access control, connectionless integrity, data origin authentication, replay protection, data confidentiality, and limited traffic flow confidentiality. In order for two hosts to communicate using IPSec, they must be predefined to use common protocols, hashes, and cryptographic ciphers.

Note 

It is important to note that IPSec's data origin authentication is machine authentication, computer-to-computer. It cannot be used to authenticate a message (i.e., digital content) as belonging to or sent by a particular person.

Working at Layer 3, the Network Layer, IPSec is independent of the upper-layer protocol traffic it is protecting. IPSec is implemented on network devices and on computers at the IP stack level. The participating upper-layer applications (and users) need not be aware of IPSec. Compare this to SSL or TLS, where the applications must be coded to implement, initiate, and understand the security protocol.

An IPSec connection is always between two participating end nodes (or endpoints). All IPSec traffic is only between one source endpoint and another participating, single destination endpoint. For example, if there are 200 client PCs connecting to a Windows server using IPSec, it means there are 200 separate IPSec VPN tunnels. Each connection can involve one or more paths (for example, multiple ports and application protocols) between the two endpoint hosts.

Tunnel versus Transport Mode

IPSec has two major mode types: tunnel and transport. Tunnel mode is used to create a VPN between two network devices (see Figure 8-1). The endpoints in this case are the two network gateway devices. They communicate with each other using IPSec. The PCs on either side of the networks are not using IPSec. Their traffic is sent clear-text across their local area networks and secured between the participating gateways. With tunnel mode, there are essentially two IP headers: an external one used by IPSec to route the packet from gateway to gateway and another internal IP header used to route the traffic to its ultimate intended destination once past the participating gateway.

image from book
Figure 8-1

Tunnel mode can be used to protect IP traffic over untrusted networks like the Internet. It can also be helpful when the participating computers cannot use IPSec natively (i.e., Windows NT or 9x). Tunnel mode protects the entire packet, including payload and packet header. Of course, in tunnel mode there is no guaranteed end-to-end security, only gateway-to-gateway security.

Note 

The limited data flow analysis feature of ESP is accomplished using tunnel mode because it obscures the ultimate source and destination of the data payload.

Transport mode is client-to-server security, typically between an endpoint PC and a server or VPN gateway device. Figure 8-2 shows examples of transport mode connections. With transport mode, protection is provided only to the upper-layer protocol or data, but not over the entire IP packet. A security gateway is only required to support tunnel mode, but computer hosts are required to support both tunnel and transport mode.

image from book
Figure 8-2

IPSec Security Protocols — AH versus ESP

IPSec can be used with two IPSec security protocols: Authentication Header (AH) and Encapsulating Security Payload (ESP).

Authentication Header

AH provides connectionless integrity, anti-replay (optional) services, and data origin authentication across an entire IP packet. It does not provide encryption. AH ensures that the packet came from where it says it came from, capable of stopping most man-in-the-middle attacks. Integrity is delivered using a cryptographic hash algorithm, usually 128-bit MD5 or 160-bit SHA-1. The IP packet hashes resulting from the applied hash algorithms are further protected by IPSec by applying a secret key only known by the IPSec endpoints. In most IPSec implementations, you can use either MD5 or SHA1 hash algorithms. Use SHA-1 when you can and MD-5 where you have to. Most IPSec implementations should support both, although Windows 2000 requires Service Pack 1 or later to support SHA-1.

Note 

Both MD5 and SHA-1 were found to have cryptographic weaknesses in 2005. Thus, in the future, it is expected that both will be replaced with better, cryptographically sound hash algorithms.

Encapsulating Security Payload

ESP provides payload data confidentiality (i.e., encryption) and limits traffic flow analysis but can also provide integrity services. Using ESP data confidentiality, the data payload is encrypted using a symmetric key algorithm, but the IP packet header is not protected. Like AH, ESP can also provide connectionless integrity, anti-replay services, and data origin authentication — but only on the payload data, not the entire packet.

ESP Encryption Ciphers — DES and 3DES

ESP provides encryption using symmetric key algorithms, such as Data Encryption Standard (DES), Triple DES (3DES), Advanced Encryption Standard (AES), and Blowfish (although Microsoft Windows only supports the first two currently). DES is a symmetric encryption algorithm chosen by the National Institute of Standards and Technology (www.nist.gov) as the government's official recommended symmetric cryptography algorithm in 1976. An updated NIST publication on DES can be found at http://csrc.nist.gov/publications/fips/fips46-3/fips46-3.pdf. DES' 64-bit encryption (actually 56 bits of effective encryption plus 8 parity bits) held up well over the years, but in the late 1990s, a few groups and contests "broke" DES in a few days' time. NIST recommended that 3DES (called triple DES), with 192-bit encryption, be used in 1999, while in the middle of replacing DES and 3DES with a new standard.

In 2001, NIST announced the government's new official symmetric encryptions standard (http://csrc.nist.gov/publications/fips/fips197/fips-197.pdf), Advanced Encryption Standard (AES). Based on the Rijndael algorithm and starting at a 128-bit key size, AES is expected to be a default encryption choice for symmetric encryption (including in IPSec) for at least one to two decades. Microsoft's AES support is currently mixed, but getting better. Windows Server 2003 and Windows XP Pro SP1 support AES in several features, including wireless WPA and EFS. Unfortunately, AES is not currently available for Microsoft Windows IPSec.

Use 3DES when you can, and DES where you have to. Windows 2000 requires Service Pack 1 or later. IPSec implementations should support both symmetric protocols per the RFCs. You can also set ESP to <None> (also known as ESP Null), to turn off the encryption. If ESP is still enabled (i.e., integrity only), IPSec will only authenticate the data packet's origin and payload integrity.

AH or ESP?

IPSec must involve one or more protocols — AH or ESP, or both. If you intend to provide integrity and encryption across as much of the IP packet as you can, you want to use both. AH works using IP protocol number 51, and ESP works with 50. If firewalls are involved, these protocol numbers must be allowed through (for reference, the normal TCP protocol uses protocol number 6, and UDP uses protocol number 17).

When IPSec is enabled, each IP packet is analyzed to see if it needs IPSec protection or filtering. IP traffic is queried for its source and destination IP address, source and destination port numbers, protocol type (i.e., TCP vs. UDP), and other factors. The collection of identification values in IPSec is called a security policy or rule. There can be one or more IPSec rules per host. Rules are collected into the host's security policy database.

Security Associations

Each IPSec tunnel involves two or more security associations (SAs). SAs are simplex (unidirectional) connections allowing two endpoints to be linked together. All traffic traveling in the same SA is given the same IPSec security treatment. SAs are identified using three pieces of information (called a triple): destination IP address, security protocol used (AH or ESP), and a Security Parameters Index (SPI). The SPI is a unique 32-bit number relating two hosts with each other. The SPI is indicated in the IPSec packet and ensures that when multiple IPSec end nodes are communicating to one IPSec host the connections aren't confused.

The Internet Security Association and Key Management Protocol (ISAKMP) is a key-independent framework defining the procedures and packet formats to establish, negotiate, modify, and delete SAs. For normal two-way communications, two SAs — mirroring each other's information — are needed. A different SA must be used if AH and ESP are both involved. A host's security associations are contained in its Security Association Database (SAD).

Key Management

Essential to IPSec is the issue of key management. What cryptographic keys to use and how they should be implemented and exchanged is controlled by a hybrid key management protocol called Internet Key Exchange (IKE). IKE also uses the concept of SAs, but is not related to the SAs used in the endpoint IPSec tunnels discussed in the preceding section. IKE SAs are bidirectional.

IKE is composed of two other protocols: Oakley and SKEME, although it only uses each partially (hence the hybrid descriptor). Oakley is a key exchange protocol describing key secrecy, identity protection, and authentication. It uses the Diffie-Hellman protocol for public key exchange. Most of the IKE protocol is based upon the Oakley processes. SKEME allows flexible key exchange with anonymity and repudiability, along with easy and frequent key refreshment. IPSec key exchange normally takes place over UDP port 500. This port must be opened if a firewall is in between the two endpoints.

IKE Modes — Main, Aggressive, and Quick

IKE begins creating the SAs by negotiating which encryption and hashing protocols can be used, how long the keys can be used for, and whether or not perfect forward secrecy (covered below) is used. Multiple proposals can be sent in one offering. Once a proposal is accepted, the Diffie-Hellman protocol is used to exchange public keys and other information. The public keys are used to establish a secure tunnel, and further negotiation and communications can be protected.

Within IKE, there are two modes for establishing an SA. Main mode requires six packets back and forth between the two endpoints, but expends additional effort to ensure endpoint identity. Aggressive mode doesn't ensure endpoint identity, but is used to allow quicker associations. All IPSec hosts must have main mode implemented, but aggressive mode is optional. After the IKE SA is established, quick mode begins. Quick mode is similar to aggressive mode in that endpoint identification is not established (again, if main mode is used), although it is accomplished only after the encrypted IKE SA is initiated.

By default, Windows IPSec will keep quick mode SAs open for up to five minutes after no activity is noticed, but the default can be changed. Windows Server 2003 IPSec is both cluster-aware and network load balance (NLB)—aware, although it does not support instant failover. If a node is in a cluster, IPSec will detect its configuration and change the default quick mode configuration time to one minute, instead of five. This change is communicated to any connecting clients using IPSec; and it means that if the cluster node goes down, the clients will be reconnected to remaining cluster nodes after one minute.

In summary, main mode creates a bidirectional SA (known as ISAKMP or IKE SA), which creates a secure channel for quick mode SAs to then be established. Main mode then negotiates the security parameters to be used, exchanges the Diffie-Hellman keying information, and authenticates the identities of the IPSec peers. Quick mode then creates unidirectional IPSec SAs to secure the network traffic. During negotiation for the latter SAs, IPSec determines encryption and hashing algorithms, security modes (AH, ESP, or both), and whether it is tunnel or transport mode. IKE uses its own secret key with each IKE SA that is set up between each participating endpoint node. The IKE secret key, and the private channel it creates, is then used to securely pass the secret keys used with each SA, which is then used to secure the network communication's traffic.

NAT versus NAT-T

IPSec in ESP is compatible with Network Address Translation (NAT), but IPSec with AH is not. Because AH calculates integrity values based on the entire packet, when a NAT device modifies a protected IP packet (as NAT must do to convert between public and private IP addressing), it invalidates the AH integrity values.

To get around this problem, another IETF working group developed NAT-Transversal (or NAT-T), which is summarized in RFCs 3947 and 3948. It accomplishes this by encapsulating NAT in UDP traffic along with several other modifications and additions. NAT-T requires NAT-T-compatible endpoints. Windows XP SP2 and Windows Server 2003 natively support NAT-T, although strangely enough Microsoft often doesn't differentiate between NAT and NAT-T in their documentation and online help. In those versions, Microsoft's "NAT" is really NAT-T. You can also download a VPN client to support previous versions of Windows XP and the NT and 9x families (www.microsoft.com/downloads/details.aspx?FamilyID=&DisplayLang=en). NAT-T also requires UDP port 4500 to be open on any participating firewalls for IKE (NAT uses UDP port 500).

In March 2005, Microsoft became of aware of a potential security problem when Windows NAT-T clients connected to Windows servers behind NAT devices. The main issue is that IPSec can get confused if the remote private IP address behind the NAT device matches a valid local IP address on the local network. It could end up routing traffic to the local private address instead. Accordingly, Microsoft recommends that participating servers be assigned a public IP address instead and not use NAT/NAT-T. They changed XP SP2's default behavior to prevent connecting to a server behind a NAT device by default, although a registry change can be made to restore the older functionality. See http://support.microsoft.com/default.aspx?scid=kb;en-us;885348 for more details.

Performance Issues

IPSec and its family of protocols provide integrity and encryption protective services on IP traffic. When implemented, IPSec's processing will slow down normal network communications and take up additional, although minimal, memory. How much slower and how much memory depends on the application, the number of SAs, and the vendor's implementation. In general, the slowest part of IPSec is the initial SA and key establishment, but every transmitted and received packet undergoes multiple cryptographic computations. In Windows, the initial establishment takes 1–4 seconds on most connections. After the initial establishment, the integrity and encryption features create ongoing cryptographic delays. How much total time IPSec adds to each transaction depends on the application, but in all cases it is measurable.

The author conducted an unscientific, one-time test, copying the same 340MB file dozens of times with and without IPSec between two relatively otherwise idle computers. Transmit times doubled in many instances with IPSec enabled, but the localized performance problems were just as noticeable. On both sides of the connection, with IPSec enabled, CPU utilization went above 75% (as compared to 10–20% without IPSec). LAN adapter network utilization dropped from 65–70% without IPSec enabled, to 20–40% with IPSec enabled.

Even IP packets not needing IPSec protection are slowed down a bit by the IPSec filter inspecting it to decide whether or not it should be included. In a normal LAN or remote access environment, the delays are usually acceptable (and to many users, unnoticeable), but in high-performance environments, software-only IPSec would probably be unacceptable. Environments needing high-volume IPSec transactions should consider using a VPN-accelerator hardware component or use external VPN devices. IPSec offload adapters are network cards with special hardware chips to take on the additional processing requirements of IPSec.

Now that you understand IPSec and some of its basic functionality, let's explore Microsoft's implementation in detail.



Professional Windows Desktop and Server Hardening
Professional Windows Desktop and Server Hardening (Programmer to Programmer)
ISBN: 0764599909
EAN: 2147483647
Year: 2004
Pages: 122

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net