11.1 IPsec VPNs

VPN technologies and their underlying protocols and key exchange mechanisms fill entire books already. One excellent book I used to research and present IPsec key exchange and authentication protocols is IPSec: Securing VPNs, by Carlton R. Davis (McGraw-Hill). If you require detailed low-level information about IPsec and its various modes and protocols, you should definitely read a book dedicated to the subject. Here I tackle the key protocols and mechanisms at a high level, and discuss known remotely exploitable weaknesses and attacks.

Standard Internet (IP) packets are inherently insecure. IPsec was developed to provide security options and enhancements to IP and to negate the following security weaknesses:

• IP spoofing and packet-source forgery issues

• Modification of data within IP packets

• Replay attacks

• Sniffing attacks

IPsec VPNs use the Internet Security Association and Key Management Protocol (ISAKMP) service to provide authentication and key exchange when establishing and maintaining an IPsec connection. After authenticating, a Security Association (SA) is established between the client and VPN gateway. The SA defines the IPsec protocol to be used, as well as cryptographic algorithms, cryptographic keys, and their lifetime.

11.1.1 ISAKMP and IKE

ISAKMP is accessible through UDP port 500, and provides Internet Key Exchange (IKE) support for IPsec VPN tunnels. IKE is used as the authentication mechanism when establishing an IPsec connection; it supports three authentication methods: preshared keys, public key encryption, or digital signatures.

IKE can be run in two primary modes: main and aggressive. Each accomplishes a phase one exchange, generating authenticated keying material for use during phase two. In late 1999, Check Point developed hybrid mode, which supports a number of extra authentication methods (including LDAP, RADIUS, and RSA SecurID). At the time of writing, hybrid mode IKE is an IETF draft, available by searching the IETF site (http://search.ietf.org) for "hybrid mode authentication." The following sections detail a breakdown of main and aggressive mode IKE.

11.1.1.1 Main mode IKE

Main mode is a phase-one key-exchange mechanism that protects the identity of the client and authentication data by using a Diffie-Hellman exchange to generate a mutual secret key. Figure 11-1 shows the main mode IKE messages sent between the initiator and responder.

Figure 11-1. Main mode IKE messages in transit

In total, six messages are transmitted between the two parties. Here is a breakdown of the messages and their purpose:

• Message 1: An IKE Security Association (SA) proposal is sent (not to be confused with an IPsec SA) to initiate the key exchange mechanism.

• Message 2: The IKE SA is accepted.

• Messages 3 and 4: Diffie-Hellman public values (KE) are exchanged, along with a random data nonce payload for each party (Ni and Nr). From this exchange, a mutual secret key is computed.

• Messages 5 and 6: Authentication data (AUTH) is sent, protected by the Diffie-Hellman shared secret generated previously. The identification of the parties (IDii and IDir) is also protected.

11.1.1.2 Aggressive mode IKE

Aggressive mode is used as a cheap alternative to main mode, in which identity protection isn't required. A total of three messages are transmitted during a successful aggressive mode IKE exchange, which reduces processor overhead and bandwidth, but also impacts security and integrity because a Diffie-Hellman exchange isn't used to protect the identity of authentication data. Figure 11-2 shows these three messages sent between the initiator and responder.

Figure 11-2. Aggressive mode IKE messages in transit

Here is a breakdown of the aggressive mode IKE messages and their content:

• Message 1: An IKE SA proposal is sent, along with a Diffie-Hellman public value (KE), random nonce data (Ni), and identity information (IDii).

• Message 2: The IKE SA is accepted, and the responder's Diffie-Hellman public value is sent, along with a nonce (Nr), identity information (IDir), and an authentication payload (AUTH).

• Message 3: Authentication information is sent back, protected by the Diffie-Hellman secret key derived previously.