IPSec Protocol Framework

IPSec is a framework of protocols that comprises a combination of standards and technologies. The protocol framework itself is open and flexible in that it does not mandate a specific key length or algorithm. IPSec contains two basic security protocols as part of the IPSec protocol standard:

• Authentication Header (AH) A protocol that is utilized when data confidentiality is not a concern. It does not perform any form of encryption for the data itself, but rather performs an integrity checksum to ensure that the payload was not modified during transit. AH is a Layer 4 (Transport layer) protocol that can be identified by the IP header's protocol field value of 51.

• Encapsulating Security Payload (ESP) This protocol can actually ensure data confidentiality with encryption, as well as data integrity and authentication. ESP protects the IP data payload by encrypting it and adding (encapsulating) an additional header and trailer. ESP can be used in conjunction with AH to provide authentication and integrity of the data, or ESP can provide this functionality within itself. When ESP authentication is enabled, the data contained between the ESP header and trailer is authenticated. ESP is also a Layer 4 (Transport layer) protocol that can be identified by the IP header's protocol field value of 50.

It is important to note that when ESP encryption and authentication is enabled on a local device, encryption is performed before any authentication. This way, when the receiver examines the datagram, it validates the authentication checksum before decrypting the packet, saving time and process cycles.

Transport Mode and Tunnel Mode

ESP and AH can operate in either transport mode or tunnel mode. The key difference between the two modes is what portion of the IP datagram is being authenticated and encrypted. Transport mode actually adds the AH and/or ESP header before the original IP header. By leaving the original IP header in front of the secured payload, routing devices can still determine the original source and destination addresses, as opposed to the tunnel endpoints. Transport mode is commonly used between end systems. In tunnel mode, the entire original IP packet is secured, including the original IP header, and a new IP header is created for tunnel routing information. This mode is common between two IPSec gateways or an IPSec end station and a gateway, because the IPSec device is securing data on behalf of the network behind itself. Figure 2.4 shows the differences between ESP in transport and tunnel modes.

Figure 2.4. ESP in transport and tunnel mode.

Internet Key Exchange (IKE)

IKE is a hybrid protocol derived from the Internet Security Association and Key Management Protocol (ISAKMP), the Secure Key Exchange Mechanism (SKEME) protocol, and the Oakley protocol. IKE automatically handles the preliminary negotiation and authentication between IPSec peers. This negotiation aspect of IKE entails an agreement between both IPSec peers, in which matching encryption and hash algorithms, as well as peer authentication methods, tunnel modes, and IPSec policy lifetimes, are also determined. As you can discern from the acronym, IKE also negotiates and implements the Diffie-Hellman groups for key exchanges.