Overview of IPsec Protocol

IPsec protocol provides the security for the unicast IP protocol traffic. Whether you implement site-to-site or remote access VPN, IPsec is used to protect information as it travels from one private network to another private network over a public network. IPsec consists of a suite of protocols that are defined in RFC 2401 as the following:

• Security protocols Authentication header (AH) and encapsulation security payload (ESP)

• Key management ISAKMP, IKE, SKEME

• Algorithms for encryption and authentication

Before going into the details of security protocols and key management components, it is important to understand the encryption and authentication protocols which are discussed in the next section.

Encryption and Decryption

The term encryption is used to describe the transformation of plain text into a form that makes the original text incomprehensible to an unauthorized recipient that does not have a matching key to decode or decrypt the encrypted message. Decryption is the reverse of encryption, which means that decryption is the transformation of encrypted data back into plain text. A cipher is a cryptographic algorithm that is used for data encryption and decryption. One function is used for encryption and other function is used for decryption. You must use a key in addition to a cipher for encryption and decryption.

Encryption is used for data confidentiality, which is one of the most important requirements for any VPN implementation. An encryption/decryption algorithm built into IPsec makes IPsec a desirable protocol for IPsec implementation.

Cryptographic algorithms can be categorized as follows:

• Symmetric algorithms

• Asymmetric algorithms

Symmetric Algorithms

Symmetric cryptographic algorithms are based on the sender and receiver of the message knowing and using the same secret key. The sender uses a secret key to encrypt the message, and the receiver uses the same key to decrypt it. The main problem with using the symmetric key approach is finding a way to distribute the key without anyone else obtaining it. Anyone who overhears or intercepts the key in transit can later read and modify messages encrypted or authenticated using that key, and can forge new messages. DES, 3DES, and AES are popular symmetric encryption algorithms. A detailed explanation of these algorithms is beyond the scope of this book.

Note

DES uses a 56-bit key and is not considered secure anymore; in 1999, the DES key was cracked in less than 24 hours by using an exhaustive key search. Triple DES (3DES) and AES are the recommended encryption algorithms as of this writing.

Asymmetric Algorithms

Asymmetric encryption algorithms, also known as public key algorithms, use separate keysone for encryption and another for decryption. The encryption key is called the public key and can be made public. Only the private key that is used for decryption needs to be kept secret. Although the public and private keys are mathematically related, it is not feasible to derive one from the other. Anyone with a recipient's public key can encrypt a message, but the message can only be decrypted with a private key that only the recipient knows. Therefore, a secure communication channel to transmit the secret key is no longer required, as it is in the case of symmetric algorithms.

The initiator and responder communicate securely using public key encryption as follows:

1.	Initiator and responder agree on a public key algorithm.
2.	Responder sends initiator his public key and initiator sends responder her public key.
3.	Initiator sends responder a message, encrypting the message using responder's public key.
4.	Responder receives the message and decrypts initiator's message using his private key.

Remember that public key encryption is rarely used to encrypt messages because it is much slower than symmetric encryption. Public key encryption is used, however, to solve the problem of key distribution for symmetric key algorithms, which is, in turn, used to encrypt actual messages. Therefore, public key encryption is not meant to replace symmetric encryption, but can supplement it and make it more secure.

Digital Signatures

Another good use of public key encryption is for message authentication, also known as a digital signature.

Encrypting a message with a private key creates a digital signature, which is an electronic means of authentication and provides non-repudiation. Non-repudiation means that senders will not be able to deny that they sent messages. That is, a digital signature attests not only to the contents of a message, but also to the identity of the sender. Because it is usually inefficient to encrypt an actual message for authentication, a document hash known as a message digest is used. The basic idea behind a message digest is to take a variable-length message and convert it into a fixed-length compressed output called the message digest. Because the original message cannot be reconstructed from the message digest, the hash is labeled "one-way." An initiator and responder's communication using digital signature proceeds as follows:

1.	Initiator computes a one-way hash of a document that she wishes to send responder.
2.	Initiator encrypts the hash with her private key. The encrypted message digest becomes the digital signature.
3.	Initiator sends the document along with the digital signature to responder.
4.	Responder decrypts the digital signature using initiator's public key and also computes a one-way hash of the document received from initiator. If the two values match, responder can be sure that the document came from initiator and the document was not tampered with in transit. The slightest change in the document will cause the values to not match and will cause the authentication to fail.

When the message digest generated is encrypted using a key, it's called a keyed message digest. Another definition for a keyed message digest is message authentication code (HMAC).

Security Protocols

There are two security protocols of IPsec. AH and ESP provide the security services to the unicast IP packets. AH and ESP are discussed in detail in subsequent sections.

Authentication Header (AH)

AH provides connectionless integrity, data authentication, and optional replay protection, but it does not provide confidentiality.

AH is an IP protocol, identified by a value of 51 in the IP header. The Next header field indicates what follows the AH header. In transport mode, it will be the value of the upper layer protocol being protected (for example, UDP or TCP). In tunnel mode, this value is 4.

AH in transport mode is useful if the communication endpoints are also the IPsec endpoints. In tunnel mode, AH encapsulates the IP packet, and an additional IP header is added before the AH header. Although the tunnel mode of AH could be used to provide IPsec VPN end-to-end security, there is no data confidentiality in AH, therefore this mode is not very useful. AH is defined by RFC2402.

Encapsulating Security Header (ESP)

ESP provides confidentiality, data integrity, and optional data origin authentication and anti-replay services. It provides these services by encrypting the original payload and encapsulating the packet between a header and a trailer.

ESP is identified by a value of 50 in the IP header. The ESP header is inserted after the IP header and before the upper layer protocol header. The IP header itself could be a new IP header in tunnel mode or the original IP packet's header in transport mode.

ESP is defined by RFC2406.

Both AH and ESP can work in one of the two following modes:

• Transport mode

• Tunnel mode

Transport Mode

Transport mode is used when both communicating peers are hosts. It may also be applied when one peer is a host and the other is a gateway, if that gateway is acting as a host. For example, a router is sending syslog message traffic to a syslog server across the tunnel. For transport mode to work, you need to have routable IP addresses. Transport mode has an advantage of adding less overhead to the original packets.

Transport mode can be configured both with AH and ESP.

• Transport modeAuthentication Header AH protocol protects the external IP header along with the data payload (see Figure 6-1). It protects all the fields in the header that are not mutable and do not change in transport. The AH header is placed between the IP header and ESP, if present, and other higher-layer protocols.

  Figure 6-1. Showing IP Packet in Transport Mode

  

• Transport modeEncapsulating Security Payload In transport mode, the IP payload is encrypted and the original headers are left intact. The ESP header is inserted between the IP header and the upper-layer protocol header (see Figure 6-1). The upper-layer protocols are encrypted and authenticated along with the ESP header. ESP does not authenticate the IP header itself. Figure 6-1 shows an IP packet after applying ESP in transport mode.

Tunnel Mode

Tunnel mode is used between two gateways or between a host and a gateway, if that gateway is the conduit to the actual source or destination. In tunnel mode, the entire original IP packet is encrypted and becomes the payload of a new IP packet. So the original IP packets don't have to be routable to the Internet. The new IP header has the destination address of its IPsec peer. The biggest advantage of tunnel mode is that all the information from the original packet, including the headers, is protected. Tunnel mode protects against traffic analysis because, although the IPsec tunnel endpoints can be determined, the true source and destination endpoints cannot be determined because the information in the original IP header has been encrypted.

Just as in transport mode, tunnel mode may be configured both with AH and ESP.

• Tunnel Mode with AH The entire original header is authenticated and the new IP header is protected in the same way as the IP header is protected in transport mode (see Figure 6-2).

  Figure 6-2. Showing IP Packet in Tunnel Mode

  

• Tunnel Mode with ESP When ESP is used in tunnel mode, the original IP header is protected because the entire original IP datagram is encrypted (see Figure 6-2). With ESP authentication mechanism, the original IP datagram and the ESP header are included; however, the new IP header is not included in the authentication. When both authentication and encryption are selected, encryption is performed first, before authentication on the initiator. On the responder, authentication occurs before the decryption. One reason for this order of processing is that it facilitates rapid detection and rejection of replayed or bogus packets by the receiving node. Before decrypting the packet, the receiver can detect the problem and potentially reduce the impact of denial-of-service attacks.

Security Associations (SAs)

To have IPsec conversations, you first need to establish a security association. Each device must agree on the policies or rules of the conversation by negotiating these policies with their potential peers. SA is nothing but an instantiation of security policy for a given data flow within the IKE/IPsec environment. An SA is unidirectional, so it represents a unidirectional connection for the traffic. So, if you have traffic flowing in both directions, you need two security associations. For example, if there are two gateways, A and B, and if they need to send and receive traffic over the IPsec tunnel, both of the gateways need to have two SAs on each, one for incoming traffic and another one for outgoing traffic. If both AH and ESP are configured for a unidirectional traffic, two sets of security associations are created for the traffic.

An SA is uniquely identified by an IP destination address, a security protocol (AH or ESP) identifier, and a unique security parameter index (SPI) value. A SPI is a 32-bit number assigned to the initiator of the SA request by the receiving IPsec endpoint. Upon receiving a packet, the destination address, protocol, and SPI are used to determine the SA, which allows the receiving gateway to authenticate or decrypt the packet according to the security policy configured for that SA.

Internet Key Exchange Protocol (IKE) Protocol, which is discussed in greater detail in the sections that follow, is used to create and maintain an SA on both ends of the tunnel.

SA and Key Management with IKE Protocol

As mentioned previously, IKE stands for Internet Key Exchange Protocol, which is defined by RFC 2409. It is used primarily to exchange keying materials and for establishing security associations. It is possible to configure both keying materials and the security associations manually, but for large-scale deployment, this is not a very scalable solution. So, IKE is a must because it addresses this issue.

IKE is a two-phase protocol:

• IKE Phase 1

• IKE Phase 2

More detailed discussion follows in subsequent sections.

IKE Phase 1

The purpose of IKE phase 1 is to authenticate peers, establish a secure IKE SA, and generate keys for IPsec for the secure communication. This is accomplished with IPsec peers agreeing on IKE parameters, exchanging keying material, and authenticating each other.

IKE Phase 1 can function in one of the two modes to accomplish this task:

• Main mode negotiation

• Aggressive mode negotiation

Main Mode Negotiation

In Main mode negotiation, IPsec peers exchange a total of six messagesthree messages in each direction. The first four messages are the same regardless of the types of authentication configured for Phase 1:

• Pre-shared keys

• Rivest, Shamir, Adleman (RSA) signatures

• RSA encrypted nonces

As the pre-shared keys and the RSA signatures are pre-dominantly used in the field for IPsec, these two authentication methods are explained in detail in this section.

Work through the following numbers and correlate them with packet numbers in Figure 6-3.

Figure 6-3. IKE Phase 1 Negotiation with Pre-Shared Key in Main Mode

1.	As shown in Figure 6-3, the first packet is sent by the initiator to the responder with the initiator cookie and the IKE policy proposal. Every policy configured on the initiator with the command crypto isakmp policy policy_number is sent to the Responder. The policy may contain parameters such as encryption algorithm, hash algorithm, authentication method, Diffie-Hellman group, and SA lifetime. These policy proposals are contained within an SA payload and its associated Proposal and Transform payloads.
2.	Upon receiving the IKE packet 1 from the initiator, the responder examines the IKE policy information and attempts to find a match with its own locally configured IKE policies. Assuming the Responder finds a matching IKE policy, it responds with a message indicating acceptance of one of the initiator's policies. Again, this proposal is contained with an SA payload. The peers have now negotiated an IKE policy.
3.	In the third packet, the initiator sends out Diffie-Hellman public values and nonces (random numbers) to the responder.
4.	In the fourth packet, the Responder does the same thing as in the third packet. These values are exchanged in the KE and NONCE payloads, respectively. With these values and the private value, the Diffie-Hellman algorithm comes up with the same shared secret key on both sides of the tunnel. The two nonce values (initiator's and responder's), together with the pre-shared key, are then used to generate the first of four session key values (SKEYID). SKEYID is used, together with the shared secret key (from the Diffie-Hellman exchange) and other keying material, to derive three other session key values called SKEYID_d, SKEYID_a, and SKEYID_e. These session key values are used to derive keys for IPsec, authenticate other ISAKMP messages, and encrypt ISAKMP messages, respectively. At this stage, an IKE policy has been agreed upon (first two messages), keying material has been exchanged (second two messages), and session key values have been calculated.
5.	In the fifth packet, the initiator exchanges hash and identification values, contained in HASH and ID payloads, with the responder.
6.	In the sixth packet, the responder does the same thing as the initiator. The peers then authenticate each other based on the hash values received. If the received hash value is the same as a hash value calculated locally, authentication succeeds.

Main mode negotiation with RSA signature (digital certificate) authentication is very similar to Main mode with pre-shared key authentication. The first four packets are exactly the same. The difference occurs with the fifth and the sixth packets. IKE peers use RSA signature authentication exchange identification, certificates, and signature. These elements are carried in the ID, CERT, and SIG payloads, respectively.

Aggressive Mode Negotiation

Unlike Main mode, Aggressive mode negotiation exchanges only three messages instead of six, so it is faster than Main mode, but slightly less secure because the ID payload is exchanged unencrypted.

Figure 6-4 illustrates aggressive mode using pre-shared key authentication.

Figure 6-4. IKE Phase 1 Negotiation with Pre-Shared Key in Aggressive Mode

Based on Figure 6-4, work through the following packets to complete the Aggressive mode negotiation with pre-shared key:

1.	In Aggressive mode with Pre-shared Key Authentication, initiator sends the first message with proposed IKE policies, its Diffie-Hellman public value, a nonce value, and identification. These are contained in the SA, KE, NONCE, and ID payloads, respectively. Note that all payloads exchanged during Aggressive mode perform the same function as in Main mode.
2.	Upon receiving the first packet, the responder replies with a message containing the accepted IKE policy, its Diffie-Hellman public value, a nonce, a hash used by the initiator to authenticate the responder, and identification. These are contained in the SA, KE, HASH, and ID payloads.
3.	In the third and final packet for phase 1 negotiation, the initiator sends a message consisting of a hash (used by the responder to authenticate the initiator). The hash is contained in a HASH payload.

If RSA signature is used instead of pre-shared key, the first packet consists of SA, KE, NONCE, and ID payloads. The second packet from the responder consists of SA, KE, NONCE, ID, CERT, and SIG payloads. Finally, the initiator sends a certificate and signature. These are contained in CERT and SIG payloads.

IKE Phase 2

Upon completion of Phase 1 with IKE SA established, Phase 2 begins to negotiate and establish IPsec SAs. Unlike Phase 1, Phase 2 can negotiate only one modeQuick mode. The session key IDs (SKEYID_a and SKEYID_e) are used to protect Quick mode.

The router negotiates Quick mode with three messages between the initiator and the responder as shown in Figure 6-5.

Figure 6-5. IKE Phase 2 Negotiation Using Quick Mode

Work through the following numbers to correlate the message ID number shown in Figure 6-5:

1.	The first message sent by the initiator to the responder contains a hash, IPsec proposals, a nonce, and optionally, another Diffie-Hellman public value, and identities. These elements are contained in HASH, SA, NONCE, KE, and ID payloads, respectively. a. The hash is used to authenticate the message to the responder. b. The IPsec proposals are used to specify security parameters, such as the security protocol (AH or ESP), encryption algorithm, hash algorithm, and IPsec tunnel mode (transport or tunnel) to be used for the IPsec SA. These parameters are configured on routers using the crypto ipsec transform-set command. c. The nonce is used to protect against replay attacks. It is also used as additional keying material. d. The Diffie-Hellman public value is included in the message only if the initiator is configured for Perfect Forward Secrecy (PFS). Normally, keys used with IPsec SA are derived from keying material generated during IKE phase 1. This means that IPsec keys generated using PFS are more secure. PFS is configured using the set pfs {group 1 | group 2} command under the crypto map). e. Phase 2 ID payloads (identities) are used to exchange addresses, protocols, and ports for which this IPsec SA is being established. This is configured using a crypto access list.
2.	In the second packet, the responder replies with a message containing a hash (used by the initiator to authenticate the responder), an IPsec proposal acceptance, a nonce value, and optionally, a Diffie-Hellman public value (if the responder supports PFS), and identities. These elements are again contained in HASH, SA, NONCE, KE, and ID payloads. The payloads in the message that the responder sends serve the same purpose as those sent by the initiator.
3.	The initiator sends a third and final message. This contains a hash (HASH payload), and it acknowledges the responder's message and proves that the initiator is alive.