IPsec protocol provides the security for the unicast IP protocol traffic. Whether you implement site-to-site or remote access VPN, IPsec is used to protect information as it travels from one private network to another private network over a public network. IPsec consists of a suite of protocols that are defined in RFC 2401 as the following:
Before going into the details of security protocols and key management components, it is important to understand the encryption and authentication protocols which are discussed in the next section.
Encryption and Decryption
The term encryption is used to describe the transformation of plain text into a form that makes the original text incomprehensible to an unauthorized recipient that does not have a matching key to decode or decrypt the encrypted message. Decryption is the reverse of encryption, which means that decryption is the transformation of encrypted data back into plain text. A cipher is a cryptographic algorithm that is used for data encryption and decryption. One function is used for encryption and other function is used for decryption. You must use a key in addition to a cipher for encryption and decryption.
Encryption is used for data confidentiality, which is one of the most important requirements for any VPN implementation. An encryption/decryption algorithm built into IPsec makes IPsec a desirable protocol for IPsec implementation.
Cryptographic algorithms can be categorized as follows:
Symmetric cryptographic algorithms are based on the sender and receiver of the message knowing and using the same secret key. The sender uses a secret key to encrypt the message, and the receiver uses the same key to decrypt it. The main problem with using the symmetric key approach is finding a way to distribute the key without anyone else obtaining it. Anyone who overhears or intercepts the key in transit can later read and modify messages encrypted or authenticated using that key, and can forge new messages. DES, 3DES, and AES are popular symmetric encryption algorithms. A detailed explanation of these algorithms is beyond the scope of this book.
DES uses a 56-bit key and is not considered secure anymore; in 1999, the DES key was cracked in less than 24 hours by using an exhaustive key search. Triple DES (3DES) and AES are the recommended encryption algorithms as of this writing.
Asymmetric encryption algorithms, also known as public key algorithms, use separate keysone for encryption and another for decryption. The encryption key is called the public key and can be made public. Only the private key that is used for decryption needs to be kept secret. Although the public and private keys are mathematically related, it is not feasible to derive one from the other. Anyone with a recipient's public key can encrypt a message, but the message can only be decrypted with a private key that only the recipient knows. Therefore, a secure communication channel to transmit the secret key is no longer required, as it is in the case of symmetric algorithms.
The initiator and responder communicate securely using public key encryption as follows:
Remember that public key encryption is rarely used to encrypt messages because it is much slower than symmetric encryption. Public key encryption is used, however, to solve the problem of key distribution for symmetric key algorithms, which is, in turn, used to encrypt actual messages. Therefore, public key encryption is not meant to replace symmetric encryption, but can supplement it and make it more secure.
Another good use of public key encryption is for message authentication, also known as a digital signature.
Encrypting a message with a private key creates a digital signature, which is an electronic means of authentication and provides non-repudiation. Non-repudiation means that senders will not be able to deny that they sent messages. That is, a digital signature attests not only to the contents of a message, but also to the identity of the sender. Because it is usually inefficient to encrypt an actual message for authentication, a document hash known as a message digest is used. The basic idea behind a message digest is to take a variable-length message and convert it into a fixed-length compressed output called the message digest. Because the original message cannot be reconstructed from the message digest, the hash is labeled "one-way." An initiator and responder's communication using digital signature proceeds as follows:
When the message digest generated is encrypted using a key, it's called a keyed message digest. Another definition for a keyed message digest is message authentication code (HMAC).
There are two security protocols of IPsec. AH and ESP provide the security services to the unicast IP packets. AH and ESP are discussed in detail in subsequent sections.
Authentication Header (AH)
AH provides connectionless integrity, data authentication, and optional replay protection, but it does not provide confidentiality.
AH is an IP protocol, identified by a value of 51 in the IP header. The Next header field indicates what follows the AH header. In transport mode, it will be the value of the upper layer protocol being protected (for example, UDP or TCP). In tunnel mode, this value is 4.
AH in transport mode is useful if the communication endpoints are also the IPsec endpoints. In tunnel mode, AH encapsulates the IP packet, and an additional IP header is added before the AH header. Although the tunnel mode of AH could be used to provide IPsec VPN end-to-end security, there is no data confidentiality in AH, therefore this mode is not very useful. AH is defined by RFC2402.
Encapsulating Security Header (ESP)
ESP provides confidentiality, data integrity, and optional data origin authentication and anti-replay services. It provides these services by encrypting the original payload and encapsulating the packet between a header and a trailer.
ESP is identified by a value of 50 in the IP header. The ESP header is inserted after the IP header and before the upper layer protocol header. The IP header itself could be a new IP header in tunnel mode or the original IP packet's header in transport mode.
ESP is defined by RFC2406.
Both AH and ESP can work in one of the two following modes:
Transport mode is used when both communicating peers are hosts. It may also be applied when one peer is a host and the other is a gateway, if that gateway is acting as a host. For example, a router is sending syslog message traffic to a syslog server across the tunnel. For transport mode to work, you need to have routable IP addresses. Transport mode has an advantage of adding less overhead to the original packets.
Transport mode can be configured both with AH and ESP.
Tunnel mode is used between two gateways or between a host and a gateway, if that gateway is the conduit to the actual source or destination. In tunnel mode, the entire original IP packet is encrypted and becomes the payload of a new IP packet. So the original IP packets don't have to be routable to the Internet. The new IP header has the destination address of its IPsec peer. The biggest advantage of tunnel mode is that all the information from the original packet, including the headers, is protected. Tunnel mode protects against traffic analysis because, although the IPsec tunnel endpoints can be determined, the true source and destination endpoints cannot be determined because the information in the original IP header has been encrypted.
Just as in transport mode, tunnel mode may be configured both with AH and ESP.
Security Associations (SAs)
To have IPsec conversations, you first need to establish a security association. Each device must agree on the policies or rules of the conversation by negotiating these policies with their potential peers. SA is nothing but an instantiation of security policy for a given data flow within the IKE/IPsec environment. An SA is unidirectional, so it represents a unidirectional connection for the traffic. So, if you have traffic flowing in both directions, you need two security associations. For example, if there are two gateways, A and B, and if they need to send and receive traffic over the IPsec tunnel, both of the gateways need to have two SAs on each, one for incoming traffic and another one for outgoing traffic. If both AH and ESP are configured for a unidirectional traffic, two sets of security associations are created for the traffic.
An SA is uniquely identified by an IP destination address, a security protocol (AH or ESP) identifier, and a unique security parameter index (SPI) value. A SPI is a 32-bit number assigned to the initiator of the SA request by the receiving IPsec endpoint. Upon receiving a packet, the destination address, protocol, and SPI are used to determine the SA, which allows the receiving gateway to authenticate or decrypt the packet according to the security policy configured for that SA.
Internet Key Exchange Protocol (IKE) Protocol, which is discussed in greater detail in the sections that follow, is used to create and maintain an SA on both ends of the tunnel.
SA and Key Management with IKE Protocol
As mentioned previously, IKE stands for Internet Key Exchange Protocol, which is defined by RFC 2409. It is used primarily to exchange keying materials and for establishing security associations. It is possible to configure both keying materials and the security associations manually, but for large-scale deployment, this is not a very scalable solution. So, IKE is a must because it addresses this issue.
IKE is a two-phase protocol:
More detailed discussion follows in subsequent sections.
IKE Phase 1
The purpose of IKE phase 1 is to authenticate peers, establish a secure IKE SA, and generate keys for IPsec for the secure communication. This is accomplished with IPsec peers agreeing on IKE parameters, exchanging keying material, and authenticating each other.
IKE Phase 1 can function in one of the two modes to accomplish this task:
Main Mode Negotiation
In Main mode negotiation, IPsec peers exchange a total of six messagesthree messages in each direction. The first four messages are the same regardless of the types of authentication configured for Phase 1:
As the pre-shared keys and the RSA signatures are pre-dominantly used in the field for IPsec, these two authentication methods are explained in detail in this section.
Work through the following numbers and correlate them with packet numbers in Figure 6-3.
Figure 6-3. IKE Phase 1 Negotiation with Pre-Shared Key in Main Mode
Main mode negotiation with RSA signature (digital certificate) authentication is very similar to Main mode with pre-shared key authentication. The first four packets are exactly the same. The difference occurs with the fifth and the sixth packets. IKE peers use RSA signature authentication exchange identification, certificates, and signature. These elements are carried in the ID, CERT, and SIG payloads, respectively.
Aggressive Mode Negotiation
Unlike Main mode, Aggressive mode negotiation exchanges only three messages instead of six, so it is faster than Main mode, but slightly less secure because the ID payload is exchanged unencrypted.
Figure 6-4 illustrates aggressive mode using pre-shared key authentication.
Figure 6-4. IKE Phase 1 Negotiation with Pre-Shared Key in Aggressive Mode
Based on Figure 6-4, work through the following packets to complete the Aggressive mode negotiation with pre-shared key:
If RSA signature is used instead of pre-shared key, the first packet consists of SA, KE, NONCE, and ID payloads. The second packet from the responder consists of SA, KE, NONCE, ID, CERT, and SIG payloads. Finally, the initiator sends a certificate and signature. These are contained in CERT and SIG payloads.
IKE Phase 2
Upon completion of Phase 1 with IKE SA established, Phase 2 begins to negotiate and establish IPsec SAs. Unlike Phase 1, Phase 2 can negotiate only one modeQuick mode. The session key IDs (SKEYID_a and SKEYID_e) are used to protect Quick mode.
The router negotiates Quick mode with three messages between the initiator and the responder as shown in Figure 6-5.
Figure 6-5. IKE Phase 2 Negotiation Using Quick Mode
Work through the following numbers to correlate the message ID number shown in Figure 6-5: