Internet Protocol Security

Internet Protocol Security (IPSec) is not really a protocol, as the name would suggestit is actually a framework of many protocols and mechanisms working together to produce a secure connection between two peers. The heart of IPSec works at the Network layer of the OSI model. Some of the features this framework provides are as follows :

• Data integrity A mechanism used to guarantee that data has not changed. The MD5 or SHA-1 hashing algorithm is used to produce a message digest of the data that is then cross-checked at the destination for data integrity. IPSec uses the MD5 and SHA-1 hashing algorithms to perform this function.

• Data origin authentication Authenticates the source of the data against the peer. For this to work, a data integrity feature is necessary.

• Data confidentiality The technical form of saying the data is encrypted, meaning that confidential information is encrypted so no one else can make sense of it. IPSec uses symmetric key encryption algorithms (DES, 3DES, AES) for data confidentiality.

• Anti-replay A feature that helps protect against hackers who want to cause harm by sending the same packet data repeatedly. Anti-replay helps guarantee that data is received only once.

IPSec Components

IPSec contains several separate components that all work together to make a secure connection between two peers. This section highlights these components; then we will link them all together to make an IPSec Security Association (SA).

Authentication Headers

The authentication header (AH) mode of IPSec provides data authentication and anti-replay protection. Data authentication is also known as data integrity , meaning that AH checks to ensure that the data has not been altered . AH also provides data origin authentication to ensure that the data is coming from the correct source. AH doesn't actually encrypt data. Also, if it's used alone, it does not provide data confidentiality. Figure 12.6 shows user data being placed in an AH-protected packet.

Figure 12.6. An AH example.

Encapsulation Secure Payload

Encapsulation Secure Payload (ESP) performs what most people need, data confidentiality. ESP also performs data authentication and anti-replay protection. ESP encrypts the whole payload, which contains layer 4 headers and the data. ESP uses protocol port 50 and can either be used with AH or stand alone to provide data confidentiality. Figure 12.7 shows user data being encrypted and protected within ESP.

Figure 12.7. An ESP example.

AH, which uses protocol port 51, helps protect against session hijacking, whereas ESP uses protocol port 50 and encrypts the data. Both AH and ESP can be used separately or together. When they are both used, ESP encryption is processed before the AH digest is created. Figure 12.8 shows data being encrypted by ESP and encapsulated by AH.

Figure 12.8. ESP encapsulated in an AH.

Internet Key Exchange

Internet Key Exchange (IKE) is a hybrid of several other protocols such as ISAKMP and the Oakley and Skeme key exchange. When two computers (peers) connect, they need to exchange policies and keys for hashing and encryption. IKE and ISAKMP perform these functions.

IKE is a hybrid protocol to exchange keys and uses UDP port number 500.

Diffie-Hellman

The Diffie-Hellman (D-H) algorithm is actually part of the IKE process to exchange keys safely and securely. D-H uses a pair of asymmetric keys, and each peer's public key is exchanged with the other peer's. These public keys are then combined with the others' private keys to generate an identical symmetric key on both peers. This new shared secret (symmetric) key is used to provide encryption and decryption during the IKE establishment phases. The PIX is capable of using either 768-bit (group 1) or 1024-bit ( group 2) D-H groups.

Authentication

When two IPSec peers connect, they need to authenticate before secure data can be transmitted. This authentication can be done in several ways; the following list describes three methods that can be used to authenticate peers:

• Pre-shared keys These are hard-coded values that are set on both peers. During authentication, if both values match, authentication succeeds. For example, if Jack codes the word "dog" and Peter also uses the word "dog," this would provide a positive authentication. This is by far the easiest to set up; however, if you had 100 VPN peers and wanted to change the key of "dog" to "cat," you have a lot of work to do.

• RSA encrypted nonces This is a time-variant mechanism that uses asymmetric public and private keys that need to be manually created on each peer. Then the public keys must be shared with the peers that will be connecting. When VPN peers connect, each peer uses its own private key to create a digital signature, which is then sent across for authentication. Each peer uses the corresponding shared public key to verify the digital signature. RSA nonces are time-consuming to set up, and if they need to be changed, they take even more time and effort to change than pre-shared keys.

• RSA signatures These are very similar to nonces and use asymmetric public and private key pairs. However, the use of a certificate authority (CA) is involved with certificate generation and authentication. During IKE phase 1, peers exchange each other's certificates. Then, they contact the respective CA to validate the received peer's certificate. This process enables you to change a peer's certificate; and all connecting peers don't need to be modified because the new certificate is sent down during the next connection. RSA signatures is a very scalable mechanism when supporting thousands of VPN peers.

Diffie-Hellman is susceptible to man-in-the-middle attacks. To mitigate this problem, authentication is used during the D-H key agreement algorithm. The authentication methods used are preshared keys, nonces, and RSA signatures.

Security Association

The security association (SA) is similar to a session between TCP hosts . Whenever two peers make a successful transfer of data using IPSec, an SA is created in the background to maintain the connection. This SA identifies each peer by IP address, security protocols, and a security parameter index (SPI).

Security associations can be created using IKE or a manual process. See Cisco for more details about manual security associations.

How IPSec Works

Now that we've reviewed hashing, keys, encryption, AH, and ESP, let's put them all together and see how the IPSec framework makes a secure connection between two peers. IPSec is similar to creating a TCP connection that uses a three-way handshake. The peers exchange several parameters to create a security association; then, after the security association has been established, data can be sent in a protected manner. This exchange is comprised of two main phases called phase 1 and phase 2.

Following are some of the steps needed to create an IPSec security association between two peers:

1. Connect to a peer.

2. IKE phase 1 starts, which involves the following:

   2a. IKE exchange polices

   2b. Diffie-Hellman key agreements are set

   2c. Authentication

3. IKE phase 2 starts (this is the IPSec phase), causing the following actions:

   3a . Crypto maps are exchanged.

   3b. The SA lifetime is established.

4. When data that matches the crypto ACL is sent, it is protected.

To prepare you, phase 1 and phase 2 are where most of your parameters need to be configured between two peers. If they match, a successful security association can be made.

Phase 1

IKE controls phase 1 to create a management connection between the two peers. The management connection exchanges ISAKMP policies, keys, and authentication information. ISAKMP polices are negotiated parameter sets that each peer uses to communicate. Some of the parameters inside an ISAKMP policy include

• The encryption algorithm used is DES or 3DES.

• The hashing algorithm used is MD5 or SHA-1.

• The Diffie-Hellman group used is group 1 or group 2.

• The authentication used is pre-shared keys, RSA nonces, or RSA signature with CAs.

• The lifetime of the management connection.

Peers can contain multiple policies; as peers connect, they iterate through the policies until a matching set is found on both sides. For example, say Jack's policy uses DES and MD5 and Peter's policy uses 3DES and MD5. Because these polices don't match, Jack and Peter will not be able to connect. However, if Peter's policy contains a second policy that matches Jack's, this phase will succeed and they can move on to the next step.

Phase 1 also uses Diffie-Hellman to generate symmetric keys for encryption or hashing. Finally, the authentication is completed between the two peers. When all this is completed, enough information exists to make a secure connection to send IPSec parameters in phase 2.

Phase 1 has two modes of operation: main and aggressive. Main mode takes three exchanges to complete and hides the peer's identity. Aggressive mode takes only two exchanges, making it faster, or more aggressive as it were. However, aggressive mode doesn't hide the peer's identity.

Phase 2

After phase 1 is completed, IKE starts phase 2. Phase 2 creates a user connection that negotiates IPSec parameters such as the transform sets. Transform sets are the modes or methods the two peers use to protect user datafor instance, AH, ESP, or both. Phase 2 also negotiates the encryption key, hash keys, and lifetime of how long the keys are valid before regeneration is necessary. When all these processes are complete, an SA is formed . The parameters contained on a crypto map that are sent between peers are as follows:

• The security protocol is AH, ESP, or both.

• The encryption algorithm for ESP is DES or 3DES.

• The authentication method is AH, ESP, or both.

• The authentication hashing algorithm is MD5 or SHA-1.

• The ESP mode is tunnel or transport.

• The lifetime is time in seconds, amount of data transmitted, or both.

After phase 2 is complete, peers start to send data securely. Figure 12.9 displays phase 1 and phase 2 checking for matching parameters before creating a security association and sending data.

Figure 12.9. Phase 1 and phase 2 example.

Phase 2 has only one mode, which is called quick mode. Because it's already using a secure connection, phase 2 only needs to send parameters.