Main Mode Negotiation

Main Mode Negotiation

Oakley main mode negotiation is used to determine encryption key material and security protection for use in protecting subsequent main mode or quick mode communications. Main mode negotiation occurs in the following steps:

1. Negotiation of protection suites

2. A Diffie-Hellman exchange

3. Authentication

   Note

   IPSec for the Windows Server 2003 family does not support aggressive mode negotiation.

Main mode negotiation consists of six ISAKMP messages: three sent by the initiator and three sent by the responder . Messages 1 and 2 are for the negotiation of protection suites. Messages 3 and 4 are for the Diffie-Hellman exchange. Messages 3 through 6 can be used for authentication. The specific contents of messages 3 through 6 depend on the authentication method.

Negotiation of Protection Suites

The first message in an IKE negotiation is sent by the initiator and contains the following payloads:

• SA    The SA payload contains a Proposal payload, which contains a series of Transform payloads, each of which contains a series of security attributes. The SA payload is the set of protection suites (which include allowed encryption algorithms, hash algorithms, authentication methods, and Diffie-Hellman Oakley groups) that are acceptable to the initiator. This set of protection suites is configured in the Key Exchange Security Methods dialog box in the IP Security Management snap-in. The authentication methods are configured in theAuthentication Methods tab from the properties of an IPSec rule.

• Vendor ID    The Vendor ID payloads contain vendor identification and capability indication.

The second message in an IKE negotiation is sent by the responder and contains the following payloads:

• SA    The SA payload contains a Proposal payload, which contains a single Transform payload corresponding to the protection suite that was offered by the initiator in the first message and is acceptable to the responder. Because the responder might select the first transform in the first message that is acceptable, it is a good practice to configure the key exchange security settings in order of most to least secure.

• Vendor ID    The Vendor ID payloads contain vendor identification and capability indication.

Table 22-6 lists the protection suite attributes that are supported by IPSec for theWindows Server 2003 family.

Key Exchange and Authentication

IPSec for the Windows Server 2003 family supports three methods of authentication:

• Kerberos

• Certificates

• Preshared key

Unlike Point-to-Point Protocol (PPP) authentication for dial-up or virtual private network (VPN) connections, which authenticates the user using the computer at the time of the authentication negotiation, IPSec authentication authenticates only the computer at the time of the negotiation.

Kerberos Authentication

Kerberos authentication is the default authentication method. It is designed for usebetween computers in an Active Directory domain. Kerberos authentication for IPSec main mode is based on the Generic Security Service (GSS) API IKE authentication method,described in the Internet draft entitled "A GSS-API Authentication Method for IKE," which can be found in the \Ietf Drafts folder on the companion CD-ROM.

The third message for a Kerberos-authenticated main mode negotiation, sent by the initiator, contains the following payloads:

• Key Exchange    The Key Exchange payload contains Diffie-Hellman private key determination information as derived by the initiator.

• Nonce    The Nonce payload contains a pseudorandom number derived by the initiator.

• Kerberos Token    The Kerberos Token payload contains the initiator's Kerberos token to be validated by the responder.

• NAT-Discovery    Two NAT-Discovery payloads contain hashes of the initiator's source and destination addresses and ports. For more information, see the section "IPSec NAT Traversal," later in this chapter.

The fourth message for a Kerberos-authenticated main mode negotiation, sent by the responder, contains the following payloads:

• Key Exchange    The Key Exchange payload contains Diffie-Hellman private key determination information as derived by the responder. The initiator and responder use both sets of Diffie-Hellman key material and derive the same private key.

• Nonce    The Nonce payload contains a pseudorandom number derived by the responder.

• Kerberos Token    The Kerberos Token payload contains the responder's Kerberos token to be validated by the initiator.

• NAT-Discovery    Two NAT-Discovery payloads contain hashes of the responder's source and destination addresses and ports.

The fifth message for a Kerberos-authenticated main mode negotiation, which is sent by the initiator and encrypted, contains the following payloads:

• Identification    The Identification payload contains information that iden-tifies the initiator.

• Hash    The Hash payload contains a hash result calculated by the initiator.

The sixth message for a Kerberos-authenticated main mode negotiation, which is sent by the responder and encrypted, contains the following payloads:

• Identification    The Identification payload contains information that identifies the responder.

• Hash    The Hash payload contains a hash result calculated by theresponder.

Kerberos authentication is successful when each peer verifies the other peer's Kerberos token and hash value.

The hash calculation is performed over the following values:

• The Diffie-Hellman key exchange values sent by both the initiator andresponder

• The initiator and responder cookies

• The ISAKMP payloads of main mode message 2

• The identity name string for the IPSec peer

• The IPSec peer's Kerberos token

Network Monitor Capture 22-02 (in the \Captures folder on the companion CD-ROM) provides an example of Kerberos authentication.

Certificate Authentication

Certificate authentication involves the exchange of public key certificates for validation. IPSec for the Windows Server 2003 family supports X.509 version 3 certificates and certificate authentication must be configured in the Authentication Method tab for a rule within an IPSec policy. When configuring certificate authentication, the list of trusted root CAs is specified. You do not necessarily specify the root CA of the issuing CA of the certificate that will be sent, only the root CAs for the issuing CAs of the certificates that are acceptable to receive. Internally, IPSec for the Windows Server 2003 family uses the CryptoAPI to perform various certificate retrieval, verification, and digital signature creation functions.

The third message for a certificate-authenticated main mode negotiation, sent by the initiator, contains the following payloads:

• Key Exchange    The Key Exchange payload contains Diffie-Hellman private key determination information as derived by the initiator.

• Nonce    The Nonce payload contains a pseudorandom number derived by the initiator.

• NAT Discovery    Two NAT-Discovery payloads contain hashes of the initiator's source and destination addresses and ports.

The fourth message for a certificate-authenticated main mode negotiation, sent by the responder, contains the following payloads:

• Key Exchange    The Key Exchange payload contains Diffie-Hellman private key determination information as derived by the responder. The initiator and responder use both sets of Diffie-Hellman key material and derive the same private key.

• Nonce    The Nonce payload contains a pseudorandom number derived by the responder.

• Certificate Request    The Certificate Request payload contains a list of trusted root CAs from which a certificate is acceptable to the responder.

• NAT Discovery    Two NAT-Discovery payloads contain hashes of the responder's source and destination addresses and ports.

The fifth message for a certificate-authenticated main mode negotiation, which is sent by the initiator and is encrypted, contains the following payloads:

• Identification    The Identification payload contains information that identifies the initiator.

• Certificate    The Certificate payload contains a certificate acceptable to the responder.

• Certificate Request    The Certificate Request payload contains a list of trusted root CAs from which a certificate is acceptable to the initiator.

• Signature    The Signature payload contains a digital signature, providing proof that the initiator has access to the private key of the sent certificate. The digital signature is calculated over the Diffie-Hellman keys, the initiator and responder cookies, the SA, and the initiator's identity.

The sixth message for a certificate-authenticated main mode negotiation, which is sent by the responder and is encrypted, contains the following payloads:

• Identification    The Identification payload contains information that identifies the responder.

• Certificate    The Certificate payload contains a certificate acceptable to the initiator.

• Signature    The Signature payload contains a digital signature, providing proof that the responder has access to the private key of the sent certificate. The digital signature is calculated over the Diffie-Hellman keys, the initiator and responder cookies, the SA, and the responder's identity.

Certificate authentication is successful when each peer verifies the other peer's certificate and digital signature.

For Layer Two Tunneling Protocol with IPSec (L2TP/IPSec) connections, you cannot configure the list of trusted root CAs. The list of trusted root CAs is based on the root CAs for the issuing CAs for certificates installed in the Local Computer certificate store. To have a successful L2TP/IPSec connection, each IPSec peer must have an installed computer certificate that was issued by an issuing CA of a trusted root CA that also issued computer certificates to the other IPSec peer.

Network Monitor Capture 22-04 (in the \Captures folder on the companion CD-ROM) provides an example of certificate authentication.

Certificate Revocation Checking

By default, IPSec for the Windows Server 2003 family does not perform certificate revocation checking on the sent certificate or the chain of certificates from the issuing CA to the root CA of the issuing CA. Certificate revocation checking is done by verifying that each certificate in the chain is not on a certificate revocation list (CRL) that wascreated and published by the issuing CA. To enable certificate revocation checking, use the following registry setting:

StrongCrlCheck

Location: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PolicyAgent\Oakley Data type: REG_DWORD Valid range: 0-2 Default: 0 Present by default: No

Set StrongCrlCheck to 1 to fail certificate revocation checking if the certificate has been revoked by verifying its revoked status with the CRL. If the CRL distribution point is not available on the network or the creator of the CRL did not issue the certificate, the certificate is assumed to be valid. Set StrongCrlCheck to 2 to fail the certification validation for any certificate revocation check error. In this case, the CRL distribution point must be reachable on the network, and it must verify that it issued the certificate and that it is not revoked. By default, StrongCrlCheck is set to 0, indicating that no certificate revocation checking is done.

If you change this value, you must stop and start the IPSec services either through the Services snap-in or by running the net stop policyagent and net start policyagent commands at a command prompt. If the computer is running Routing and Remote Access, you should stop the Routing and Remote Access service before you stop the IPSec services and start the Routing and Remote Access service after you start the IPSec services.

Similar products