Addressing IPSec Security Considerations | MCSE Planning and Maintaining a Windows Server 2003 Network Infrastructure: Exam 70-293 Study Guide and DVD Training System

EXAM 70-293 OBJECTIVE 3.3.1, 5, 5.2, 5.7

As you begin to deploy IPSec throughout your organization, you will need to decide on the encryptions methods you wish to implement and whether to use firewall packet filtering. The following sections provide some guidelines to use when considering IPSec security.

Strong Encryption Algorithm (3DES)

Earlier in the chapter, we discussed the two encryption algorithms supported by IPSec for data encryption: DES and 3DES. The 3DES algorithm is the strongest of these, using three unique 56-bit keys. In a high-security environment, the 3DES algorithm is the appropriate choice for encrypting your data.

DES and 3DES are block ciphers. This refers to an algorithm that takes a block of plaintext of a fixed length and changes it into a block of ciphertext (encrypted data) of the same length. The key length for DES is 64 bits total, but because 8 of the bits are used for parity information, the effective length is only 56 bits. With 3DES, the DES process is performed three times with different 56-bit keys, making the effective key length 168 bits. When using 3DES in encrypt-encrypt-encrypt (EEE) mode, 3DES works by processing each block as follows:

A block of plaintext is encrypted with key one.
The resulting block of ciphertext is encrypted with key two.
The result of step 2 is encrypted with key three.

When using 3DES in encrypt-decrypt-encrypt (EDE) mode, step 2 is run in decryption mode. When 3DES is decrypting a packet, the process is done in reverse order. 3DES offers you the best mode for data confidentiality.

Firewall Packet Filtering

To allow for secured packets to be passed through a firewall, you need to configure the firewall or other device, such as a security gateway or router, to allow these packets to pass through the external interface.

The following ports and protocols can be used for firewall filtering:

IP protocol and port 50, ESP traffic
IP protocol and port 51, AH traffic
UDP port 500, IKE negotiation traffic

Diffie-Hellman Groups

As we discussed earlier in the chapter, Diffie-Hellman groups are used to define the length of the base prime numbers that are used during the key-exchange process. There are three types of Diffie-Hellman groups, as follows:

Diffie-Hellman group 1 This is the least secure group and it provides only 768 bits of keying strength.
Diffie-Hellman group 2 This group is set to a medium level, at 1024 bits of keying strength.
Diffie-Hellman group 3 This group is set to the highest level, at 2048 bits of keying strength.

Diffie-Hellman group 3 is available only on Windows Server 2003 family machines. If you wish to use this algorithm on Windows 2000 machines, you must have either Service Pack 2 or the High Encryption Pack installed. If you configure one client machine for a Diffie-Hellman group 1 key exchange and another client machine for the Diffie-Hellman group 3 exchange, negotiation will fail.

For the best security, use the highest Diffie-Hellman group 3 key exchange. When using the quick mode, new keys are created from the Diffie-Hellman main mode master key material. If you have the master key or session key PFS enabled, a new master key will be created by performing a Diffie-Hellman exchange. The master key PFS will require a reauthentication of the main mode SA in addition to the Diffie-Hellman exchange. The session key PFS will not require this reauthentication.

Pre-shared Keys

To authenticate L2TP protocol and IPSec connections, you can select to use a pre-shared key. This is the simplest of three choices of authentication methods that you have with IPSec. The other two authentication methods are Kerberos and digital certificates. Before selecting to use a pre-shared key, you should be aware of all the implications of doing so.

A pre-shared key is a string of Unicode characters. You can use the Routing and Remote Access management console to configure connections to support authenticated VPN connections using the pre-shared key. A server that has the Windows Server 2003 operating system installed may also be configured to use a pre-shared key to authenticate connections from other routers via the Routing and Remote Access console.

As we discussed earlier in the chapter, when you create IPSec policies for a computer, you can define the authentication method to be used. In order for two computers to communicate via IPSec, they must have a common authentication method configured. To increase the chances that this will happen, you can configure a machine to use multiple authentication methods. You might want to set up a computer to be able to use a pre-shared key for this reason.

Advantages and Disadvantages of Pre-shared Keys

Pre-shared key authentication does not have the overhead costs that a PKI implementation does. This type of authentication is relatively easy to configure using the Routing and Remote Access console (for L2TP/IPSec connections) or the IP Security Policy Management console (for IPSec secured communications).

Pre-shared keys are stored as plaintext. This means the key can be compromised if a hacker is able to access the file on the computer. Thus, the pre-shared key is the weakest of the three IPSec authentication methods.

Another drawback of pre-shared keys in relation to L2TP/IPSec connections is that a remote access server can use one pre-shared key for all L2TP/IPSec connections that require a pre-shared key for authentication. In this case, you need to issue the same pre-shared key to all L2TP/IPSec VPN clients that connect to the remote access server using a pre-shared key. Unless you are using the Connection Manager profile to distribute the pre-shared key, each user must manually type the pre-shared key. If you change the pre-shared key on a remote access server, clients with manually configured pre-shared keys will not be unable to connect to the server until the pre-shared key on the client is changed.

Exam Warning

Microsoft’s recommendation is that pre-shared keys be used for authentication only for testing. It is recommended that you not use this authentication method on your production network. Pre-shared keys do not offer good security for sensitive communications, and if you did not need a high-security solution, you would not be implementing IPSec in the first place. Microsoft documentation emphasizes that Windows Server 2003 includes the pre-shared key option only for interoperability with computers that don’t support Kerberos and in environments without a PKI.

Considerations when Choosing a Pre-shared Key

Remember that a pre-shared key is just a sequence of characters that is configured on both computers that are parties to an IPSec-secured communication. The pre-shared key can be any non-null string of any combination, up to 256 Unicode characters.

When you choose a pre-shared key, consider that users who use the New Connection Wizard to create a VPN client connection must type the pre-shared key manually. A key that is long and complex enough to provide adequate security might be difficult for the majority of your users to remember or type accurately. If the pre-shared key presented by one party to the communication deviates in any way from the pre-shared key configured on the other, IPSec authentication will fail.

Soft Associations

A soft association refers to an SA that was created with a computer that hasn’t responded to main mode association attempts since the last time the IPSec service was started. If the IPSec policy is so configured, the communications will be allowed, even though there was no response to the main mode negotiation attempt. It’s important to understand that a soft association is not protected by IPSec.

The soft association is just a communication that is not secured. This occurs when one of the two communicating computers doesn’t support IPSec, and the IPSec policy allows unsecured communications in this situation.