Configuring and Troubleshooting IPSec

As networks throughout the world become more and more interconnected , network administrators are challenged with ensuring data transferred across a network is secure. This is where the IP Security (IPSec) protocol comes into play and allows for authentication of hosts , data integrity, and data encryption.

IPSec is used to protect data that is sent between hosts on a network, which can be remote access, VPN, LAN, and WAN. IPSec ensures that data cannot be viewed or modified by unauthorized users while being sent to its destination. Before data is sent between two hosts, the source computer encrypts the information. It is decrypted at the destination computer. IPSec provides the following benefits:

• Secure end-to-end communication between hosts.

• Secure connections for remote access clients using the Layer 2 Tunneling Protocol (L2TP).

• Secure router-to-router connections.

As you will see when you begin to configure IPSec, different levels of security can be implemented to meet varying needs. IPSec is implemented through IPSec policies. The policies are created and assigned to individual computers or groups of computers (or groups of users). The policies determine the level of security that will be used.

IPSec can use Data Encryption Standard (DES) or Triple DES (3DES) for encryption data. DES uses a 56-bit key, whereas 3DES uses two 56-bit keys.

IPSec consists of several components that work together to provide secure communications between hosts:

• IPSec Policy Agent This component is responsible for retrieving policy information from the local computer or Active Directory.

• ISAKMP/Oakley Key Management Service This component is responsible for establishing a secure channel between hosts and creating the shared key that is used to encrypt the data. It also establishes a security association between hosts before data is transferred. The security association determines the mechanisms that are used to secure data.

• IPSec Driver On the sending computer, this component monitors IP packets. Packets matching a configured filter are secured using the security association and shared key. The IPSec Driver on the receiving computer decrypts the data.

The following steps outline how the different components work together to provide secure communications:

1. When Computer1 starts, the IPSec policy agent retrieves policy information from the local computer or Active Directory.

2. When Computer1 attempts to send data to Computer2 , the IPSec Driver examines the IP packets to determine whether they match the configured filters. If a match is determined, the IPSec Driver notifies the ISAKMP/OAKLEY.

3. The ISAKMP/OAKLEY service on the two computers is used to establish a security association and a shared key.

4. The IPSec Driver on Computer1 uses the key and security association to encrypt the data.

5. The IPSec driver on Computer2 decrypts the information and passes it to the requesting application.

In summary, before any data is transferred between two hosts, the level of security must be negotiated. This includes negotiating an authentication method, a hashing method, and an encryption method.

Configuring IPSec

IPSec can be enabled in one of two ways depending on where you want the policy settings stored. An IPSec policy can be configured for a local computer using the Advanced tab from the TCP/IP Properties window for the Local Area Connection. To enable the policy, select the TCP/IP options tab, click IP Security, and click the Properties button (see Figure 5.12). To enable IPSec, select the Use This IP Security Policy option. Using the drop-down box, select one of the following three security policies:

• Client (Respond Only) This is used for computers that should not secure communications most of the time, but when requested to set up a secure communication, they can respond.

• Server Secure (Require Security) When this option is selected, the server requires all communications to be secure. If a client is not IPSec-aware, the session will not be allowed.

• Server (Request Security) This is used for computers that should secure communications most of the time. In this policy, the computer accepts unsecured traffic, but always attempts to secure additional communications by requesting security from the original sender.

Figure 5.12. Enabling an IPSec policy on a local computer.

You can also enable IPSec using the Local Security Policy snap-in. The three default policies are listed. Any policy can be enabled for the local computer by right-clicking the policy and choosing the Assign option.

If you are running Active Directory, you can create an IPSec policy that's stored within Active Directory. To view the policies, open the Group Policy snap-in, as shown in Figure 5.13.

Figure 5.13. IPSec policies within the Group Policy snap-in.

The three policies that exist by default are Client, Server Secure, and Server (the process of creating new IPSec policies is outlined later in the chapter). To assign an IPSec policy to group policy, right-click the policy and click the Assign option.

Configuring IPSec for Transport Mode

IPSec can be used in one of two modes: transport or tunnel mode. Tunnel mode is used for server-to-server or server-to-gateway configurations. The tunnel is the path a packet takes from the source computer to the destination computer. This way, any IP packets sent between the two hosts or between the two subnets, depending on the configuration, are secured.

There are two formats that can be used with tunneling mode: ESP tunnel mode and AH tunnel mode. With Authentication Header (AH) tunnel mode, the data itself is not encrypted. It provides authentication, integrity, and protects the data from modification but it is still readable. With Encapsulating Security Payload (ESP) tunnel mode, authentication, integrity, and data encryption are provided.

Tunnel mode is not used for remote access VPNs. IPSec/L2TP or PPTP (point-to-Point Tunneling Protocol) is used for VPN connections. Tunnel mode is used for systems that cannot use IPSec/L2TP or PPTP VPNs.

To configure an IPSec tunnel, perform the following steps:

1. From the Properties window of the IPSec policy you want to manage, select the rule you want to edit, and click the Edit button.

2. Select the Tunnel Setting tab.

3. Select "The tunnel endpoint is specified by this IP address" and type the IP address of the tunnel endpoint.

4. Once the tunnel endpoint has been specified, you can configure the tunneling mode using the Filter Action tab (see Figure 5.14). For ESP tunnel mode, select High. For AH tunnel mode, select Medium.

   Figure 5.14. Configuring AH and ESP tunnel mode.

   

Customizing IPSec Policies and Rules

Each of the policies can be edited using the policy's Properties window (see Figure 5.15). IPSec policies consist of several components, including the following:

• Rules IPSec rules determine how and when communication is secured.

• Filter lists Filter lists determine what type of IP packets trigger security negotiations.

• IPSec Security Methods The security methods determine the security requirements of the rule.

• IPSec Authentication Methods Authentication methods determine the ways in which hosts can identify themselves .

• IPSec Connection Types This determines which types of connections, such as remote access or local area connections, to which the rule applies.

Figure 5.15. Configuring the properties of an IPSec policy.

From the General tab, you can change the name and description for the policy and configure the interval at which the computer will check for policy updates. Using the Advanced button, you can configure the Key Exchange Settings.

When configuring the Key Exchange Settings, you can select the Master Key Perfect Forward Secrecy option. This ensures that no previously used keying material is used to generate new master keys. You can also specify the interval at which authentication and key generation must take place.

The Rules tab lists all the rules that are configured for the policy. Additional rules can be added by clicking the Add button; existing ones can be edited using the Edit button. Clicking the Edit button brings up the Edit Rule Properties window (see Figure 5.16).

Figure 5.16. Editing IPSec rules.

The Filter Lists tab defines the type of traffic to which the rule will apply. The Filter Action tab defines whether the rule negotiates for secure traffic and how the traffic will be secured. Configuring the filter actions allows you to define the different security methods that can be negotiated. The security algorithms supported by IPSec include MD5 and SHA. The encryption algorithms supported include DES and 3DES.

The Authentication Methods tab allows you to configure the method used to establish trust between the two computers (see Figure 5.17). If there are multiple authentication methods configured for a rule, you can change the order in which they are used. The authentication methods available include

• Kerberos Kerberos 5 is the default authentication method in a Windows 2000 domain. Users running the Kerberos protocol within a trusted domain can authenticate using this method.

• Certificates If there is a trusted certificate authority available, certificates can be used for authentication.

• Preshared key For non-Windows 2000 computers or those not running Kerberos, a preshared key can be used for authentication.

Figure 5.17. Configuring IPSec authentication methods.

The Connection types tab allows you to define the types of connections to which the rule applies. This allows you to define different rules for different types of connections. Rules can be applied to Local Area Connections, Remote Access connections, or all network connections.

The Tunnel Setting tab allows you to specify a tunnel endpoint when communication will take place between two specific computers.

You can edit the existing policies or you can create and assign a new policy through the Group Policy snap-in. To create a new policy, right-click IP Security Policies on Active Directory and select Create IP Security Policy. A wizard walks you through the process of creating the initial policy, which you can configure further using the Properties window for the new policy.

Once an IP Security policy has been assigned, you can have the group policy automatically refreshed on a computer by running the secedit /refreshpolicy machine_policy .

Managing and Monitoring IPSec

There are a number of tools available that you can use to manage and monitor IPSec. In terms of management, you can use the Group Policy snap-in to create, assign, and configure the IPSec policies or you can use the IP Security Policy Management snap-in. You can also perform a number of other management tasks using the following settings:

• Restore Default Policies

• Import/Export Policies

• Check Policy Integrity

• Manage IP Filter Lists and Filter Actions

When it comes to monitoring IPSec, you can use Network Monitor to capture IPSec packets. You can also use the IPSec Security Monitor utility, which you start from the command prompt to determine whether IPSec communications are secured. It displays the active security associations with other computers as well as several other IP security stats.