Lesson 1: Configuring IPSec Within a Domain | MCSA/MCSE Self-Paced Training Kit (Exam 70-214): Implementing and Administering Security in a Microsoft Windows 2000 Network (Pro-Certification)

Lesson 1: Configuring IPSec Within a Domain

IPSec provides the ability to encrypt TCP/IP communications between computers. While it was originally developed to provide encryption between public Internet hosts, there are many private environments in which encrypted communications between servers and clients would improve security, including loosely controlled environments such as universities, high-security environments such as military operations, and industries such as finance that have a need to protect sensitive information or trade secrets.

Within a domain, enabling IPSec between servers and clients is easy, and IPSec can be deployed throughout a domain of any size with minimal administrative effort.

To complete this lesson, you will need

The dc01 domain controller
The ms01 member server
The CLIENT01 client computer

After this lesson, you will be able to

Understand the purpose of IPSec
Enable secure encrypted communications on servers and clients within a domain

Estimated lesson time: 30 minutes

Understanding the IPSec Basics

IPSec is the standard method of authenticating and encrypting traffic between IP hosts. IPSec provides the IP packet structure and protocols necessary to automatically exchange keys between hosts and negotiate encryption and authentication protocols. IPSec performs these two primary functions using two complementary features:

Authenticated Headers (AH) digitally encrypts the IP header (the portion of a packet containing the source and destination address) and the payload (the portion of a packet containing the user's data) to ensure that they have not been modified at any time during their transit between hosts. AH does not encrypt traffic.
Encapsulating Security Payload (ESP) encrypts packets and applies a new non-encrypted header to facilitate routing. Beyond providing encryption, ESP does not guarantee the authenticity of header data.

These two methods can be used together to provide both authenticated headers and encrypted data payload.

Windows 2000 and Windows NT 4 both contain support for SMB message signing, which provides the ability to authenticate each SMB (Windows file and print sharing) packet between a server and its clients. This functionality has been superceded by IPSec AH, which works for SMB as well as all other protocols. Configure SMB message signing only when you can't use IPSec, such as when you must support backward compatibility with Windows NT 4 machines.

ESP Modes

ESP functions in two modes, as determined by the functionality that is required and the capability of the IPSec aware hosts or routers:

Transport mode, in which data payload is encrypted but header data is unchanged. Transport mode is intended to encrypt data between two hosts that are IPSec aware and capable of decrypting the payload data directly, as is the case with Microsoft Windows 2000 systems.
Tunnel mode, in which the entire original packet is encrypted and becomes the payload of a new packet, which is then transmitted between IPSec aware routers. Tunnel mode enables IPSec aware routers to encapsulate and encrypt network traffic from non-IPSec aware hosts, transmit it over a non-secure network, and then decrypt it for use on the destination network by other hosts that are not IPSec aware. Tunnel mode in Windows 2000 is provided primarily for interoperability with third-party IPSec solutions when Windows 2000 is being used as a router.

IPSec Applications

IPSec is employed in three widely used scenarios:

Host-to-host (H2H) signifies secure connections between individual computers that are both IPSec capable. H2H connections are used frequently to secure communications in internal networks, extranets, and on the Internet. H2H applications use AH and ESP in transport mode.
Host-to-gateway (H2G) specifies secure connections between hosts and a network gateway to a private network. H2G applications are used to provide secure telecommuting for remote clients. Windows uses Layer 2 Tunneling Protocol (L2TP) to support H2G applications. L2TP is essentially Point-to-Point Protocol (PPP) that has been encrypted using IPSec transport mode, and it is covered in Chapter 9, "Remote Access and VPN."
Gateway-to-gateway (G2G) specifies secure connections between border gateways to create a secure wide area network (WAN) connection over the Internet. IPSec tunnel mode is supported to create a single encrypted link between two networks when the link must be compatible with earlier IPSec-compliant routers that do not support L2TP. Using IPSec tunnel mode is not recommended for use in other scenarios, because it is less secure and more difficult to configure than L2TP.

IPSec tunnel mode is not specifically covered in this chapter, but by using the exercises in this chapter, you will learn enough to create this type of IPSec security association if necessary. L2TP is covered in Chapter 9. This chapter covers only host-to-host scenarios, which comprise the majority of IPSec configuration problems.

Establishing IP Security Using Internet Key Exchange

IPSec itself does not provide encryption or authentication algorithms. Instead, it provides a framework for existing algorithms to work within. Determining which encryption and authentication algorithms to use for an IPSec session is performed by a protocol within IPSec, the Internet Key Exchange (IKE) protocol. IKE establishes secure communications by proving trust between hosts, negotiating a compatible set of encryption and authentication algorithms, and performing other minor functions so that IPSec security associations (SAs) do not have to be manually specified and keyed. To prove trust between hosts, IKE requires that both hosts have knowledge of the same shared secret key. Once trust has been established between the hosts, bulk encryption keys are exchanged using public key cryptography and are automatically refreshed according to intervals defined in the IPSec policy. A reasonably short key refresh time reduces the lifetime of a key, and therefore the length of time that attackers have to use brute-force methods to crack it.

IPSec in Windows 2000

Windows 2000 makes the security of IPSec easily accessible by integrating it with the standard Windows 2000 configuration management system, Active Directory. By defining IPSec configuration using Group Policy and distributing secret keys using standard Windows mechanisms, administrators can completely automate the deployment of IPSec within an Active Directory forest.

Manual methods of keying, distributing secret keys, and configuring IPSec are also provided for situations in which centralized configuration management is not possible or is not a concern, such as when creating a security association between untrusted hosts or when establishing a single security association for a specific one-time purpose.

Distributing IKE Secret Keys

In Windows 2000, you can use three methods to distribute the shared secret key required by IKE to automatically negotiate security associations:

Use Kerberos to distribute the secret key within a domain or between trusted domains, and whenever a domain or trust relationship exists between the hosts involved.
Install certificates with private keys that are both rooted in the same trusted certificate authority (CA). Use certificates when you can't use Kerberos.
Type it directly in the IPSec filter (manual keying). Use manual keying when you can't use Kerberos or certificates, and use it for IPSec testing.

IPSec Within a Private Network

IPSec is most commonly used to secure traffic between hosts on the Internet, but it is designed to operate in any context where network layer security is important. Many modern private networks are very large IP networks, spanning vast distances. IPSec can be used within a private network in the following scenarios:

Within an Active Directory forest, IPSec can be used to authenticate or encrypt traffic between servers and clients as necessary. Authentication can be based on the Kerberos protocol to ensure security for specific applications.
Between clients or servers within a domain, creating encrypted or authenticated sessions is a simple matter of creating a Group Policy that organizes the clients and servers according to their roles and then applying a GPO with the proper IPSec configuration. Once the GPO is created and applied to the OUs, the policy will deploy automatically to the affected servers and clients, and IP communications will be secured as described in the policies.
Within a domain or between domains, where a trust relationship exists, configuring IPSec is particularly easy, because Kerberos can be used to provide the secret keys necessary for IKE to establish the requisite IPSec security associations without administrator intervention.

Determining IP Security Method by Server Role

Windows 2000 makes it easy to enable security within a domain, but you cannot simply require IPSec on all computers within a domain and consider the problem solved. Here are two potential issues:

IPSec can create a "chicken before the egg" problem. If IPSec is required by domain controllers, new computers that do not have IPSec configured cannot contact domain controllers to join the domain. Because IPSec keys are distributed only to domain members, they can't be used to configure a new computer. The inability to join a domain without an existing IPSec configuration and the inability to configure IPSec without an existing domain association creates a mutually exclusive problem that can prevent new computers from being added to a domain. To avoid this problem, domain controllers must be configured to allow both IPSec and non-secure communications.
Overly restrictive IPSec policy can cause widespread problems for administrators because of the increased complexity required to administer IPSec, the addition of a new possibility for failure, and the fact that encrypted communications can make many key troubleshooting tools, such as sniffers, impossible to use.

Rather than applying blanket IPSec policy within your domain, consider moving sensitive applications and data to a few servers (as few as possible, considering the amount of data and the various security compartmentalization issues you have) and then requiring IPSec security on those servers only.

Certain services within your domain should be available whether or not clients can negotiate IP security with them. These services include domain log on and authentication, DNS services, DHCP, and other infrastructure services. If these services are not available to all clients, you might wind up with mutually exclusive requirements that cannot be resolved, such as computers that cannot receive an IP address from a DHCP server without already having an IPSec connection (which requires a valid IP address).

You can apply IPSec to computers in Windows 2000 by choosing one of three different policy modes:

Require security (the default Secure Server role)
Request security without requiring it (the default Server role)
Respond to security requests without making them (the default Client role)

It is good practice to accumulate infrastructure services such as DNS and DHCP on domain controllers and to allow those servers to speak to any clients by setting their IPSec policy mode to request security without requiring it.

Clients within a domain should be configured to accept IP security if a server requests it, except for any clients that you explicitly want to prevent from accessing secure servers. For example, you might want to prevent access for computers used as guest kiosks, Internet browsing stations, or for untrusted users, such as students in a university environment.

Practice: Enabling IPSec Between Domain Members

Trade secrets at Fabrikam, Inc. have to be transmitted using authentication and encryption even within the enterprise. To facilitate this requirement, all trade secret information is stored on member server ms01.domain.Fabrikam.com, which you will configure to require IPSec security. To allow access to the information, you will configure clients to respond to IP Security requests.

In this practice, you create a set of GPOs to require security for member servers and enable IPSec on clients if the servers request IPSec. This configuration will prevent servers within the Secure Servers OU from communicating with any client that is not specifically configured as an IPSec client, which by extension eliminates any computers that are not within the same or a trusted domain.

Exercise 1: Configuring IPSec Logging and Monitoring

The first step in any IPSec configuration session is to allow troubleshooting by enabling IPSec logging. Follow the procedures in this exercise to enable IPSec logging.

Perform this exercise while logged on to the domain controller as the Administrator.

To enable IPSec security logging

Click Start, point to Programs, point to Administrative Tools, and click Active Directory Users And Computers. The Active Directory Users And Computers management console appears.
Right-click domain.fabrikam.com, and click Properties.
In the domain.fabrikam.com Properties dialog box, click the Group Policy tab.
Double-click the Domain Security Policy GPO. The Group Policy management console appears.
Expand Computer Configuration, Windows Settings, Security Settings, and Local Policies.
Click Audit Policy. Your screen should now appear similar to Figure 8.1.

Figure 8-1. Audit Policy in the Group Policy management console
Double-click Audit Logon Events. The Security Policy Setting dialog box, shown in Figure 8.2, appears.

Figure 8-2. Enabling IPSec auditing
Select Define These Policy Settings, select Success, select Failure, and click OK.
Double-click Audit Object Access. The Security Policy Setting dialog box appears.
Select Define These Policy Settings, select Success, select Failure, and then click OK.
Close the Group Policy management console.
Click OK to close the domain.fabrikam.com Properties dialog box.
Leave the Active Directory Users And Computers management console open for the next procedure.

Exercise 2: Enabling IPSec on Servers

In this exercise, you create an OU for secure servers and apply an IPSec policy that forces them to require security to communicate.

Perform this exercise while logged on to the domain controller as the Administrator with the Active Directory Users And Computers management console open.

To create organizational units for secure servers

Right-click the domain.fabrikam.com domain, point to New, and then click Organizational Unit. The New Object Organizational Unit dialog box appears.
Type Secure Servers in the Name box, and click OK.
Click the Computers container, as shown in Figure 8.3, in the domain.fabrikam.com domain.

Figure 8-3. Member servers and computers are located in the Computers folder by default
Right-click MS01, and click Move to open the Move dialog box.
Click Secure Servers, and then click OK.

MS01 is removed from the list in the Computers container and appears in the list of Secure Servers.

To create a GPO for Secure Servers

Right-click the Secure Servers OU, and click Properties.
In the Secure Servers Properties dialog box, click the Group Policy tab.
Click the New button. A new GPO appears in the Group Policy Object Links list.
Type Secure Servers IPSec Policy as the name of the GPO and press Enter.
Double-click Secure Servers IPSec Policy. The Group Policy management console appears.
Expand Computer Configuration, Windows Settings, and Security Settings.
Click IP Security Policies On Active Directory. A list of available IPSec policies appears in the right panel.
Right-click the Secure Server policy, and click Assign.
Close the Group Policy management console.
Close the Secure Servers Properties dialog box.
Leave the Active Directory Users And Computers management console open for the next exercise.

Exercise 3: Enabling IPSec on Clients

In this exercise, you create an OU for clients that will be able to establish IPSec communications.

To create an OU for secure clients

Perform this procedure while logged on to the domain controller as the Administrator, with the Active Directory Users and Computers management console open.

Right-click the domain.fabrikam.com domain, point to New, and then click Organizational Unit. The New Object Organizational Unit dialog box appears.
Type Secure Clients in the Name box, and click OK.
Click the Computers container.
Right-click CLIENT01, and click Move to open the Move dialog box.
Click Secure Clients, and click OK.

CLIENT01 is removed from the list in the Computers container and appears in the list of Secure Clients.

To create a GPO for secure clients

Perform this procedure while logged on to the domain controller as Administrator with the Active Directory Users And Computers management console open.

Right-click the Secure Clients OU, and click Properties.
In the Secure Clients Properties dialog box, click the Group Policy tab.
Click the New button. A new GPO appears in the Group Policy Object Links list.
Type Secure Clients IPSec Policy as the name of the GPO and press Enter.
Double-click the Secure Clients IPSec Policy. The Group Policy management console appears.
Expand Computer Configuration, Windows Settings, and Security Settings.
Click IP Security Policies On Active Directory. A list of available IPSec policies appears in the right panel.
Right-click the Client policy, and click Assign.
Close the Group Policy management console.
Close the Secure Clients Properties dialog box.

To test IPSec connectivity between computers

Perform this exercise on the workstation CLIENT01.

Click Start, and click Run. The Run dialog box appears.
Type ipsecmon and click OK. The IP Security Monitor appears as shown in Figure 8.4.

Figure 8-4. The IP Security Monitor
Click Start, point to Programs, Accessories, and then click Command Prompt. The command prompt appears.
Type net view \\ms01 and press Enter.

The command prompt displays a list of shares on the server. The list might be empty.
Type exit and press Enter.

The IP Security Monitor now shows a security association existing between the client and the server, as shown in Figure 8.5.

Figure 8-5. The list of active security associations in the IP Security Monitor
Close the IP Security Monitor.

Exercise 4: Enabling IPSec on Domain Controllers

In this exercise, you configure domain controllers to respond to secure requests if an IPSec security association can be converged. IPSec is said to have converged if IKE can successfully negotiate a compatible set of encryption and authentication protocols at both ends and successfully transmit data between hosts. By allowing domain controllers to accept secure communications, you enable secure servers that require security to communicate with them.

In this scenario, because you're using default IPSec templates, Internet Control Message Protocol (ICMP) communications will pass even when an IPSec has not been negotiated, but higher-level communications will fail.

To test connectivity to the domain controller

Perform this exercise on the member server ms01.

Click Start, and click Run. The Run dialog box appears.
Type cmd and click OK. The command prompt appears.
Type ping dc01.

Notice that dc01 can be pinged.
Type net view \\dc01.

Notice that the operation fails, as shown in Figure 8.6.

Figure 8-6. Testing high-level network connectivity without an IPSec security association
Close the command prompt window.

To enable discretionary IPSec on domain controllers

Perform this procedure while logged on to the domain controller as the Administrator.

Click Start, point to Programs, point to Administrative Tools, and click Active Directory Users And Computers. The Active Directory Users And Computers management console appears.
Expand domain.fabrikam.com.
Right-click Domain Controllers, and click Properties.
In the Domain Controllers Properties dialog box, click the Group Policy tab.
Click New. A new GPO appears in the Group Policy list.
Type IPSec Policy for Domain Controllers and click Edit.
Expand Computer Configuration, Windows Settings, and Security Settings and click IP Security Policies.
Right-click the Server policy, and click Assign.
Close the Group Policy management console.
Click OK to close the Domain Controllers Properties dialog box.
Close the Active Directory Uses And Computers management console.

To test connectivity to the domain controller

Perform this exercise on the member server.

Click Start, and click Run to open the Run dialog box.
Type cmd and click OK. The command prompt appears.
Type ping dc01.

Notice that dc01 can be pinged.
Type net view \\dc01. The operation succeeds, as shown in Figure 8.7.

Figure 8-7. Testing high-level connectivity with an IPSec security association in place
Close the command prompt window.

Lesson Review

The following questions are intended to reinforce key information in this lesson. If you are unable to answer a question, review the lesson and try the question again. Answers to the questions can be found in the appendix.

What are the two primary methods IPSec uses to authenticate and encrypt IP packets?
What are the two encrypted payload modes that IPSec supports?
Explain the difference between transport mode and tunnel mode.
How does IKE determine whether to trust the participants when it establishes a security association?
How is IPSec managed in Windows 2000?
What mechanism would you use to distribute secret keys automatically in a domain?

Lesson Summary

IPSec provides mechanisms to authenticate and encrypt IP traffic between hosts in any IP network. IPSec operates at the network layer and is transparent to higher-level applications, which need not be aware of IPSec to benefit from it.
Authenticated Headers (AH) provides the functionality to authenticate packets and guarantee that they have not been modified. Encapsulating Security Payload (ESP) provides the functionality to encrypt packet payloads. ESP transport mode encrypts data, while tunnel mode encrypts and encapsulates entire packets.
Internet Key Exchange (IKE) is used to prove trust between hosts using shared secrets, and it creates security associations by negotiating keys and compatible encryption and authentication algorithms. IKE also periodically refreshes keys in existing security associations.
IPSec in Windows 2000 is managed using Group Policy and can use Kerberos secret keys as IKE secret keys. With these two capabilities, IPSec security can be deployed throughout an Active Directory forest at large scale using only Group Policy configuration.
A computer's role on the network determines which of three IPSec negotiation methods should be used. Secure servers should require security, standard servers should request security, and clients should respond to security requests.