Section 11.4. IP Security

11.4. IP Security

IPSec is designed to secure traffic at the IP layer in a way that is transparent to the other layers and applications. By operating at such a low level, user applications and protocols that the OS uses need not be concerned with how security is applied; they can act normally, and the mechanics of IPSec are taken care of on an entirely separate level. IPSec addresses the weaknesses in the plain IP protocol by providing:

Authentication of hosts before and during communications
Confidentiality through encryption of IP traffic
Integrity of IP traffic by identifying modified or spoofed traffic
Prevention of replay attacks

You might be wondering how VPNs differ from the use of IPSec. If you are concerned only with securing traffic to and from a destination, there is little difference between the use of IPSec and the use of a VPN. VPN, however, comes with its own support burden, both on the end user and on the administrator, and in some cases VPNs are just too much to add to an administrator's current workload. On the other hand, VPNs also enable more complete functionality such as file sharing, browsing on the destination network, and other features that more closely mimic a "local" connection at the destination network. IPSec, meanwhile, supports only security features and does not add any of this type of functionality, and although the processor is pegged a bit more on both the client and server end because of the encryption and decryption routines, it has less of an effect on system resources than a VPN does. The bottom line comes down to whether you are considering using a VPN only for security purposes, in which case you might consider using a lighter-weight IPSec deployment instead, or if you want more localized features of the destination network, in which case a VPN would be a more logical choice.

IPSec in Windows consists of three main components: the policy agent, the Internet Key Exchange (IKE) module, and the IPSec driver. The policy agent collects the IPSec policies that the administrator has configured and then distributes the policies to the IKE module or IPSec driver. This component runs as the "IPSec Services" service in Windows Server 2003.

The IKE module negotiates security associations (SAs) for both the SA and key management systems and IPSec itself. It does this based on the authentication and security settings it receives from the policy agent. The IKE component is actually started (stopped and restarted, too) by the policy agent service.

And finally, the IPSec driver is responsible for executing the filters and monitoring connections to make sure they do not conflict with policy.

11.4.1. How IPSec Policies Work

Essentially, an IPSec policy consists of filter lists, which contain filters that match inbound and outbound traffic over a certain type of connection that has certain characteristics, such as IP address, port, or protocol. These filters have actions associated with them, which tell Windows what to do with the traffic that matches the filter. When traffic matches the filter and actions are taken with the traffic, actions consist of several methods of security that are used for authentication and key exchange between the two hosts that are exchanging the filtered traffic. All of these elements come together into an IPSec rule, which is the easiest unit of management when you're dealing with an organization full of security policies.

Filters can restrict traffic via one of four methods:

Block the traffic and drop it before even processing it
Encrypt the traffic before sending it on its way
Sign the traffic to ensure its contents aren't changed during transmission and reception
Permit the traffic to flow, untouched by IPSec

These filters can restrict that traffic:

By the IP address, subnet, or fully qualified domain name of the source computer
By the IP address, subnet, or fully qualified domain name of the destination computer
By the port type (TCP, UDP, etc.) and number used in the transmission

Out of the box, Windows Server 2003 comes with three predefined security policies, which you can see in the default right pane of the IP Security Policy Management snap-in:

Client (respond only): This policy is used on computers that never need to transmit secure data but occasionally might need to respond to clients using secure transmissions.
Secure Server (require security): This policy, the strictest of the three, requires all inbound and outbound communications to be secure, and completely denies requests for insecure transmissions. This policy requires Kerberos.
Server (request security): Contrary to the popular definition of server, this policy can be used on machines acting in client or server roles that are required to protect all outbound transmissions but prefer to receive only secured inbound transmissions. This requires Kerberos.

Only one IPSec policy can be in force (or "assigned") at a time. Assigning a new IPSec policy will automatically unassign the policy in effect immediately before the change.

11.4.1.1. Deconstructing an IPSec policy

Let's examine the Secure Server policy to see what components comprise it. Right-click Secure Server and select Properties to raise the Secure Server (Require Security) Properties box, as shown in Figure 11-34.

You can see the three rules that make up this policy; all are active, as indicated by the accompanying filled checkboxes. Note the filter list, action, authentication method, and tunnel setting of each rule. You can edit a particular rule by clicking it and then clicking the Edit button, which brings up the Edit Rule Properties dialog box, as shown in Figure 11-35.

Each of the five tabsIP Filter List, Filter Action, Authentication Methods, Tunnel Setting, and Connection Typeallow you to edit the portion of the rule contained therein. You can use the Add and Edit buttons on each page to add new actions and methods to the rule. Click OK when finished.

Figure 11-34. Examining the properties of an IPSec policy

Figure 11-35. Editing an IP filter's rule properties

11.4.2. Creating an IPSec Policy

Now let's create an IPSec policy. In this example, I'll assume you want to block all traffic except that to ports 80 and 443, which is something you'd want to do for IIS web servers that operate on a standalone basis. (You would lose the ability to remotely manage these machines using the web management tools you learned about in Chapter 8, but I've already explained that it's a bad idea to use those, anyway.) Then you can modify the following procedure as appropriate for other rules you might want to create.

First, you need to create a filter action that describes what to allow and what to deny:

In the IP Security Policy Management snap-in, right-click IPSec Security Policies on Local Machine and then select Manage IP filter lists and filter actions.
Navigate to the Manage Filter Actions tab, and click Add to create a new filter action. Click Next to skip the introduction.
Type LockDownWeb-Permit as the name for the new filter action because we'll be using this action to permit web traffic. Click Next when you're finished.
The Filter Action General Options screen appears next, as shown in Figure 11-36. Select Permit, and then click Next.
Click Finish to complete the first filter action.
Create a second filter action called LockDownWeb-Block using the same procedure, except choose Block on the Filter Action dialog box this time.
Click Close once you're finished.

Figure 11-36. The Filter Action General Options screen

Now it's time to actually create the filters and lists based on the actions you just defined:

Right-click IPSec Security Policies on Local Machine and then select Manage IP filter lists and filter actions.
Click the Add button to create a new IP filter list.
Use AllTraffic for the filter list name.
Now, click the Add button again to create a new filter. Just select the default options through the IP Filter Wizard to create a filter that will catch all traffic.
Click Close when you're finished.
You'll be returned to the IP Filter List screen, shown in Figure 11-37. Now, click Add to create another filter list, and use CatchWebTraffic for the name.

Figure 11-37. The IP Filter List screen
Select Any IP Address from the Source address drop-down list, and then click Next.
Select My IP Address from the Destination address drop-down list, and then click Next.
Choose TCP from the Select a protocol type drop-down list, and then click Next.
The IP Protocol Port screen appears which is depicted in Figure 11-38. Select To this port and then indicate port 80.
Click Next and then Finish.

At this point, you've filtered all traffic except that flowing to port 80. If you have an SSL site that needs access, click Add, and then repeat this procedure to create another filter that permits traffic to go to port 443. You might want to make a note of the values presented in Table 11-1.

Figure 11-38. The IP Protocol Port screen

Table 11-1. Values to create IPSec rule for SSL web serving
Prompt	Information
Source Address	Any IP address
Destination Address	My IP address
Protocol	TCP
From Port	Any
To Port	443

11.4.3. IPSec Caveats

As with any technology, there are a few "gotchas " when it comes to using IPSec in production. For one, the protocol is supported only on Windows 2000 operating systems or later. And although the Microsoft version supposedly adheres to the IPSec protocol standardthe one all vendors are supposed to use when developing their implementations of a protocolsupport for interoperability can be limited with some more obscure vendors. Always be sure to test your scenarios in a lab with identical equipment to make sure you have no issues with that.

With that said, foremost on the list of things to consider is that IPSec by default won't work with NAT-based connections. However, there is a modification of the IPSec protocol called NAT Traversal which will allow bridging of IPSec communications, intact, through a NAT router or firewall and back to the other side of the transmission. You do need a NAT machine that knows about this traversal feature, and you also need Windows Server 2003 to do the encryptingprevious versions of Windows simply won't cut it.

Since the first edition of this book was published, Microsoft has released an L2TP/IPSec NAT-T update for 2000 and XP, which you can find at the following location:

http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B818043