Windows IPSec

Microsoft first introduced IPSec in Windows 2000. When enabled, the driver file called IPSec.sys handles packet inspection and overall IPSec operations. Another file, Oakley.dll, is the IKE component. The IPSEC Services service also needs to be running, although it should be by default. IPSec policies can be created at the command prompt (using different tools and commands between Windows 2000, XP, and 2003) or using the IP Security Policies console. IPSec policies can be created and applied locally, or pushed down using a GPO. A computer can only have one active IPSec policy at one time, stored in HKLM\Software\Policies\Microsoft\Windows\IPSec.

Note 

Windows Vista will allow IPSec configuration using a new console called Windows Firewall with dvanced Security.

IPSec Console

The most convenient way to configure IPSec is to use the IP Security Policies console. To use it, open the Microsoft Management Console (Mmc.exe) and choose File ð Add/Remove Snap-in. Click the Add button, choose the IP Security Policy Management snap-in (see Figure 8-3) and click the Add button. Select the correct focus (usually the Local Computer) and then click Finish ð Close ð OK.

Figure 8-3

IPSec policies can also be created in GPOs and Local Computer Policies (under Computer Configuration\Windows Settings\Security Settings\IP Security Policies). This chapter will use the console in most of the examples.

IPSec at the Command Line

IPSec can also be controlled and configured at the command line. Windows 2000 uses the Ipsecpol.exe command (www.microsoft.com/windows2000/techinfo/reskit/tools/existing/ipsecpol-o.asp). Ipsecpol syntax can be a bit intimidating for the casual user. Some examples are provided at http://support.microsoft.com/default.aspx?scid=kb;en-us;813878. The following example syntax would block all outbound connections to any remote computer over UDP port 1434:

 ipsecpol -w REG -p "Block UDP 1434 Filter" -r "Block Outbound UDP 1434 Rule" -f 0=*:1434:UDP -n BLOCK 

As cryptic as it may seem, Ipsecpol, and other configuration tools like it, allows IPSec policies to be generated quickly when using batch files and scripts. An admin requiring IPSec during a troubleshooting event could send a single batch file to a remote user that, when run, would completely configure and enable IPSec on the remote computer. Windows XP uses Ipseccmd.exe and Windows Server 2003 uses Netsh ipsec.

IPSec Monitor Tools

To use any of the IPSec monitoring tools you need at least a basic understanding of IPSec and its SAs. Windows 2000 computers can use the command-line tool Ipsecmon.exe. Windows XP computers can use the command-line tool Ipseccmd.exe with the show all syntax. Windows Server 2003 and later computers can use the Netsh ipsec static show or Netsh ipsec dynamic show commands. Better yet, Windows XP and later computers can use the IP Security Monitor snap-in GUI console. Any of these tools will show you whether or not an IPSec connection has been made and basic statistics (see Figure 8-4).

Figure 8-4

Detailed IPSec packet traffic can be recorded using many network packet sniffers, including Ethereal (www.ethereal.com) and Microsoft's own Network Monitor (Windows 2000 and later, and SMS versions). Figure 8-5 shows a small packet trace in Ethereal resulting from a ping test. You can see at the IP layer that the protocol number is hexadecimal 32 (which equates to decimal 50 or ESP). The SPI value for the SA is highlighted. Of course, if ESP encryption is enabled, you will not be able to see any payload data.

Figure 8-5

You can turn on rudimentary Oakley logging that can be enabled for the IPSec troubleshooting professional. When turned on, the Oakley.log file will be stored in %Windir%\Debug. I can be turned on in Windows XP by typing in Ipseccmd set logike, or in Windows Server 2003 by typing in Netsh ipsec dynamic set config ikelogging 1. Windows 2000 requires that a new registry key and value be made. It is HKLM\System\CurrentControlSet\Services\PolicyAgent\Oakley\EnableLogging and is set to 1. A new Oakley.log file is created each time IPSec is started. Previous versions will be saved as Oakley.log.sav.

You can also enable IPSec driver events to be logged to the Event log. To enable, set HKLM\System\CurrentControlSet\Services\IPSEC\DiagnosticMode to 1 and then restart the computer. IPSec driver events are written to the System log once every hour. You can also see IPSec policy changes in the Security log if you enable Audit policy change. IKE events can be recorded by enabling Audit logon events.

Default IPSec Policies

As Figure 8-6 shows, this will reveal three default IPSec policies in environments that have not modified the defaults. The three default polices are example policies that Microsoft created in order to help people create their own. Although they can be used, their settings are heavy-handed, meaning all traffic is treated a certain way. Custom IPSec policies are more flexible, and they are a good place to start learning.

Figure 8-6

Table 8-1 describes the three default policies. All three policies use Kerberos to authenticate to computer peers, involve all IP traffic, and will use ESP encryption if allowed.

Table 8-1
Open table as spreadsheet

Default IPSec Policy Name	Description
Client (Respond Only)	Will only enable IPSec security if requested to by another end-point
Secure Server (Require Security)	Requires IPSec security on IP traffic, but allows ICMP traffic to pass without security
Server (Request Security)	Requests, but doesn't require, IPSec security on IP traffic, but allows ICMP traffic to pass without security

You can create as many IPSec polices as you like, but only one can be enabled (or assigned) per computer at any one time. You can assign a policy in the console by right-clicking the policy and choosing Assign. You can also delete, rename, and modify existing policies or create your own by clicking on a blank area in the console and choosing Create IP Security Policy.

Setting Up Windows IPSec

IPSec is a flexible framework. Unfortunately, that flexible framework and its myriad of terms can make IPSec configuration confusing. This section will lead you through the steps to create a very simple IPSec policy for securing web traffic to an internal server. As we go through the process of setting up an IPSec policy, the details might be confusing for a first-time IPSec policy creator. The basic IPSec setup process (see Figure 8-7) is as follows:

1. Only one IPSec policy can be enabled per computer.

2. Each IPSec policy can have multiple rules, each determining how to handle a particular type of traffic (as determined by the associated filter).

3. Each rule can only have one IP filter that defines a particular class of traffic (by IP address, protocol number, or port number).

Figure 8-7

Creating an IPSec Policy

The first step is to open the IP Security Policies console, right-click a blank area in the right-pane, choose Create IP Security Policy, and then click the Next button. When prompted, type in a name and description (see Figure 8-8) for your policy, and then click the Next button.

Figure 8-8

Next, you will be prompted about whether to Activate the default response rule. Enabled by default, the Default Response rule can be deactivated but not deleted. It can be used in any IPSec policy (as can most IPSec rules). The Default Response rule tells the affected computer to respond to any IPSec request by enabling ESP security if it matches the other endpoint request (or authentication if it matches the other endpoint request), and if not does not require security. In most cases, leaving the rule enabled will not causes problems, although in this case, I know we don't need it, so unselect the checkbox and click the Next button. Keep the Edit properties checkbox selected and click the Finish button.

IPSec Rules

Now you can create or choose one or more IPSec rules. IPSec rules determine how to treat particular types of traffic (as defined by the IPSec filters). You can have one or more IPSec rules active in any single IPSec policy, but every IPSec rule is attached to only one IPSec filter. Click the Add button and then the Next button. The next dialog box will prompt the creator to input tunnel endpoints if necessary. Tunnel endpoints are only needed when the host on which the IPSec policy will be enabled will be a gateway endpoint participating in IPSec tunneling mode. If so, the creator would input the IP address of the other endpoint's network interface connecting the tunnel. We are setting up a transport mode policy, so just leave This rule does not specify a tunnel endpoint enabled and click the Next button.

Connection Type

When prompted to select the network connection type in the Network Connection dialog box, you can choose All network connections, Local Area Network (LAN), or Remote access. Choose the appropriate option, although in this example, just accept the default of All network connections and click the Next button. Which option you choose is also dependent on whether you will be requiring just a particular type of traffic to be secured, or choosing to filter most traffic (and just to allow particular traffic). The latter case should always be All network connections, whereas the former might be more granular.

Authentication Method

As Figure 8-9 shows, the next decision is asked in the Authentication Method dialog box. In Windows IPSec, you can choose between three methods to authenticate the endpoints to each other:

• Kerberos authentication

• Certificate from a certification authority (CA)

• Pre-shared key

Figure 8-9

IPSec will not work unless the participating nodes can authenticate (i.e., machine authentication) to each other. You can choose up to three authentication methods and place them in order of precedence. If an end node can't do the first one listed, the others are offered up. The most secure method is the certificate, but it is also the most difficult to set up. First, each participating IPSec endpoint must have a machine certificate signed by a certification authority (CA) designated in the dialog box. Microsoft Certificate Services in Windows Server 2003 fills the bill, although its setup and configuration is beyond the scope of this book. For more information on setting up a CA using Microsoft Certificate Services, read Microsoft's Windows Server 2003 PKI Certificate Security (ISBN 0735620210). Although the certificate method is the hardest to deploy initially, it is the most secure method. It can be used on the LAN, across domains and forests, and remotely over the Internet.

Note 

The certificate selected in this step is the CA's certificate, not the endpoint's machine certificate.

The Kerberos authentication option is the easiest to deploy — just enable the selection and let Windows do the rest. In order for the Kerberos method to be used, the participating IPSec endpoint must have a valid computer account in the forest. If the Active Directory forest is a Windows Server 2003 forest, then Kerberos can be used across forests when a forest trust is established. The Kerberos method is the easiest to deploy, but it doesn't work well across the Internet or outside the trusted forest(s).

The pre-shared key (PSK) is also fairly easy to deploy. Just input an identical PSK (such as a password) on both sides of the IPSec connection and IPSec will do the rest. It works locally, across forests and domains, and over the Internet. Unfortunately, it is also the least secure method. Microsoft recommends only using it to test IPSec. I use PSK when initially setting up new IPSec connections to help rule out other authentication problems. After the PSK link is up, I move the IPSec connection to one of the other two methods.

If you use a PSK, be sure to enter a very long (longer than 15 characters) and complex PSK. Unlike most password treatments, the PSK will be displayed in plaintext in the IPSec policy window, so don't expect it to remain hidden. If you use PSKs, you should change them frequently. Unfortunately, much of the world uses short, non-complex PSKs that are never changed. This is an IPSec hacker's dream. As covered below, malicious hackers can capture traffic encrypted with the PSK and brute force it to learn the PSK. Once the PSK is discovered, it is trivial to exploit the rest of the IPSec protections. For this example policy, choose the weaker PSK method and type any short word as a PSK. I'm asking you to use the weakest method so you can see how it is displayed in the policy. Then click the Next button.

IPSec Filters

You can create as many IPSec filters as you like, and they are available for any IPSec policy, but you can only associate one IPSec filter per IPSec rule. Each IPSec filter can have one or more types of traffic (i.e., multiple protocols) associated with it. By default, Windows IPSec has two filters to choose from: All ICMP traffic and All IP traffic (Figure 8-10 shows the two defaults plus one custom filter). Let's create our own new rule to cover web traffic over TCP port 80. In the IP Filter List dialog box, click the Add button.

Figure 8-10

Type a filter name and description. Filter names are displayed in the IPSec Policy rules list in reverse alphabetical order. Their listed order does not affect the order in which they are applied if multiple rules exist, but you may want to consider the name carefully before entering one if you want a particular filter listing order. In this example, type Web Port 80 Traffic Heading to Internal Web Site and add an appropriate description. Click the Add button and then the Next button. The IP Traffic Source dialog box will then prompt you for the IP address for the traffic you plan to filter (see Figure 8-11).

Figure 8-11

You can choose among the following options:

• My IP Address

• Any IP Address

• A specific DNS name (XP only)

• A specific IP Address

• A specific IP Subnet

• DNS Servers <dynamic>

• WINS Servers <dynamic>

• DHCP server <dynamic>

• Default Gateway <dynamic>

My IP Address is the first network card's first IP address if there are multiple network cards or multiple IP addresses. Windows 2000 has only the Any IP Address, A specific IP Address, and A specific IP Subnet options. Only XP has A specific DNS name (especially useful when an endpoint has a DHCP-assigned IP address that could change. Only Server 2003 and later have the latter four options. If they are chosen, IPSec will take the current values for those fields and update them if the related IP addresses change.

If the IPSec rule involves traffic headed from the current PC to another endpoint, choose the first option or specify the correct local IP address. If you want to filter traffic headed from another endpoint, choose its IP address or choose Any IP Address. For this example, choose My IP Address. When finished, click the Next button. You will then be prompted to fill in the IP Traffic Destination value in the next dialog box and click the Next button. For this example, choose A specific IP Address and fill in an IP address for a web server host you know that you can enable IIS and IPSec on for testing purposes.

IP Protocol Type

In the IP Protocol Type dialog box, you must indicate the IP protocol number for the involved IP transport layer protocol, or you can choose the Any option to allow the filter to inspect all transport layer protocols. In most cases, the value will be TCP, UDP, or ICMP, but there are many other values you can cycle through. Choose TCP for this example and then click the Next button.

IP Protocol Port

In the next dialog box, you must choose source and destination transport layer protocol numbers (e.g., 25 for SMTP, 53 for DNS, 80 for WWW, 443 for HTTPS, etc.). If most cases, if the current computer is connecting to another host, its source port number will be randomly generated as a number higher than 1023. In order to handle this, you must choose the From any port option. If the current PC is connecting to another computer's service with a well-known port, choose To this port and fill in the appropriate port number. In this example, type 80 (see Figure 8-12), click the Next button, and then click the Finish button.

Figure 8-12

This will take you back to the IP Filter List dialog box. You can add more IP addresses and protocols to a single IPSec filter. In this example, add port 443. You should have a dialog box that resembles the one shown in Figure 8-13. Click the OK button and then select the new custom IP filter created (see Figure 8-10 again) and click the Next button. Filters can also be mirrored. Because an IPSec SA is a unidirectional entity, two-way communications (like most network traffic requires) needs an IPSec filter that mimics the original rule, only going in the reverse direction. In most cases (not all), enabling mirroring is necessary. Mirroring must often be disabled when setting up IPSec in tunnel mode.

Figure 8-13

In summary, an IPSec IP filter allows network traffic to be intercepted according to the following characteristics:

• Source IP address

• Destination IP address

• Source address IP subnet mask (if a specific IP subnet is chosen)

• Destination address IP subnet mask (if a specific IP subnet is chosen)

• Source transport port

• Destination transport port

• IP protocol type

Each one of these characteristics can be defined to create one or more IP filters, which are then attached to a Filter Action within an IPSec rule.

Filter Action

Figure 8-14 shows the next dialog box, Filter Action, which contains three default options:

• Permit

• Request Security (Optional)

• Require Security

Figure 8-14

Permit allows the filtered traffic through without requiring IPSec security. Request Security will ask the other endpoint to enable IPSec encryption or authentication, but will still connect if the request is not met. Require Security requires that IPSec ESP encryption be enabled between the two endpoints for that traffic type before the filtered network traffic can be communicated.

You can accept and use one of the default three security methods or create your own and place them in order of precedence. The Request Security (Optional) option (see Figure 8-15) defines four security methods. The first method attempts to use the most secure encryption method currently available in Windows IPSec, 3DES. If the connecting endpoint cannot do 3DES, it will attempt to negotiate DES. Both of these methods have ESP integrity checking with SHA-1 enabled. If encryption can't be done, AH authentication is attempted, first using the stronger SHA-1, and lastly, MD5 if nothing else works. The Require Security method never attempts to use AH, only ESP encryption and integrity using various combinations of 3DES, DES, SHA-1, and MD5 (from strongest to weakest).

Figure 8-15

You can create your own security method and enable it in the Filter Action dialog box. Whatever security method you choose, the other connecting endpoints must support the same method in order for IPSec protection to be enabled. As Figure 8-16 shows, there are other options as well.

Figure 8-16

The Accept unsecured communications, but always respond using IPSec option tells an IPSec-enabled computer to allow unsecured requests, but to always reply using IPSec. While this would prevent a computer without IPSec from participating as an endpoint, it is a normal option to choose when the other endpoint uses the Default Response rule or Client (Respond Only). The Allow unsecured communications with non IPSec-aware computer option is self-explanatory. Normally this option is only enabled when you want a computer that is IPSec-aware to always enable IPSec when contacted by IPSec-enabled computers but also still be able to initiate connections to computers not capable of IPSec. A good example of this is when you want to use IPSec inside your LAN, but not require it when it is connecting to Internet resources (which are probably not using IPSec). The Session key perfect forward secrecy (PFS) option requires more explanation.

Perfect Forward Secrecy

Perfect Forward Secrecy (PFS) is an IPSec option (discussed in the IPSec RFCs) that attempts to ensure that a compromise of any single IPSec key will not allow access to data protected by other keys. Master key PFS works by ensuring that a new IKA SA secret key is regenerated for each new session. Normally, without PFS, individual SA secret keys are generated from the IKE SA "master" secret key, which means there is a cryptographic link an intruder can follow.

If master key PFS is enabled and an intruder breaks the IKE SA secret key, it doesn't immediately give them help in cracking the individual SA keys. The attacker would have to attack each SA secret key individually, and even then it would only see the data protected by each SA secret key. One key compromise doesn't make it easier to compromise another. Without PFS, an attacker could use the information learned by breaking the IKE SA secret key to quickly compromise the SA private keys. Without PFS enabled, the master key is only regenerated when it has already been used to generate a predetermined maximum number of SA secret keys. Microsoft also allows PFS to be set on the SA private keys to further complicate cracking efforts.

Enabling PFS does provide stronger security, although in Windows it is disabled by default because it adds additional computational overhead and security often not needed in normal security environments. An environment with high-security requirements or expecting attacks against its IPSec infrastructure should enable PFS.

In summary, an IPSec rule can contain the following characteristics:

• Tunnel endpoint (if IPSec tunnel mode is desired)

• Connection Type (All, LAN, or remote)

• Authentication Method (Kerberos, Certificate, Pre-Shared Key)

• Filter List (covered above)

• Filter Action (Permit, Block, Negotiate Security, or custom)

• Other IPSec Rule Options

An IPSec policy can have one or more IPSec rules. When multiple rules are in effect, Windows automatically orders them operationally from most specific to least specific. This means that blocking or allowing specific endpoints will take precedence over larger ranges of affected computers. Some IP traffic is never considered or blocked in Windows IPSec.

IPSec Exemptions

By default, Windows makes certain exemptions (packet traffic types that cannot be blocked or controlled by IPSec) when IPSec is turned on. In Windows Server 2003, only IKE traffic, needed for IPSec establishment, is exempted from IPSec filtering. In Windows 2000 and XP, there are many more exemptions (see http://support.microsoft.com/kb/253169):

• Kerberos authentication

• IKE

• Multicast traffic

• Broadcast traffic

• RSVP traffic

You can also configure a registry key to control exemptions. HKLM\System\CurrentControlSet\Services\IPSec\NoDefaultExempt can be configured with a value from 0 to 3, as shown in Table 8-2.

Table 8-2
Open table as spreadsheet

No Default Exempt Value	Description
0	Default in Windows 2000 and XP. Exempts Kerberos, IKE, Multicasts, Broadcasts, and RSVP.
1	Recommended exemption setting by Microsoft. Only exempts Multicast, Broadcast, and RSVP traffic. Requires SP1 or later in Windows 2000. Default setting in Windows 2000 SP4 and Windows XP SP2.
2	Only available in Windows Server 2003. Exempts only
3	Default in Windows Server 2003. Only IKE exempted. Not available in XP or 2000.

Note 

Caution: Although a setting of 3 might be desired for all participating IPSec nodes, requiring all broadcast traffic to have IPSec could cause a lot of problems (e.g., NetBIOS, ARP, etc.). For that reason, Microsoft's recommendation of putting the setting to 1 is a realistic trade-off setting. In the future, it would be nice for Microsoft to have a setting to only exempt IKE and broadcast traffic, or even better, let the end user define all exemptions. You can also set the NoDefaultExempt value on Windows Server 2003 using Netsh ipsec dynamic set config ipsecexempt value = command. See http://support.microsoft.com/kb/811832/EN-US for more details on exemptions.

For our current IPSec policy example, choose the Require Security option, and then click the Next and Finish buttons. If done correctly, the IPSec wizard will display the IPSec Rule as configured among multiple tabs (see Figure 8-17). This is the same view you would see if you disabled the Create a IP Security Policy wizard. As shown in Figure 8-17, for this example I added another two authentication protocols beyond the PSK method. I then placed the PSK method lowest down on the list since it has the most risk involved. Click the OK button to save the rule to the IPSec policy.

Figure 8-17

Once saved, you can edit the IPSec policy. You can add, delete, or edit rules/filters attached to the IPSec policy on the Rules tab. As discussed above, rule names are based upon IP filters and are displayed in reverse alphabetic order. The listed order has nothing to do with how they are applied. Rules/filters are applied from most specific to least specific (i.e., broad). On the IPSec policy General tab (see Figure 8-18) you can further define IPSec key characteristics, including whether the IKE SA uses Master key perfect forward secrecy.

Figure 8-18

Once you are sure the IPSec policy meets the required criteria, you need to ensure that the other endpoint has a similar policy with mirrored rules and filters configured. You can manually create the mirrored policy, use one of the IPSec command-line tools and a script to create it, or, in the console, right-click the IP Security Policies on Local Computer option, and choose All Tasks and then Export Policies. The resulting file can then be imported on the other endpoint and the appropriate modifications made.

When both sides have the correct and aligned IPSec policies, you can assign them and test. If you are unsure about its functionality, use the IPSec Monitor console (or a tool) or a network sniffer to validate whether or not IPSec is working as intended. Additional configuration may be needed outside of IPSec in order to get it working. See Microsoft Knowledge Base article 816514 (http://support.microsoft.com/default.aspx?scid=kb;en-us;816514) for more step-by-step IPSec instructions.

Firewall Ports Needed

If a firewall is inline between the two participating IPSec endpoints, the following ports must be opened:

• UDP port 500 (for IKE traffic)

• UDP port 4500 (for NAT-T traffic only)

• IP Protocol 50 and 51

If Windows Firewall is enabled (along with the Application Layer Gateway service) when IPSec is enabled, it will automatically open up these ports. However, most other ports will need to be opened in IPSec for two hosts to communicate. For example, don't forget to allow DNS (TCP and UDP port 53) for clients needing DNS resolution. Most Windows hosts need NetBIOS naming resolution as well (ports 137 and 445).

Windows Firewall IPSec Bypass

In Windows XP Pro SP2 and Windows Server 2003 SP1, Microsoft has added an interesting Windows Firewall feature. If the GPO setting WindowsFirewall: Allow Authenticated IPsec bypass is enabled (located in a GPO or Local Computer Policy at Computer Configuration\Administrative Templates\Network\Network Connections\Windows Firewall), users and groups protected by IPSec can be allowed to bypass all Windows Firewall filtering. In order for a user or group to be allowed to bypass Windows Firewall filtering, the user or group must be communicating using IPSec secured traffic (i.e., traffic covered by a rule) and be specified in the GPO setting (see Figure 8-19). Unfortunately, the setting is severely hampered by the fact that users or groups can only be specified by a Microsoft Security Descriptor Definition Language (SDDL) string. Prolific author Mark Minasi has an excellent article on SDDL use and syntax at www.minasi.com/showdoc.asp?docname=nws0502.htm.

Figure 8-19

Note 

Windows Vista has a much-improved interface for IPSec and the Windows Firewall called the Windows Firewall with Advanced Security console.