Lesson 3: Troubleshooting IPSec

---

IPSec has the potential to modify every network communication sent to or from a computer. If that were not intimidating enough, IPSec is extremely complicated to configure. Your standard troubleshooting tools might not work either, since network traffic is usually encrypted. Therefore, you will need specialized skills to effectively troubleshoot problems with IPSec communications. This lesson covers common IPSec troubleshooting scenarios and describes how you should isolate and resolve the problems.

After this lesson, you will be able to

• Describe the first steps to take when isolating a communications problem that might or might not be related to IPSec.

• Describe the overall process you would use to troubleshoot IPSec problems.

• List common problems related to using Kerberos and public key certificate-based authentication.

• Describe how to avoid and resolve problems with firewalls and routers, Network Address Translation (NAT), and computers that do not run Windows.

Estimated lesson time: 25 minutes

General Troubleshooting Guidelines

Regardless of the type of problem you are experiencing, you should first make sure that the necessary services are started and set to automatic on both IPSec peers. On computers running Windows Server 2003, the IPSec Services service must be started. On computers running Windows 2000, the IPSec Policy Agent service must be started.

Sometimes, especially after making significant changes, you might be able to resolve a problem by restarting IPSec services. This completely clears the IKE negotiation state. You can restart IPSec services from a command prompt by running the following commands:

net stop policyagent net start policyagent

This is simply a quick way to restart IPSec without restarting the computer. After restarting the IPSec services on both computers, attempt to establish a secure connection. If the problem persists, restart the operating systems on both IPSec peers and try again.

Next, verify that the correct policy has been assigned to each of the IPSec peers by using the IP Security Monitor snap-in. Remember, creating a new policy does not automatically assign it to a computer. Additionally, if you deploy a policy by using a GPO, the settings do not propagate instantly. If the policy is different from the one you were expecting, identify the responsible GPO by using the Resultant Set Of Policy snap-in or the command Netsh Ipsec Static Show Gpoassignedpolicy.

Verify that IKE negotiation is succeeding by enabling auditing and checking the Security event log, as described in Lesson 2. If IKE negotiation is not succeeding, verify that authentication, encryption, and key change settings are the same on both IPSec peers. If there is no indication that negotiations are succeeding, use Network Monitor to verify that packets are reaching the correct computer.

If IKE negotiation is succeeding but IPSec communications are still failing, verify that Quick Mode is succeeding by checking the Security event log and the IKE tracing log. If Quick Mode is not succeeding, check to see that all Quick Mode encryption, integrity, key strength, and key change settings are the same on both IPSec peers.

If IPSec negotiations simply do not occur, or if the type of negotiations are different from what you expected, double-check the IPSec filters. If you are specifying source and destination IP addresses in the filter, verify that the filters on each of the IPSec peers are mirrors of each other. Likewise, if you are specifying source and destination port numbers, the remote IPSec peer must have the same port numbers configured, but with the source and destination numbers reversed.

Kerberos Authentication Problems

Kerberos authentication is the default IPSec authentication method. You can quickly identify whether IPSec connectivity problems are caused by authentication by temporarily changing the IPSec authentication method on both IPSec peers to Preshared Key. If IPSec communications succeed with Preshared Key, Kerberos authentication is probably the source of the problem.

For Kerberos authentication to be successful, both IPSec peers must have valid computer accounts in trusted domains, and they must be able to authenticate the remote computers. Each IPSec peer must be able to communicate with domain controllers without having the authentication requests filtered. In earlier versions of Windows, IPSec automatically allowed Kerberos traffic. However, the Kerberos protocol is no longer a default exemption in Windows Server 2003.

To modify the default filtering behavior for Windows Server 2003, use Netsh to set the IPSecExempt value to 0, 1, or 2. By default, the IPSecExempt value is set to 3, which specifies that only Internet Security Association And Key Management Protocol (ISAKMP) traffic is exempt from IPSec filtering. Specifying a value of 2 allows Resource Reservation Setup Protocol (RSVP), Kerberos, and ISAKMP traffic. A value of 1 allows multicast traffic, broadcast traffic, and ISAKMP traffic. Specifying a value of 0 allows multicast, broadcast, RSVP, Kerberos, and ISAKMP traffic. To specify the value, run the following command:

netsh ipsec dynamic set config ipsecexempt value={ 0 | 1 | 2 | 3} 

Both IPSec peers must authenticate to each other; therefore, the problem could be authenticating either of the two computer accounts. On both computers, enable the failure Audit Logon Events audit policy, as described in Lesson 2. Attempt to initiate an IPSec connection, and then review the Security event log on both computers. You should see a failure audit event indicating the nature of the error.

Certificate Authentication Problems

Certificates are a common method for authenticating computers that are not in a trusted domain environment. If you are experiencing problems with IPSec and want to verify that the problem is related to authentication, temporarily change the IPSec authentication method on both IPSec peers to Preshared Key. If IPSec communications succeed with Preshared Key but fail with certificates, the problem is almost certainly related to certificates.

If you have multiple rules in a policy, double-check that those rules will use the same authentication method consistently for any single remote computer. It is acceptable to have a policy that configures Kerberos authentication for hosts on an internal network and uses certificates for hosts on an external network. However, you cannot create one rule that uses Kerberos to authenticate just Transmission Control Protocol (TCP) data and a second rule that authenticates User Datagram Protocol (UDP) traffic by using certificates, for example. The IP Security Policy Management snap-in will not prevent you from creating these rules, but they will not work properly. All rules that apply to a single remote host must use a single authentication method.

Another potential cause of authentication problems is incorrectly obtained certificates. Incorrectly obtained certificates can result in a condition in which the certificate exists and is chosen to be used for IKE authentication, but fails to work because the private key corresponding to the certificate’s public key is not present on the local computer. To verify that a certificate you plan to use for IPSec does indeed have a private key:

1. Open a blank MMC console, and then add the Certificates snap-in. When prompted, select Computer Account.

2. Expand Certificates (Local Computer), and then expand Personal.

3. Click the Certificates node.

4. In the right pane, double-click the certificate you want to check. On the General tab, you should see the text “You have a private key that corresponds to this certificate.” If you don’t see this message, the system won’t use this certificate successfully for IPSec.

Depending upon how the certificate was requested and populated into the host’s local certificate store, this private key value might not exist, or it might not be available to be used during the IKE negotiation. If the certificate in the personal folder does not have a corresponding private key, certificate enrollment has failed.

If a certificate was obtained from Certificate Services with the option set for Strong Private Key Protection, the user must enter a PIN number to access the private key each time that the private key is used to sign data in the IKE negotiation. Because the IKE negotiation is being done in the background by a system service, there is no window available to the service to prompt the user. So certificates obtained with this option will not work for IKE authentication.

See Also 

For information about troubleshooting certificate revocation lists, refer to Chapter 8.

Troubleshooting Firewalls, Routers, and Packet Filtering

Packet filtering at firewalls is a common source of IPSec problems because IPSec cannot be permitted or blocked by applying the techniques used for most applications. First, your firewall must allow two-way traffic with a UDP destination port of 500. If the firewall is also a NAT server and you will be using Network Address Translation Traversal (NAT-T), you must also allow UDP traffic with a destination port of 4500. Second, the firewall must allow traffic with an IP protocol ID of 50, which is used by ESP. If you are using AH instead of ESP, you must allow IP protocol 51.

Important 

The IP protocol ID 50 is not the same as UDP or TCP port 50.

You cannot use conventional packet filters to selectively block traffic that is tunneled or transported within IPSec. Most packet filters block traffic based on TCP or UDP port number, but those numbers are usually encrypted by ESP. Only IPSec filters implemented on the two IPSec peers can perform port-based packet filtering. You can still filter traffic based on source or destination IP address, however.

Network Address Translation Problems

Network Address Translation (NAT) is a common technique for connecting a privately numbered internal network to a public network such as the Internet. As Chapter 8 discussed, earlier implementations of IPSec were not compatible with NAT. This makes sense, because NAT’s purpose is to modify the source or destination IP address in a packet without the client or server being aware, and part of IPSec’s purpose is to discard packets that have been modified in transit.

Note 

You won’t have problems establishing IPSec connections between two clients on a single private network. It’s only when the NAT server performs address translation on IPSec- protected communications that problems can occur.

IPSec NAT Traversal (NAT-T) allows IPSec traffic to pass through compatible NAT servers. However, both the IPSec hosts and the NAT server must support NAT-T, and the NAT server must be configured to allow traffic on UDP port 4500. Windows Server 2003, Windows XP Professional, and Windows 2000 support NAT-T as IPSec clients. Microsoft Internet Security And Acceleration (ISA) Server and Windows Server 2003 support NAT-T as a firewall.

If you are having problems establishing an IPSec connection across a NAT server, first verify that both IPSec clients and the NAT servers support NAT-T. Clients using Windows 2000 or Windows XP must have update 818043 installed. (This update is available from http://support.microsoft.com/default.aspx?kbid=818043.) If you are using the built-in NAT capabilities of a computer running Windows XP or Windows 2000, or versions of Windows released prior to Windows 2000, you cannot use IPSec across NAT.

Warning 

Ensure that you are using ESP. You cannot use AH across a NAT server, because AH is not compatible with NAT-T.

Windows Server 2003 supports IPSec NAT-T as described by version 2 of the IPSec NAT-T Internet drafts, but some applications might not work when their traffic is first protected with IPSec and then passed through a network address translator. Test how your planned implementation of IPSec interacts with network address translators in a lab before deploying IPSec in your environment.

If possible, verify that IPSec works properly between two computers on the same side of the NAT server. If problems persist, the problem might not be related to NAT-T. Verify that the NAT server is not filtering packets required by IPSec, as described in the section “Troubleshooting Firewalls, Routers, and Packet Filtering” in this lesson.

Interoperability Problems

Most problems, particularly problems related to interoperating with non-Windows operating systems, can be resolved by creating the simplest policy possible instead of using the default policies. For example, when you create a new policy, use the IP Security Policy Wizard to clear the default response rule. Then, when you add new rules, choose the default This Rule Does Not Specify A Tunnel option, rather than creating a tunnel.

Next, edit the policy to remove all but a single key exchange security method. To do this, open the policy’s property sheet, and then click the General tab. Then click the Settings button, and then click the Methods button. Use the Key Exchange Security Methods dialog box to remove all but the one option that the destination will accept. For example, use the RFC 2049–required IKE security algorithms of DES, SHA1, and the Low (1) Diffie Hellman group, as shown in Figure 9.13.

Figure 9.13: Configuring a policy to use the most common IKE security algorithms

Next, create a filter list with one mirrored filter that specifies the source My IP Address and a destination with the IP address that you are trying to establish IPSec communications with, and then configure the filter list to match only ICMP traffic. Create a new filter action to negotiate security by using only one security method. For example, use the RFC 2049–required set of parameters, such as format ESP using Data Encryption Standard (DES) with SHA1, with no lifetimes specified. Make sure both the Use Session Key Perfect Forward Secrecy and Allow Unsecured Communication With Non-IPSec- Aware Computers check boxes are cleared. Use an authentication method of preshared keys for the rule, and make sure there are no white spaces in the string of characters. The destination must use exactly the same preshared key. Then configure the destination with the same configuration, but reverse the source and destination IP addresses.

Tip 

If you want to see the traffic in the IPSec-formatted packets with Network Monitor, use Medium Security (AH format).

When you are able to establish an IPSec connection using the lowest possible settings, begin to change the settings one at a time to match your security standards. Test IPSec after each change. When the IPSec connection fails, you will have identified the source of the problem. Then you can choose to either leave that setting at the compatible— but potentially less secure—setting, or you can work with the software vendors to improve interoperability between the IPSec implementations.

Lesson Review

The following question is intended to reinforce key information presented in this lesson. If you are unable to answer a question, review the lesson materials and try the question again. You can find answers to the questions in the “Questions and Answers” section at the end of this chapter.

1. IPSec peers ComputerA and ComputerB, as shown in Figure 9.14, are unable to establish a connection. What is the cause of the problem?

   
   Figure 9.14: Problematic IPSec architecture

Lesson Summary

• Many IPSec problems can be resolved by simply restarting the IPSec services on the IPSec peers, or by restarting the computers.

• You can isolate IPSec authentication problems by temporarily changing both peers to the Preshared Key authentication method. Authentication problems when using Kerberos are often caused by one of the IPSec peers not being able to reach a domain controller. IPSec authentication using certificates will fail if a private key is not associated with the certificate.

• Firewalls, routers, and other packet-filtering devices must allow traffic on UDP port 4500 and traffic with IP protocol ID 50 for IPSec communications to succeed.

• Both IPSec clients and any firewalls and proxy servers that implement NAT must be compatible with NAT-T for IPSec communications between public and private networks to be successful.

---