You can divide the problem areas of IPsec VPN on the VPN 3000 Series Concentrator into the following categories based on their implementations:
Sections that follow discuss in detail the configuration and troubleshooting aspects of these topics.
LAN-to-LAN Tunnel Issues
As mentioned earlier, the Cisco VPN 3000 Series Concentrator can build LAN-to-LAN tunnels with a wide variety of Cisco and non-Cisco devices. It is beyond the scope of this text to discuss the configuration and troubleshooting details of all the devices with which the VPN Concentrator interoperates. Regardless of the types of devices that you have on the other side of the tunnel, configuration for LAN-to-LAN on the VPN Concentrator is the same. This section discusses how to configure a LAN-to-LAN between two Concentrators, which will lead to the discussion of some of the issues and how to resolve them:
The following sections discuss the configuration and troubleshooting aspects of configuration and troubleshooting of LAN-to-LAN tunnel between two VPN 3000 series Concentrators.
LAN-to-LAN tunnel configuration involves configuring two tunnel end points (two concentrators for example) as peers of each other. This section explains the configuration required on one of the VPN Concentrators for a LAN-to-LAN tunnel. The same configuration with minor changes can be implemented on the other Concentrator. Work through the following steps to configure one of the VPN 3000 Concentrator for LAN-to-LAN tunnel:
Go to Configuration > Interfaces and configure the IP addresses on both public and private interfaces. Also, be sure that a filter is applied for each interface. It is recommended to select the default filter for the respective interface (for example, apply the Private Default) filter on the Private interface).
Browse to page Configuration > System > IP Routing > Default Gateways to set up a Default Gateway for the Concentrator and a Tunnel Default Gateway for the decrypted tunneled traffic.
To define a more specific route for the private network on both sides of the tunnel, go under Configuration > System > IP Routing > Static Routes to define the routes.
To define Network Lists for the interesting traffic that will go through the tunnel, go to Configuration > Policy Management > Traffic Management > Network Lists, and either Add or Modify the Network List for both the local side and the remote side. Remember that these network lists should be the mirror images of each other for both the Concentrators.
If a certificate is used for IKE authentication, you need to install a certificate by going to Administration > Certificate Management (see the section in this chapter entitled "Digital Certificate Issues" for more details). If a pre-shared key is used, ignore this step.
If you want to configure an IKE proposal (and do not want to use the default ones), you can go to Configuration > Tunneling and Security > IPsec > IKE Proposals, and Add or Modify the IKE proposal. Be sure that the IKE proposal is under Active Proposals.
To define a custom IPsec Security Association (IPSEC SAs), go to Configuration > Policy Management > Traffic Management > Security Associations. Then either Add or Modify the IPsec SAs.
Go to Configuration > Tunneling and Security > IPsec > LAN-to-LAN to add or modify a LAN-to-LAN connection. Use the parameters defined in Steps 4-7 to complete the LAN-to-LAN tunnel connection configuration.
For more details on configuration of VPN 3000 Concentrator refer to the following link:
The problem areas of a LAN-to-LAN tunnel can be classified as follows:
The sections that follow present detailed discussions of these topics.
Tunnel Not Established
If end users complain that they are unable to send any traffic across the tunnel, you should first check whether the LAN-to-LAN tunnel is established. You can verify this by going into Administration > Administer Sessions page and looking under LAN-to-LAN Sessions. If you do not see any sessions, perform a ping on the other Concentrator's private interface IP address and see if you can ping and if the tunnel shows that it is established. If you still have issues, you need to troubleshoot the issue further with the steps outlined in this section. But, before you look into the possibilities of why the tunnel is not being built up, you need to understand the Event Log messages for a good LAN-to-LAN tunnel establishment process. Table 8-3 shows the highlights of the sequence of events involved in LAN-to-LAN tunnel establishment on both initiator and responder sides for Main Mode negotiation.
Table 8-3. Event Log of Both Initiator and Responder for a Successful LAN-to-LAN Tunnel Phase I (Main Mode)
9 04/10/2005 22:48:24.280 SEV=8 IKEDBG/0 RPT=4 172.16.172.50 SENDING Message (msgid=0) with payloads : ! This is where the SA payload is sent ! by the Initiator HDR + SA (1) + NONE (0) ... total length : 84
1 04/10/2005 10:50:03.820 SEV=8 IKEDBG/0 RPT=1 172.16.172.119 SENDING Message (msgid=0) with payloads : ! This is where the SA payload is sent ! by the responder HDR + SA (1) + NONE (0) ... total length : 84
27 04/10/2005 22:48:24.490 SEV=8 IKEDBG/0 RPT=10 172.16.172.50 SENDING Message (msgid=0) with payloads : ! This is the 3rd packet and sent by ! the initiator with DH key and the ! Nounce HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) ... total length : 256
10 04/10/2005 10:50:04.020 SEV=8 IKEDBG/0 RPT=7 172.16.172.119 SENDING Message (msgid=0) with payloads : ! This is the 4th packet sent by the ! Responder with DH key and the Nounce HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) ... total length : 256
54 04/10/2005 22:48:24.700 SEV=8 IKEDBG/0 RPT=18 172.16.172.50 SENDING Message (msgid=0) with payloads : ! This is the ID payload and the 5th ! packet sent by the Initiator HDR + ID (5) + HASH (8) + IOS KEEPALIVE (14) + VENDOR (13) + NONE (0) ... total length : 92
41 04/10/2005 10:50:04.240 SEV=8 IKEDBG/0 RPT=14 172.16.172.119 SENDING Message (msgid=0) with payloads : ! This is the ID payload and the 6th ! packet sent by the Responder HDR + ID (5) + HASH (8) + IOS KEEPALIVE (14) + VENDOR (13) + NONE (0) ... total length : 92
Table 8-4 shows the sequence of events for a LAN-to-LAN tunnel on both initiator and responder sides for Quick Mode negotiation (only highlights are shown).
Table 8-4. Event Log of Both Initiator and Responder for a Successful LAN-to-LAN Tunnel Phase II (Quick Mode)
124 04/10/2005 22:48:24.800 SEV=8 IKEDBG/0 RPT=29 172.16.172.50 SENDING Message (msgid=70633ee2) with payloads : ! This is the IPSEC Phase II proposal ! sent by the Initiator, and the first ! packet of Quick Mode negotiation. HDR + HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NOTIFY (11) + NONE (0) ... total length : 180
95 04/10/2005 10:50:04.450 SEV=8 IKEDBG/0 RPT=34 172.16.172.119 SENDING Message (msgid=70633ee2) with payloads : ! This is the 2nd packet by the ! Responder in response to the first ! packet by the Initiator HDR + HASH (8) + SA (1) + NONCE (10) + ID (5) + ID (5) + NONE (0) ... total length : 152
142 04/10/2005 22:48:24.820 SEV=8 IKEDBG/0 RPT=36 172.16.172.50 SENDING Message (msgid=70633ee2) with payloads : ! This is the 3rd and the final packet ! by the Initiator before the Quick ! Mode is established. HDR + HASH (8) + NONE (0) ... total length : 72
136 04/10/2005 10:50:04.470 SEV=8 IKEDBG/0 RPT=35 172.16.172.119 RECEIVED Message (msgid=70633ee2) with payloads : HDR + HASH (8) + NONE (0) ... total length : 48
Now that you have become familiar with the debug messages of a normal LAN-to-LAN tunnel build-up process on VPN 3000 series Concentrator, work through the following steps to troubleshoot the issue with the tunnel establishment process:
Check the connectivity between the Concentrators.
On the VPN Concentrator, go to Administration > Ping, and ping to the public interface IP address of the other Concentrator, and see if you receive a response. If you do not receive a response, work through the following steps to correct the problem:
- (a). Check to see if you have a default gateway set up for the Concentrator by going to Configuration > System > IP Routing > Default Gateways.
- (b). Ping to the Gateway IP address, and see if you get any response. If the Gateway responds, the problem may be with a device in transit dropping the packet reaching to the other Concentrator.
- (c). Perform Traceroute to the IP address of the other Concentrator by going into the Administration > Traceroute page on your VPN Concentrator and find out where the packets are being dropped.
- (d). Your ISP may be blocking ICMP packets, and if so, you will not be able to ping. Under this circumstance, you can send a UDP probe instead of ICMP for the traceroute. This process is discussed in the steps that follow.
- (e). If the ping and traceroute both succeed or both fail due to ISP blocking of the ICMP protocol, perform Traceroute using UDP probes on port 500 instead of ICMP pings by going into Administration > Traceroute page. This will ensure that the ISKMP is allowed across the path between VPN 3000 Concentrators. If traceroute fails, talk to your ISP to correct the problem.
Check the configuration on both sides of the tunnel.
Once you verify the IP connectivity between the Concentrators, verify the configuration of both Concentrators to see if they match each other. Everything should match except the network lists, which should be mirror images of each other.
Verify that the Network Lists are configured correctly.
The Network List defines the interesting traffic of the LAN-to-LAN tunnel. Therefore, if the source and destination of a packet do not match the network list, an IPsec tunnel will not be initiated. Hence, you must ensure that Network Lists are defined correctly on both Concentrators. Pay close attention when you pass NATed traffic across the tunnel. When you configure NAT on the Concentrator to translate the private network, be sure to define the NATed IP rather than the actual private IP of the network in the Network Lists. For instance, if you have 10.1.0.0 NATed to 18.104.22.168, your Network List should include 22.214.171.124 instead of 10.1.0.0 as interesting traffic. It is an absolute requirement for Network Lists to be mirror images of each other on both Concentrators, so verify this configuration.
Analyze the IKE packets on both Concentrators.
Once you verify the configuration, you need to analyze the IKE messages on both sides to find out the cause of failure for tunnel establishment. See the section in this chapter entitled "Diagnostic Commands and Tools" for details and follow the procedure explained under the subsection entitled "Debug Tool" to enable severity level 1-9 for IKE, IKEDBG, IPsec, IPsecDBG , AUTH, and AUTHDBG event classes. IKE and IKEDBG show the phase I information, IPsec and IPsecDBG show phase II, and AUTH/AUTHDBG shows the authentication-related messages. You should see the IKE- and IPsec-related messages on both of the Concentrators.
Download the "VPN3000 Concentrator Event Information" file from the following location and look for the message from the index. As of writing this book, the latest file name is "vpn3000events-4.7.Rel.zip".
Check the Filter on the public interface to allow ISKMP (UDP/500) and ESP (IP/50).
If you don't see the IKE packets, and if you have a custom filter or a misconfigured public filter applied on the public interface, you need to check the filter to ensure that the filter is allowing *IPSEC-ESP In (forward/in)* rule. If you do not have this rule under "Current Rules in Filter", you need to insert it there. Otherwise, the tunnel will not be established.
Check Filter Configuration on the Public Interface.
If the firewall has the necessary ports open, check to see if the default public filter is applied on the public interface. If the filter is not applied, apply the filter, and if the filter is not the default public filter (but is a custom filter), be sure to allow UDP/500 for ISAKMP, and ESP packets.
Tunnel is Established but Unable to Pass Traffic
If, over a period of time, the tunnel in the Administration > Administer Sessions page shows Established but the counters for Bytes Tx or Bytes Rx shows zero or the same number, then work through the following steps to troubleshoot the issue:
Check the filter for the tunneled traffic.
If you have a filter for the tunneled traffic, choose None, and see if the traffic is flowing across the tunnel properly. To select None for the tunneled traffic filter, go to Configuration > Tunneling and Security > IPsec > LAN-to-LAN, and select your LAN-to-LAN Connection and click on Modify. On the next Window, select None for the Filter. If the traffic flows across the tunnel without problems, verify the Filter by going into Configuration > Policy Management > Traffic Management > Assign Rules to Filter for your specific filter rules. Check the rules being applied by going into Configuration > Policy Management > Traffic Management > Rules > Modify, and be sure the desired traffic is allowed by the rules.
Check the routing issue.
If you have a routing issue, you might run into a problem with one side sending packets across the tunnel but the other side not responding. This can be verified by going to the Administration > Administer Sessions page. The Bytes Tx is for Transmit and Bytes Rx is for Receive. This problem might be caused by an overlap in the network. Take an example to illustrate this problem. Assume that you have a LAN-to-LAN tunnel between the central and remote site. Additionally, assume that your LAN segment for the remote site is 10.1.1.0/24, which overlaps with the LAN segment of the central site, which is 10.1.0.0/16. In this circumstance, if the Remote VPN Concentrator is the initiator of the tunnel, the tunnel will be built up, but the head end will be unable to route packets to the remote site with the default gateway. This is because you will have a more specific route for network 10.1.0.0/16 pointing to the 10.1.0.1 as the next hop, which is local to its network. So, to overcome this problem, in addition to your default gateway, you must have a more specific route for the 10.1.1.0/24 network pointing to the next hop, which is the default gateway for this Central side Concentrator.
Check the Network Address Translation (NAT) issue.
When working with the IPsec tunnel, you can encounter two NAT issuesone when NAT is on the VPN Concentrator; and the other when NAT/PAT is between the Concentrators:
- - NAT On the VPN Concentrator Having NAT on the VPN Concentrator is required if you have the same network on the LAN sides of both Concentrators. For instance, if you have 10.1.0.0/24 network on both sides of the tunnel, you need to translate one side of the tunnel to some other network address, so that addresses are unique. It is recommended to perform the translation on the head end (central site). Also note that for outbound traffic, NAT is applied on the source address, and for inbound traffic NAT is applied on the destination address. The destination NAT (d-NAT) is not supported on the Concentrator as of the writing of this book. For more details on how to apply NAT for LAN-to-LAN tunnels, refer to the following link: http://www.cisco.com/univercd/cc/td/doc/product/vpn/vpn3000/4_7/config/polmgt.htm#wp1640172
You must be running VPN Concentrator version 3.6 to configure NAT for a LAN-to-LAN setup. More details on configuration can be found in the following location:
- - NAT/PAT between the Concentrators If you have NAT/PAT configured between Concentrators, you must configure NAT-T (NAT-Transparency) on the VPN Concentrators so that the tunnel gets built up on UDP/4500. When the NAT-T is configured, both VPN Concentrators detect the presence of a NAT device, and tunnel negotiation takes place on UDP/4500. As the initial negotiation takes place using UDP/500, you need to open up UDP/500, UDP/ 4500, and ESP for successful tunnel establishment and for passing data traffic across the tunnel. For more details on this subject, go to the following links:
For NAT-T configuration details between a VPN client and Concentrator (the same method can be used for VPN Concentrator LAN-to-LAN tunnel), refer to the following link:
Interpretability issues with other vendors
Troubleshooting interoperability issues that occur between Cisco devices and devices from other vendors can be tricky sometimes, but if the devices are configured correctly, many problems can be avoided without even running a debug. All the techniques discussed in the preceding section are applicable for troubleshooting VPN 3000 concentrators with other vendor devices. Following are two suggestions that will help in interoperating a VPN 3000 appliance with other vendors' devices:
Start with configuring the two ends side-by-side with exact Matching policies (both phase I and Phase II). Table 8-5 lists the minimum required parameters for Phases I and II.
Table 8-5. Required Parameters for Phase I and II
Phase I Parameters
Phase II Parameters
IKE authentication method
IPsec mode (tunnel or transport)
ISAKMP SA lifetime
IPsec SA Lifetime
Matching pre-shared secret
Interesting traffic definition
Turn off vendor-specific features such as Mode-config, x-Auth, IKE keepalive, Dead Peer Detection (DPD) and so on, and then build up the tunnel. Once the tunnel build-up is confirmed, start adding extra unique features that are supported by the concentrator.
Remote Access VPN Connection
As mentioned before in the "Diagnostic Commands and Tools" section, troubleshooting Remote Access VPN involves analyzing the logs on both the VPN client and the VPN Concentrator. This section looks at the configuration and troubleshooting details of Remote Access VPN implementation.
You need to configure both the VPN Concentrator and VPN Client for a successful Remote Access VPN connection. The next two sections explain this configuration in detail.
VPN Concentrator Configuration
Work through the following steps to configure a Remote Access VPN connection on the VPN Concentrator:
Log in to the VPN Concentrator GUI, and verify that there are IP addresses assigned to the private (inside) and public (outside) interfaces. Also verify that there is a default gateway assigned so that the Concentrator can forward to the default gateway the packets for the destinations that it does not know about. In addition, be sure the private and public filters are applied to their respective interfaces.
To assign an available range of IP addresses, go to Configuration > System > Address Management > Pools > Add. Specify a range of IP addresses that do not conflict with any other devices on the inside network.
To instruct the VPN Concentrator to use the pool, go to Configuration > System > Address Management > Assignment, and check the Use Address Pools box.
To enable split tunneling, define a network list under Configuration > Policy Management > Traffic Management > Network Lists. This list should contain the networks on which you want traffic to be encrypted. These are usually the internal subnets of the VPN 3000 Concentrator.
To globally define the Domain Name System (DNS), NetBIOS Name Server (NBNS), Authentication, Authorization, and Accounting server (AAA), and so on, go to Configuration > System > Servers. Refer to Chapter 12, "Troubleshooting AAA on VPN 3000 Series Concentrator," for AAA implementation on VPN 3000 Concentrator.
Define a Group name and password for the Remote Access VPN connection by going to Configuration > User Management > Groups > Add. Under the Identity tab, be sure to select Internal for Group Authentication.
Under General Tab, be sure that IPsec is checked for Tunneling Protocol. You may also define the DNS and WINS Server per group or define Inherit from the Base Group.
To use the globally defined DNS or WINS Server, click on the IPsec tab, and set the authentication to Internal locally for user authentication on the Concentrator.
Next select the Client Config tab, and scroll down to Split Tunneling Policy. Ensure that the following option is selected: Only tunnel networks in this list. Next, go to the Split Tunneling Network List and select the previously defined Network List.
Go to Configuration > User Management > Users > Add, and add a user to the previously defined group.
VPN Client Configuration
Work through the following steps to configure the VPN client profile:
Open Start > Programs > Cisco Systems VPN Client > VPN Client.
Click on New to create a new VPN Connection Entry.
Define the Connection Entry and Host IP address for the VPN Concentrator.
Under the Authentication tab, select Group Authentication and provide the Group Name and Group Password. If a certificate is used instead of Group Authentication, choose the Certificate Authentication Radio button. Refer to the section of this chapter entitled "Digital Certificate Issues" for details on Certificate Authentication on the VPN Client.
Under Transport tab, check the Enable Transparent Tunneling check box. Choose IPsec over UDP or IPsec over TCP depending on your configuration on the VPN Concentrator and your NAT/PAT and Firewall setup. If local LAN access is needed so that you can still access the local LAN when you have the VPN tunnel established, check the Allow Local LAN Access check box.
If you want to define Alternate VPN Concentrator's IP, click on the Backup Servers tab.
Before looking at some possible reasons of tunnel failure, it is worth examining the debug of a successful Remote Access VPN tunnel build-up process. Remote Access VPN Client uses Aggressive Mode negotiation (three packets exchange instead of six packets for Main Mode in the case of LAN-to-LAN tunnels) to build up the tunnel with the VPN Concentrator. Example 8-1 shows the default event log on the VPN Concentrator for a successful phase I Aggressive Mode negotiation.
Example 8-1. Default Event Log on VPN 3000 Concentrator for the Aggressive Mode Negotiation
! Shows the VPN Client IP Address, mode supported, and Fragmentation Capabilities 1 06/20/2005 10:39:28.760 SEV=5 IKEDBG/64 RPT=7 172.16.172.119 IKE Peer included IKE fragmentation capability flags: Main Mode: False Aggressive Mode: True ! Shows the Group and User Authentication Status 3 06/20/2005 10:39:33.670 SEV=4 IKE/52 RPT=3 172.16.172.119 Group [mygroup] User [myuser] User (myuser) authenticated. ! Provides the VPN Client OS and Client Version information 4 06/20/2005 10:39:33.700 SEV=5 IKE/184 RPT=3 172.16.172.119 Group [mygroup] User [myuser] Client Type: WinNT Client Application Version: 4.0.5 (B) ! Shows the connection Type 6 06/20/2005 10:39:34.560 SEV=4 AUTH/22 RPT=6 172.16.172.119 User [myuser] Group [mygroup] connected, Session Type: IPsec ! Shows the Aggressive Mode Status Completed Successfully 7 06/20/2005 10:39:34.560 SEV=4 IKE/119 RPT=6 172.16.172.119 Group [mygroup] User [myuser] PHASE 1 COMPLETED
Once phase 1 is established, phase 2 negotiation starts. A successful phase 2 (Quick Mode) is shown in example 8-2.
Example 8-2. Default Event Messages For Working Client on VPN 3000 Concentrator with Quick Mode
! Shows the IP address assigned by the VPN Concentrator to the VPN Client 8 06/20/2005 10:39:34.570 SEV=5 IKE/25 RPT=3 172.16.172.119 Group [mygroup] User [myuser] Received remote Proxy Host data in ID Payload: Address 10.1.1.40, Protocol 0, Port 0 ! Local and Remote Proxies are the networks that will be encrypted by the IPSec tunnel. ! All Zeros Indicate, all traffic will go through the tunnel, hence no Split Tunneling 11 06/20/2005 10:39:34.570 SEV=5 IKE/34 RPT=4 172.16.172.119 Group [mygroup] User [myuser] Received local IP Proxy Subnet data in ID Payload: Address 0.0.0.0, CAsk 0.0.0.0, Protocol 0, Port 0 ! Shows the locally Matched IPsec SA 14 06/20/2005 10:39:34.570 SEV=5 IKE/66 RPT=4 172.16.172.119 Group [mygroup] User [myuser] IKE Remote Peer configured for SA: ESP-3DES-MD5 ! Shows the negotiated Phase 2 Re-Key Interval 15 06/20/2005 10:39:34.570 SEV=5 IKE/75 RPT=3 172.16.172.119 Group [mygroup] User [myuser] Overriding Initiator's IPsec rekeying duration from 2147483 to 28800 seconds ! Shows Inbound and Outbound SPIs 17 06/20/2005 10:39:34.580 SEV=4 IKE/49 RPT=6 172.16.172.119 Group [mygroup] User [myuser] Security negotiation complete for User (myuser) Responder, Inbound SPI = 0x1999168f, Outbound SPI = 0xa3bd8da1 ! Shows User and GroupQuick Mode Completed Successfully 20 06/20/2005 10:39:34.580 SEV=4 IKE/120 RPT=6 172.16.172.119 Group [mygroup] User [myuser] PHASE 2 COMPLETED (msgid=d2d8d26a) ! Finally shows that Network Access Control (NAC) is disabled 21 06/20/2005 10:39:34.580 SEV=4 NAC/27 RPT=3 NAC is disabled for peer - PUB_IP:172.16.172.119, PRV_IP:10.1.1.40
To analyze the Remote Access VPN establishment efficiently, and quickly, you must analyze the log on both the VPN 3000 Concentrator and the VPN Client. Example 8-3 shows the phase 1 establishment on the VPN Client using Aggressive mode.
Example 8-3. Phase 1 Establishment for Aggressive Mode Negotiation on VPN Client
! The following lines indicate that the VPN Client Environment is fine. 453 12:16:26.937 06/20/05 Sev=Info/4 CM/0x63100002 Begin connection process ! Following lines indicates that Microsoft IPsec server is stopped successfully 454 12:16:26.968 06/20/05 Sev=Info/4 CVPND/0xE3400001 Microsoft IPsec Policy Agent service stopped successfully ! Showing the Interface Used for the Connection 455 12:16:26.968 06/20/05 Sev=Info/4 CM/0x63100004 Establish secure connection using Ethernet . . . ! Shows the VPN Concentrator IP address the VPN Client is trying to establish VPN ! connection 457 12:16:27.968 06/20/05 Sev=Info/6 IKE/0x6300003B Attempting to establish a connection with 172.16.172.119. ! This is the 1st packet in Aggressive Mode sent by the VPN Client 458 12:16:27.968 06/20/05 Sev=Info/4 IKE/0x63000013 SENDING >>> ISAKMP OAK AG (SA, KE, NON, ID, VID(Xauth), VID(dpd), VID(Nat-T), VID(Frag), VID(Unity)) to 172.16.172.119 ! Following line indicates that IPSec Driver has successfully started 459 12:16:27.968 06/20/05 Sev=Info/4 IPSEC/0x63700008 IPsec driver successfully started ! Following is the 2nd message sent by the VPN VPN Concentrator, which is received ! by the VPN Client 463 12:16:28.125 06/20/05 Sev=Info/4 IKE/0x63000014 RECEIVING <<< ISAKMP OAK AG (SA, KE, NON, ID, HASH, VID(Unity), VID(Xauth), VID(dpd), VID(Frag), VID(?), VID(?)) from 172.16.172.119 ! The following message shows that VPN Concentrator is Cisco-Unity compliant 464 12:16:28.125 06/20/05 Sev=Info/5 IKE/0x63000001 Peer is a Cisco-Unity compliant peer ! Shows that the Peer Supports Extended Authentication 465 12:16:28.125 06/20/05 Sev=Info/5 IKE/0x63000001 Peer supports XAUTH ! Shows that the Peer Supports Dead Peer Detection (DPD) 466 12:16:28.125 06/20/05 Sev=Info/5 IKE/0x63000001 Peer supports DPD ! IKE Packet Fragmentation is supported. 467 12:16:28.125 06/20/05 Sev=Info/5 IKE/0x63000001 Peer supports IKE fragmentation payloads ! The Peer Supports Delete with Reason Messages 468 12:16:28.125 06/20/05 Sev=Info/5 IKE/0x63000001 Peer supports DWR Code and DWR Text ! The 3rd message which is the final message in Aggressive mode is sent by the VPNClient. 470 12:16:28.125 06/20/05 Sev=Info/4 IKE/0x63000013 SENDING >>> ISAKMP OAK AG *(HASH, NOTIFY:STATUS_INITIAL_CONTACT, VID(?), VID(Unity)) to 172.16.172.119 ! UDP Port Number Used for IKEPort 500 (Not NAT-T) as shown by the local and remote ! port which is 0x01F4 in Hex. 471 12:16:28.125 06/20/05 Sev=Info/4 IKE/0x63000083 IKE Port in use - Local Port = 0x01F4, Remote Port = 0x01F4 ! The following line shows the IKE Aggressive Mode Complete 472 12:16:28.125 06/20/05 Sev=Info/4 CM/0x6310000E Established Phase 1 SA. 1 Crypto Active IKE SA, 0 User Authenticated IKE SA in the system
In a Remote Access VPN connection, right after the phase 1 establishment, X-auth and mode configuration take place as shown in Example 8-4. This happens when the IP address assignment, DNS server IP address, and so on are assigned to the VPN Client.
Example 8-4. X-Auth and Mode Configuration Messages After Phase 1 Establishment
! Shows VPN client received Username/Password Prompt from VPN Concentrator 474 12:16:28.125 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 172.16.172.119 ! Username/Password prompt is presented to the user here 475 12:16:28.125 06/20/05 Sev=Info/4CM/0x63100015 Launch xAuth application ! Users entered username/password is sent to the VPN Concentrator 477 12:16:30.687 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 172.16.172.119 ! Received Authentication Acknowledgement from the VPN Concentrator 479 12:16:30.984 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 172.16.172.119 ! VPN Client is sending Authentication Acknowledgement back to the VPN Concentrator 480 12:16:30.984 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 172.16.172.119 ! Show user authentication is successful 481 12:16:30.984 06/20/05 Sev=Info/4CM/0x6310000E Established Phase 1 SA. 1 Crypto Active IKE SA, 1 User Authenticated IKE SA in the system ! VPN Client is sending firewall status to the VPN Concentrator 483 12:16:31.000 06/20/05 Sev=Info/5IKE/0x6300005D Firewall Policy: Product=Cisco Systems Integrated Client, Capability= (Centralized Protection Policy). 484 12:16:31.000 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK TRANS *(HASH, ATTR) to 172.16.172.119 ! Receiving Mode Config Attributes from the VPN 3000 Concentrator 486 12:16:31.859 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK TRANS *(HASH, ATTR) from 172.16.172.119 ! Tunnel Assigned IP Address from the VPN Concentrator to the VPN Client 487 12:16:31.859 06/20/05 Sev=Info/5IKE/0x63000010 MODE_CFG_REPLY: Attribute = INTERNAL_IPV4_ADDRESS: , value = 10.1.1.40 ! Tunnel Assigned Subnet CAsk is shown below 488 12:16:31.859 06/20/05 Sev=Info/5IKE/0x63000010 MODE_CFG_REPLY: Attribute = INTERNAL_IPV4_NETCASK: , value = 255.255.255.0 ! Following lines show if the Password Allowed to Be Stored Locally. Value 0x00000000 ! means NO 489 12:16:31.859 06/20/05 Sev=Info/5IKE/0x6300000D MODE_CFG_REPLY: Attribute = MODECFG_UNITY_SAVEPWD: , value = 0x00000000 ! Number of Entries in the Split Tunnel List. In this log it shows 2. 490 12:16:31.859 06/20/05 Sev=Info/5IKE/0x6300000D MODE_CFG_REPLY: Attribute = MODECFG_UNITY_SPLIT_INCLUDE (# of split_nets), value = 0x00000002 ! Detail of all Networks in the Split List 491 12:16:31.859 06/20/05 Sev=Info/5IKE/0x6300000F SPLIT_NET #1 subnet = 10.1.1.0 CAsk = 255.255.255.0 protocol = 0 src port = 0 dest port=0 ! Perfect Forward Secrecy (Disabled) as the value is set to 0x00000000 493 12:16:31.859 06/20/05 Sev=Info/5IKE/0x6300000D MODE_CFG_REPLY: Attribute = MODECFG_UNITY_PFS: , value = 0x00000000 ! Peer version information is shown here 494 12:16:31.859 06/20/05 Sev=Info/5IKE/0x6300000E MODE_CFG_REPLY: Attribute = APPLICATION_VERSION, value = Cisco Systems, Inc./VPN 3000 Concentrator Version 4.7.Rel built by vmurphy on CAr 10 2005 12:35:39
After the successful X-Auth and mode-config of the VPN connection, phase 2 negotiation (Quick Mode) starts as shown in Example 8-5.
Example 8-5. Shows a Successful Quick Mode Negotiation (Phase II) on VPN Client
! Quick Mode Starts here. Sending Quick Mode Message 1 497 12:16:31.875 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK QM *(HASH, SA, NON, ID, ID) to 172.16.172.119 ! Received IKE Notify message from the VPN Concentrator 500 12:16:31.890 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK INFO *(HASH, NOTIFY:STATUS_RESP_LIFETIME) from 172.16.172.119 ! Received Quick Mode Message 2 from the VPN Concentrator 504 12:16:31.890 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK QM *(HASH, SA, NON, ID, ID, NOTIFY:STATUS_RESP_LIFETIME) from 172.16.172.119 ! Sending Quick Mode Message 3 as shown below 506 12:16:31.890 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK QM *(HASH) to 172.16.172.119 507 12:16:31.890 06/20/05 Sev=Info/5IKE/0x63000059 ! Loading SPIs here Loading IPsec SA (MsgID=D335FB24 OUTBOUND SPI = 0x448FCE67 INBOUND SPI = 0x26DCA8D9) ! Non-VPN Routing Table 510 12:16:31.937 06/20/05 Sev=Info/5CVPND/0x63400013 Destination NetCAsk Gateway Interface Metric 0.0.0.0 0.0.0.0 172.16.172.119 172.16.172.119 20 127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1 172.16.172.119 255.255.255.255 127.0.0.1 127.0.0.1 20 ! Virtual Adapter Enabled with the Tunnel Assigned Address 511 12:16:32.562 06/20/05 Sev=Info/4CM/0x63100034 The Virtual Adapter was enabled: IP=10.1.1.40/255.255.255.0 DNS=0.0.0.0,0.0.0.0 WINS=0.0.0.0,0.0.0.0 DoMain= Split DNS Names= ! Routing Table post VPN, Repeated for All Subnets in the Network List 512 12:16:32.562 06/20/05 Sev=Info/5CVPND/0x63400013 Destination NetCAsk Gateway Interface Metric 0.0.0.0 0.0.0.0 172.16.172.119 172.16.172.119 20 10.1.1.0 255.255.255.0 10.1.1.40 10.1.1.40 10 10.1.1.40 255.255.255.255 127.0.0.1 127.0.0.1 10 10.255.255.255 255.255.255.255 10.1.1.40 10.1.1.40 10 127.0.0.0 255.0.0.0 127.0.0.1 127.0.0.1 1 172.16.172.119 255.255.255.255 127.0.0.1 127.0.0.1 20 ! Virtual Adapter is Added 514 12:16:32.609 06/20/05 Sev=Info/6CM/0x63100036 The routing table was updated for the Virtual Adapter ! Locally Assigned and Tunnel Assigned Addresses 516 12:16:32.953 06/20/05 Sev=Info/4CM/0x63100038 Address watch added for 172.16.172.119. Current hostname: rail, Current address(es): 10.1.1.40, 172.16.172.119. ! Tunnel Assigned Address to the Virtual Adapter 522 12:16:33.015 06/20/05 Sev=Info/4IPSEC/0x6370002E Assigned VA private interface addr 10.1.1.40
After becoming comfortable with the sequence of events that takes place for a successful Remote Access VPN connection, examine the possible causes of tunnel failure as listed in the following sections:
VPN Client cannot connect
VPN Client can connect but Tunnel is not passing traffic
VPN Client can connect but users cannot access Internet
VPN Client can connect but users cannot access local LAN
Details on the above listed items follow in the following sections.
VPN Client Cannot Connect
Unlike LAN-to-LAN tunnel, with the Remote Access VPN, you can immediately determine if the tunnel is establishing or not. When the tunnel is successfully established, this message displays: "You are connected."
The Remote Access VPN tunnel establishment may fail for various reasons. Using a systematic approach is the best way to check various possibilities and correct them as you analyze the best approach to troubleshooting Remote Access VPN issues. Work through the following steps to correct the Remote Access VPN tunnel establishment failure:
Check the connectivity between the VPN Client and the Concentrator.
From the VPN client PC, ping to the public interface IP addresses of the VPN Concentrator. If you cannot ping, work through the following steps to correct the problem:
- (a). Be sure that the default gateway is defined on the VPN client host, and that the host can ping to the default gateway IP address.
- (b). Go to the VPN Concentrator GUI, and verify that you have a default gateway defined for the Concentrator. If none is defined, define one. Otherwise, go to Administration > Ping, and ping to the default gateway of the Concentrator.
- (c). If both the VPN Concentrator and VPN client can ping each other, then ensure that ISKMP packets are allowed by a firewall that is between them. This can be done by performing Traceroute using a UDP probe instead of the ICMP ping to the IP address of the other Concentrator. To perform this action, go to Administration > Traceroute page on your VPN Concentrator.
Be sure that IKE packets are being exchanged between the VPN Client and the Concentrator.
Once connectivity is verified with the previous step, check the event logs on both VPN client and Concentrator. See the "Diagnostic Commands and Tools" section for details on how to use the Event Log features on both VPN Client and the Concentrator. If the IKE packets are being exchanged, you should see messages similar to the one shown in examples 8-6 on the VPN Client.
Example 8-6. IKE Messages Shown on VPN Client
Example 8-7 shows the IKE packets seen on the VPN concentrator.
121 20:04:56.778 06/20/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK INFO (NOTIFY:INVALID_HASH_INFO) to 172.16.172.119 135 20:12:54.580 06/20/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK AG (SA, KE, NON, ID, HASH, VID, VID, VID, VID) from 172.16.172.119
Example 8-7. IKE Messages on VPN Concentrator
If you do not see the IKE packets on the VPN client, then the problem is on the VPN client. You may need to stop and restart the cvpnd service with net stop cvpnd and net start cvpnd, or you may need to reboot the VPN client PC. As a last resort you may end up re-installing the VPN client software. If you see the IKE packets on VPN client but do not see the IKE packets on the VPN 3000 Concentrator, go to the next step.
1 04/07/2005 20:04:16.640 SEV=8 IKEDBG/0 RPT=2955 192.168.1.100 RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + KE (4) + NONCE (10) + ID (5) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) ... total length : 561
Be sure the firewall between the VPN Client and Concentrator allows ISKMP (UDP/500) packets.
If you do not see the IKE packets on VPN 3000 Concentrator, check to see if you have a firewall between the VPN client and Concentrator. If you do, be sure that ISKMP (UDP/500) packets are allowed through the firewall. Otherwise, IKE packets will be dropped by the firewall. If you have a NAT device between the VPN client and Concentrator, and you have NAT-T configured, then you need to allow UDP/4500 for the NAT-T. If a firewall between blocks the UDP/500 packets, you will see the event log on VPN Client as shown in Example 8-8.
Example 8-8. VPN Client Log When the NAT-T Fails Due to UDP/4500 Packets Block
On the VPN Concentrator, you will not see any re-transmission. Instead, you will see the messages shown in Example 8-9.
! Received Aggressive Mode Message 2 595 20:47:46.335 06/21/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK AG (SA, KE, NON, ID, HASH, VID(Unity), VID(Xauth), VID(dpd), VID(Nat-T), NAT-D, NAT-D, VID(Frag), VID(?), VID(?)) from 172.16.172.119 ! Sending Aggressive Mode Message 3 to the VPN Concentrator. Here it shows NAT-T ! Negotiated UDP Port 4500 603 20:47:46.355 06/21/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK AG *(HASH, NOTIFY:STATUS_INITIAL_CONTACT, NAT-D, NAT-D, VID(?), VID(Unity)) to 172.16.172.119 ! The Client Receives the Retransmissions 608 20:47:54.327 06/21/05 Sev=Info/5IKE/0x6300002F Received ISAKMP packet: peer = 172.16.172.119 609 20:47:54.327 06/21/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK AG (Retransmission) from 172.16.172.119 ! The Client Retransmits AM MSG 2 610 20:47:54.327 06/21/05 Sev=Info/4IKE/0x63000021 Retransmitting last packet 611 20:47:54.327 06/21/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK AG *(Retransmission) to 172.16.172.119 ! The Client Receives the Unencrypted Delete Message 625 20:48:18.321 06/21/05 Sev=Warning/3IKE/0xA3000058 Received CAlformed message or negotiation no longer active (message id: 0xB7381790) ! The Client Sends It's Own Delete Message 636 20:49:18.007 06/21/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK INFO *(HASH, DWR) to 126.96.36.199
Example 8-9. VPN Concentrator Log When the NAT-T Fails Due to UDP/4500 Packets Block
If IPsec over TCP is configured, the default port you need to allow is TCP/10000. If another port is used, you need to allow that specific port. Additionally, you need to allow ESP (IP/50) to enable the tunneled traffic.
333 05/06/2005 09:55:03.860 SEV=7 IKEDBG/65 RPT=1 172.16.172.1190 Group [mygrou] ! FSM ErrorTime Out Waiting for AM MSG 3 is shown below IKE AM Responder FSM error history (struct &0x7ea8590) <state>, <event>: AM_DONE, EV_ERROR_CONT AM_DONE, EV_ERROR AM_WAIT_MSG3, EV_TIMEOUT AM_WAIT_MSG3, NullEvent ! Sending a Delete MSG After the Time Out. You will not see Retransmissions. The ! Concentrator Resends AM MSG 2 Three Times at 8 Second Intervals 338 05/06/2005 09:55:03.860 SEV=8 IKEDBG/81 RPT=7 172.16.172.1190 SENDING Message (msgid=d0257b9c) with payloads : HDR + HASH (8) + DELETE (12) total length : 76
Be sure that the filter applied on the public interface allows ISKMP (UDP/500) and ESP (IP/50) traffic.
If the firewall has the necessary ports open, check to see that the filter is applied on the public interface of the concentrator. By default, the public filter allows all the necessary ports for the IKE message. However, if the filter is not public or if you have customized the filter, be sure to have the IPSEC-ESP In (forward/in) rule under "Current Rules in Filter" on your filter.
If NAT-T is configured, you must assign the NAT-T In and NAT-T Out rules to that filter, applied and active on the public interface.
IKE Proposal Parameters mismatch between the VPN Client and VPN Concentrator.
In Aggressive Mode Message 1, the VPN client sends a list of supported proposals to the VPN Concentrator. On the concentrator, you need to have at least one of the proposals sent by the VPN client active. The concentrator will match based on order in the active proposal list. To verify the proposals on the VPN Concentrator, go to Configuration > Tunneling and Security > IPsec > IKE Proposals.
Check for Group Authentication Failure.
Upon receiving the IKE proposal, the VPN concentrator first finds the group name and authenticates the group. Example 8-10 shows a successful group authentication in VPN 3000 Concentrator.
Example 8-10. Successful Group Authentication on VPN 3000 Concentrator
Table 8-6 shows some of the most common group-authentication problems and how to resolve them.
15 04/07/2005 20:04:16.640 SEV=9 IKEDBG/23 RPT=42 192.168.1.100 Starting group lookup for peer 192.168.1.100 39 04/12/2005 01:54:03.230 SEV=6 AUTH/41 RPT=26 192.168.1.100 ! The following line shows the group authentication is successful. Authentication successful: handle = 17, server = Internal, group = mygroup 40 04/07/2005 20:12:14.500 SEV=7 IKEDBG/0 RPT=2984 192.168.1.100 Group [mygroup] Found Phase 1 Group (mygroup)
Table 8-6. Common Group Authentication Issues and Resolution On VPN Concentrators
Client Error Message
VPN Concentrator Error message
How to resolve
Group Name MisMatch
GI VPN start callback failed "CM_PEER_NOT_RESPONDING" (16h).
No Group found Matching mygroupo for Pre-shared key peer 192.168.1.100
Check group name. If missing configure it in VPN Concentrator, or if it exists, correct the group name in client configuration.
Group Password MisMatch
Hash verification failed... May be configured with invalid group password.
Group [mygroup] Received non-routine Notify message: Invalid hash info (23)
Correct the group password on the concentrator or specify it correctly on the VPN client.
Group Lock Configuration
GI VPN start callback failed "CM_PEER_NOT_RESPONDING" (16h).
User (U1) not member of group (test_grp), authentication failed.
If the Group Lock feature isenabled on the Group test_grp, then the User must be part of test_grp to connect.
Verify that User Authentication (X-Auth) is successful.
Once group authentication is successful, user authentication occurs if it is configured on the VPN Concentrator. If the user authentication fails at this stage, the VPN tunnel will not be built up. Note that user authentication can be performed either locally on the VPN Concentrator or using an external AAA server. If the authentication is configured with an AAA Server, refer to Chapter 12, "Troubleshooting AAA on VPN 3000 Series Concentrator." If authentication is performed locally on the VPN Concentrator, turn on XAuth Verbose level to high on the VPN Client and set the level of logging for AUTH, AUTH, and AUTHDBG to 1-9 following the procedure explained in the "Diagnostic Commands and Tools" section. The same section also explains how to interpret the event log message. Example 8-11 shows an example of a successful user authentication on the VPN 3000 Concentrators Event Log.
Example 8-11. A Successful User Authentication Event Log on VPN Concentrator
116 04/12/2005 02:08:52.970 SEV=6 AUTH/4 RPT=9 192.168.1.100 Authentication successful: handle = 19, server = Internal, user = vpn3k 165 04/12/2005 02:08:53.170 SEV=7 IKEDBG/14 RPT=20 192.168.1.100 Group [mygroup] User [vpn3k] Authentication configured for Internal 171 04/12/2005 02:08:53.170 SEV=4 IKE/52 RPT=9 192.168.1.100 Group [mygroup] User [vpn3k] User (vpn3k) authenticated.
If authentication fails, be sure the appropriate authentication server is set by going into Configuration > System > Servers > Authentication servers. To ensure that the specific group configuration for the authentication server does not override the server configuration setup under System, go into Configuration > User Management > Groups > Authentication Servers, and check to see if the VPN Concentrator is configured correctly to assign the address.
If the VPN client can connect to the VPN Concentrator but is unable to get the IP address, check the log on both client and Concentrator to find the cause of the problem. Example 8-12 presents the Event Log on the VPN Concentrator that shows it is unable to assign the IP address to the VPN client.
Example 8-12. Event Log on the VPN Concentrator Shows That it Is Unable to Assign an IP Address to the VPN Client
Typically, the address assignment problem occurs due to misconfiguration. But there also can be other reasons for the VPN Concentrator being unable to assign an IP address to the VPN Client. The list that follows outlines procedures to deal with the most common problems:
! The following line indicates that VPN Concentrator is unable to allocate an IP ! address Group [mygroup] User [U1] IKE received response of type [FAILED] to a request from the IP address utility . . . 204 04/11/2005 00:29:42.500 SEV=5 IKE/132 RPT=2 192.168.1.100 ! The following line reaffirms that the obtaining of IP address is indeed ! unsuccessful. Group [mygroup] User [U1] Cannot obtain an IP address for remote peer
Figure 8-7 shows how to create the IP address pool and apply it on a VPN 3000 Concentrator.
- - Be sure that the IP address Pool is configured To allocate an IP address from a local pool, be sure that the pool is configured under Configuration > System > Address Management > Pools. Be sure that you have a correct pool defined, and if you do not, define one. On the other hand, if you want to assign the address from an AAA server, define the pool on the AAA server.
- - Be sure Method of Assignment is selected Merely defining a pool is sufficient unless you check the correct box under Configuration > System > Address Management > Assignment page. This is one of the most common mistakes an engineer makes.
- - Be sure you are not reaching to max of address from address pool If you are having address assignment issues with a specific client intermittently, or after a certain number of host connection, chances are that your Concentrator is hitting the max on the address pool. Consider redefining the address pool to add additional addresses to the pool.
Figure 8-7. Configuration for Creating an IP Pool
VPN Client can Connect but Tunnel Is Not Passing Traffic
If the VPN Client is able to connect but unable to pass any traffic, work through the steps that follow to isolate and resolve the problem:
Check Routing for Issues on the VPN Client PC.
If the tunnel is up and you still cannot send any traffic across the tunnel (this can be verified by checking the number of encrypts/decrypt packets in both client and VPN concentrator session logs), then most likely the problem is with client routing. Traffic originating from the laptop will be sent to the NIC, not through the VPN tunnel, if these circumstances apply: first, you connected your laptop earlier to the corporate network that allocated an IP address from a 10.1.0.0 network; and second, when you connect from home, the VPN 3K assigns you an IP address from the 10.1.0.0 network. This problem is very prominent in Microsoft Windows 95/98/NT.
To solve the problem, release the Dynamic Host Configuration Protocol (DHCP) that learned the IP address that you learned while you were connected to the LAN. Also, be sure that the default gateway is set up correctly. For more details on this specific issue, refer to the following link:
Check for Routing on the VPN 3000 Concentrator.
If the Tunnel default gateway is defined, traffic will be sent out of the private interface of the Concentrator to the Tunnel Default Gateway (Router, PIX or any other device). The default gateway device should have a route for tunneled traffic pointing back to the VPN Concentrator private interface for the return traffic. The tunneled route can be defined based on the address pools, network extension mode, DHCP scope, and so on, based on your setup. You can use Reverse Route Injection (RRI) to redistribute tunneled routes into the non-tunneled network. If RRI is configured, you can use the Event Class IPDBG with Severity 1-8 to see routes dynamically added and removed from the Concentrator's Routing Table.
Check to see if you have a Port Address Translation (PAT) device in the middle.
Once you verify the routing issue, if you are still unable to pass traffic, check to see if you have a PAT device in the middle, which may cause problems with the Encapsulating Security Payload (ESP). This can be verified by checking to see if the number packet encryption is increasing and that the number of the packet decryption is unchanged on the VPN Client. On the VPN Concentrator, if you look at the VPN Concentrator session statistics, you will see that there is no change in the number of packet encryptions or to the number of packet decryptions (see under Administration > Administer Sessions). If you have a NAT/PAT device, configure for NAT Transparency (NAT-T) or IPsec over Transmission Control Protocol (TCP) or User Datagram Protocol (UDP) on both VPN 3000 Concentrator and on the VPN client as per the following link:
On the VPN Concentrator, go to the Client Config tab of the Group to configure NAT-T or IPsec over TCP.
On the VPN Client Properties window, click on the Transport tab, and turn on NAT-T or IPSEC over TCP as shown in Figure 8-8.
Figure 8-8. NAT-T Configurations on VPN Client PC
Check maximum Transmission Unit (MTU) Issues.
If you have solved both the routing and PAT issues, and you can ping the resources in your corporate network, you still might be unable to use certain applications. If so, there is a very good chance that you are running into MTU issues across the path. You can verify this by pinging the application server with "l" and "f" options. l is used to define the number of bytes and f forces the packet not to be fragmented. Start with a 1500 bytes packet and reduce this number until you stop receiving the message "Packet needs to be fragmented but DF set," and get a ping reply.
Example 8-13 shows a sample ping test performed from a Windows machine to uncover any MTU issues.
Example 8-13. Sample Test To Identify MTU Issues
To set up MTU on Cisco VPN client, follow Start > Programs > Cisco Systems VPN Client > Set MTU
D:\>ping -f -l 1300 www.xyz.com Pinging www.xyz.com [10.1.1.1] with 1300 bytes of data: ! The following lines indicate that 1300 bytes are too big along the path, so ! need to reset the MTU to some sCAller number on the VPN client. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Packet needs to be fragmented but DF set. Ping statistics for 10.1.1.1: Packets: Sent = 4, Received = 0, Lost = 4 (100% loss), ApproxiCAte round trip times in milli-seconds: Minimum = 0ms, CAximum = 0ms, Average = 0ms D:\>ping -f -l 1250 www.xyz.com Pinging www.xyz.com [10.1.1.1] with 1250 bytes of data: ! As the following lines indicate a reply, that means MTU can be set to this ! number minus 42 byeted for layer VII to layer IV header. So, ideally you ! should set the MTU on the client to be 1208 bytes. Reply from 10.1.1.1: bytes=1250 time=51ms TTL=47 Reply from 10.1.1.1: bytes=1250 time=29ms TTL=47 Reply from 10.1.1.1: bytes=1250 time=32ms TTL=47 Reply from 10.1.1.1: bytes=1250 time=34ms TTL=47 Ping statistics for 10.1.1.1: Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), ApproxiCAte round trip times in milli-seconds: Minimum = 29ms, CAximum = 51ms, Average = 36ms D:\>
A more detailed discussion on MTU can be found at the following location:
On the VPN 3000 Concentrator's public interface (where the VPN tunnel terminates), set the highest MTU value possible.
Check for filters.
Three types of filters can affect the traffic going through the tunnel. These three filters are:
Work through the following steps to verify if the filter is causing the problem:
- Static filters applied to an interface
- Static filters applied to the user group
- Dynamic filters applied through the RADIUS Server
- (a). Try to determine if traffic is indeed being dropped due to a filter by checking the counters on Monitoring > Statistics > Filtering. If the "Filtered" counter increases over a period of time, there is a very good chance that traffic is being dropped by the filter.
- (b). Next, verify if the rule you have defined for the filter is working the way you intended. You can verify this by changing the Action of the rule to Drop and Log or Forward and Log by going into Configuration > Policy Management > Traffic Management > Rules > Add or Modify. Then enable the FilterDBG Event Class to Severity to Log 1-9. This will allow traffic affected by the rule to be logged to Syslog or to the Event Log.
- (c). Check the Dynamic filter assigned by the RADIUS Server to see the rules being applied to a dynamic filter. To do so, go to the Monitoring > Dynamic Filters page. You can find details on Dynamic Filters in the detail screen of a specific user session on the Monitor > Session > Detail screen. Analyze the dynamic filter applied for the specific user and make sure the filter is accurate. If the filter is applied incorrectly, correct the configuration on the RADIUS Server.
Check for Microsoft networking issues.
If you have a problem specific to Microsoft networking through the VPN tunnel, refer to the following link:
VPN Client can connect but User Cannot Access the Internet
If you can access all resources on the private network of the other side of the tunnel (VPN 3000 Concentrator) but cannot access anything on the Internet, it could be because one of the following setups is not configured for tunneled traffic:
You are tunneling all traffic but do not have a Gateway on the VPN Concentrator for Internet-bound traffic On the VPN Concentrator, you can configure it to force all traffic to go through the tunnel from the VPN client. This ensures that the VPN client cannot make any connections to the Internet in clear text bypassing the IPsec tunnel. This provides extra security to the client while it is connected to the corporate network via tunnel. If you want the users from the VPN client under this setup to be able to connect to the Internet as well, you need to define a route for the Internet-bound traffic on the VPN Concentrator. The easiest solution is to define a tunnel default gateway to an Internal Router and have the internal router make the routing decision for the inside traffic versus Internet-bound traffic. The VPN Concentrator itself cannot be used to redirect the traffic on the public interface where the tunnel is terminated.
You do not have the Split tunneling configured If you do not want to send the Internet-bound traffic across the tunnel from the VPN Client, then be sure to configure split tunneling so that VPN client can make the connections to the Internet in clear text from the client PC itself. Note that with split tunneling, you are leaving the client PC vulnerable. Therefore, be sure to install a Personal Firewall or other host base Intrusion Detection Agent (for example CSAgent). To configure Split Tunneling, you first need to define a Network List that includes the networks that need to be tunneled. To define a network list, go to Configuration > Policy Management > Traffic Management > Network Lists. Then click on Add or Modify.
Once you have defined the Network List, you need to apply the Network List under the Client Config Tab of the Group Setup page. Under the Client Config tab, select only tunnel networks in the list. From the Split Tunneling Network List drop-down list, select the Network List defined earlier to allow going through the tunnel. No special configuration is required on the VPN Client.
VPN Client Can Connect But Users Cannot Access the Local LAN
If you cannot access any resources on the local LAN (for example, you are unable to print to the local printer) after building up the VPN tunnel, use the VPN client local LAN feature to allow local LAN traffic that is not encrypted. Go to the Client Config Tab of the Group Setup page. Under the Client Config tab, select the Tunnel everything button and check the Allow the networks in list to bypass the tunnel box. From the Split Tunneling Network List, select VPN Client Local LAN (Default).
To turn on Local LAN access, on the VPN Client GUI, for the user profile, go under the Transport tab, and check Allow Local LAN Access.
Digital Certificate Issues
Digital Certificate can be used to authenticate peers for both LAN-to-LAN and Remote Access VPN connections. As the creation and usage of Certificate for peer authentication is very similar for both LAN-to-LAN and Remote Access VPN connections, this section describes the Digital Certificate implementation only for the Remote Access VPN connection. One important point to note is that when a digital certificate is used for the peer authentication, both Remote Access VPN and LAN-to-LAN connections perform Main Mode negotiation. As you have seen before, with the pre-shared key, however, although the LAN-to-LAN connection performs Main Mode negotiation, Remote Access VPN connection uses Aggressive Mode negotiation for phase I tunnel establishment. The sections that follow provide detailed information on these topics:
Digital Certificate on the VPN Client
Work through the following steps to configure a VPN client with the digital certificate:
Enroll or import the CA root certificate and digital certificate for the VPN client connection entry by clicking on Certificates > Enroll or Import and refer to the following links:
Under the Certificate Tab, Select Verify. This will check if the certificate has a complete chain and its validity dates are acceptable.
Assign the certificate to the connection entry through the Authentication tab by selecting the Certificate Authentication radio button and selecting the certificate from the drop-down list of Certificate name.
Optionally choose Send CA Certificate Chain option if you want to send the chain of the certificate to the Concentrator.
Digital Certificate on the VPN Concentrator
Just as with a VPN Client, you must import the CA root and digital certificate on the VPN Concentrator. Work through the following steps to import the CA root certificate and the digital certificate on the VPN Concentrator:
On VPN 3000 Concentrator GUI, go to Administration > Certificate Management, and import/install the CA root certificate and the VPN Concentrator. Identity the certificate (Digital certificate) by using the instructions in the following links:
Go to the Administration > Certificate Management page to verify both the root and the to identity the certificates and their validity. The concentrator does not allow the installation of an ID certificate that does not have a complete chain. Remember that the concentrator cannot install an ID cert that was not requested by the specific concentrator. Select the View option of the Identity (ID) certificate to verify its validity. If the certificate validity dates have expired, you will see a message that is similar to this: The certificate is not yet valid. Regenerate the certificate to correct the problem. Also be sure that the VPN Concentrator time is not off by much; otherwise, a valid certificate may be shown as invalid, as the VPN Concentrator time may not fall between the validity dates of the certificate.
Go to Configuration > Tunneling and Security > IPsec > IKE Proposals on the VPN Concentrator GUI, and be sure that the VPNClient-3DES-MD5-RSA rule is under ActiveProposals. By default this is not under Active Proposals.
Configure a custom or modified IPsec SA that will use the certificates that you have installed and verified by going to the Configuration > Policy Management > Traffic Management > Security Associations > Add page. The Issuer CN field designates the certificates in the SA configuration. The Negotiation Mode must be set to Main Mode. If the client does not have the full certificate chain installed you, may have to send the entire certificate chain. The IKE Proposal must support digital certificates.
Group Matching is performed based on one or more fields in the incoming client certificate DN. If you are matching to a group from the OU field in the certificate, the defined group name must match the OU exactly. If you "Default to Group", that group must have an SA that supports certificates. Select the options by going into Configuration > Policy Management > Group Matching > Policy.
The group that matches must have an IPsec SA assigned that supports certificates. You can also enable peer identity checking. Be sure that the configuration setting matches the certificates in use by the clients. To configure the group to use the certificate, go to Configuration | User Management | Groups | Modify, which will allow you to modify the group. Click on the IPsec tab, and select IPsec SA from the drop-down list that you have created earlier.
As mentioned before, when a certificate is used for IKE phase I authentication, Main mode negotiation takes place between two IPsec peers for both LAN-to-LAN and Remote Access VPN connections. Before looking into the details of how to troubleshoot the IKE phase I negotiation when the Certificate is used, it is important to go through the event log on both the VPN Client and Concentrator and become absolutely comfortable with the sequence of events that takes place to isolate the problem with certification failure.
Example 8-14 shows the event log on the VPN Client with a successful phase 1 Main Mode negotiation.
Example 8-14. Event Log for a Successful Phase I Main Mode Negotiation on VPN Client with Certificate for Authentication
! Following message indicates that the tests for the Certificate Chain, its Validity ! dates and the Presence of the Private Key are successful. 850 16:03:23.964 05/28/05 Sev=Info/4CERT/0x63600013 Cert (cn=HTTS,ou=TAC,o=Cisco,l=San Jose,st=CA,c=US) verification succeeded. ! The VPN Client Sends Main Mode Message 1 with IKE Proposals that required certificate. ! If the VPN Concentrator doesn't have an Active IKE Proposal with Certificate ! Capability, the Negotiations Fails at this stage on the Concentrator. 857 16:03:25.105 05/28/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK MM (SA, VID(Xauth), VID(dpd), VID(Nat-T), VID(Frag), VID(Unity)) to 172.16.172.157 ! The VPN Client Receives Main Mode Message 2 from Concentrator indicating that the IKE ! Proposal Was OK. 862 16:03:25.198 05/28/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK MM (SA, VID(Frag)) from 172.16.172.157 ! VPN Client is sending Main Mode Message 3 865 16:03:25.214 05/28/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK MM (KE, NON, VID(?), VID(Unity)) to 172.16.172.157 ! The VPN Client receives Main Mode Message 4 containing the Cert Request 871 16:03:25.370 05/28/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK MM (KE, NON, CERT_REQ, CERT_REQ, CERT_REQ, CERT_REQ, CERT_REQ, VID(Unity), VID(Xauth), VID(?), VID(?)) from 172.16.172.157 ! The VPN Client sends Main Mode Message 5 with the Client Cert to the Concentrator. 875 16:03:25.511 05/28/05 Sev=Info/4IKE/0x63000013 SENDING >>> ISAKMP OAK MM *(ID, CERT, CERT_REQ, SIG, NOTIFY:STATUS_INITIAL_CONTACT) to 172.16.172.157 ! The VPN Client receives Main Mode Message 6 Containing the Concentrator Cert. 885 16:03:25.792 05/28/05 Sev=Info/4IKE/0x63000014 RECEIVING <<< ISAKMP OAK MM *(ID, CERT, SIG, VID(dpd)) from 172.16.172.157 ! VPN Client validates the Concentrator Cert signed by the right Root certificate and is ! ready to Start X-AUTH 886 16:03:25.886 05/28/05 Sev=Info/4CERT/0x63600013 Cert (cn=ID from Break,ou=TAC,o=Cisco,l=San Jose,st=CA,c=US) verification succeeded. ! Phase 1 SA is established here 889 16:03:25.886 05/28/05 Sev=Info/4CM/0x6310000E Established Phase 1 SA. 1 Crypto Active IKE SA, 0 User Authenticated IKE SA in the system
Example 8-15 shows the corresponding event log on the VPN 3000 Concentrator with a successful phase 1 Main Mode negotiation.
Example 8-15. Event Log for A Successful Phase I Main Mode Negotiation on Concentrator with Certificate for Authentication
! Main Mode message 1 sent by the VPN Client with IKE proposals is received by the VPN ! Concentrator 12 05/16/2005 16:15:01.910 SEV=8 IKEDBG/81 RPT=19017 172.16.172.1570 RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 1144 ! As the Concentrator does not log sending Main Mode Message 2 unless there is a failure ! you do not see the MM Message 2. If there is misMatch on Active IKE Proposals, you will ! receive "All SA Proposals Found Unacceptable". As there is no message logged, this is ! an indication that IKE proposals are accepted by the VPN Concentrator. ! Main Mode message 3 is received on the Concentrator from the VPN Client. You May ! receive duplicate of this message in certain version of code, which you can ignore. 1305 05/16/2005 16:15:02.110 SEV=8 IKEDBG/81 RPT=19018 172.16.172.1570 RECEIVED Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 224 ! Following is the Main Mode Message 4 sent to the VPN Client Requesting for the ! Client Certificate. 1319 05/16/2005 16:01:37.690 SEV=9 IKEDBG/0 RPT=30523 172.16.172.1570 constructing certreq payload 1332 05/16/2005 16:01:37.710 SEV=8 IKEDBG/81 RPT=18861 172.16.172.1570 SENDING Message (msgid=0) with payloads : HDR + KE (4) + NONCE (10) total length : 961 ! The following is Main Mode message 5 that the Concentrator Receives from the VPN client ! which contains the Client Cert and a Request for Concentrator's cert by the client 1334 05/16/2005 16:01:37.920 SEV=8 IKEDBG/81 RPT=18862 172.16.172.1570 RECEIVED Message (msgid=0) with payloads : HDR + ID (5) + CERT (6) + CERT_REQ (7) + SIG (9) + NOTIFY (11) + NONE (0) total length : 1555 ! The VPN Concentrator first tries to Match a Group to the incoming client Cert ! based on the IP address. 1343 05/16/2005 16:01:37.930 SEV=9 IKEDBG/23 RPT=42 172.16.172.1570 Starting group lookup for peer 172.16.172.1570 ! VPN Concentrator will be unable to find out the group based on the IP address of the ! client, which is expected, and you can ignore the following failure. 1344 05/16/2005 16:01:37.930 SEV=5 IKE/21 RPT=10 172.16.172.157 No Group found by Matching IP Address of Cert peer 172.16.172.157 ! Now Concentrator will perform the group Match based on the Cert subject and DN. You ! need to raise the CERT class up to level 1-9 to see the following level of details. 1345 05/16/2005 16:01:37.930 SEV=9 CERT/107 RPT=5 Performing group Match for cert peer 172.16.172.157 using the following cert subject DN info: OID: CountryName (26) value: US OID: StateProvinceName (28) value: CA OID: LocalityName (27) value: San Jose OID: OrganizationName (30) value: Cisco OID: OrganizationalUnitName (31) value: TAC OID: CommonName (23) value: HTTS 1353 05/16/2005 16:01:37.930 SEV=9 CERT/108 RPT=5 Performing group Match for cert peer 172.16.172.1570 using the following cert issuer DN info: OID: EmailAddress (13) value: email@example.com OID: CountryName (26) value: US OID: StateProvinceName (28) value: CA OID: LocalityName (27) value: San Jose OID: OrganizationName (30) value: Cisco OID: OrganizationalUnitName (31) value: TAC OID: CommonName (23) value: system ! Following three messages indicates the group Cisco is found based on issuer value ! O=Cisco. 1362 05/16/2005 16:01:37.930 SEV=9 CERT/109 RPT=5 Group Match component succeeded for cert peer 172.16.172.1570 OID (30) "CISCO" = "CISCO" 1364 05/16/2005 16:01:37.930 SEV=5 CERT/110 RPT=10 Group Match for cert peer 172.16.172.1570 succeeded using rule issuer-o="cisco" 1365 05/16/2005 16:01:37.930 SEV=5 CERT/105 RPT=10 Group [Cisco] found for cert peer 172.16.172.157 by group Match rule issuer-o="cisco" ! Finally, Group Cisco meets the Criteria 1367 05/16/2005 16:01:38.040 SEV=7 IKEDBG/80 RPT=42 172.16.172.1570 Group [Cisco] Found Phase 1 Group (Cisco) ! Following message indicates the Concentrator finds the Client Certificate to be valid ! in terms of Dates and Certificate Authority (CA). 1376 05/16/2005 16:01:38.050 SEV=7 CERT/1 RPT=5 Certificate is valid: session = 9 ! Certificate Revocation List (CRL) checking is done here. 1390 05/16/2005 16:01:38.050 SEV=5 CERT/116 RPT=5 Requesting CRL using HTTP. The HTTP URL is: http://xyz.com/CertEnroll/system.crl ! If you want to see the details of CRL http transaction, you need to use the CLIENT ! Event Class. 1392 05/16/2005 16:01:38.060 SEV=7 CLIENT/28 RPT=1 CLIENT_InitiateRequest(718d9b8, 7) 1407 05/16/2005 16:01:38.210 SEV=9 CLIENT/23 RPT=1 Total HTTP data bytes received: 327 ! The following message shows the CRL Check 1427 05/16/2005 16:01:38.210 SEV=7 CERT/2 RPT=5 Certificate has not been revoked: session = 9 ! The Concentrator validates it's own Certificate as shown below. 1429 05/16/2005 16:01:38.220 SEV=5 IKE/79 RPT=8 172.16.172.1570 Group [Cisco] Validation of certificate successful (CN=HTTS, SN=44B4C7C8000000000030) ! All Client and Concentrator Certificate Processing is SuccessfulSending Main Mode ! Message 6 as shown by the following lines. 1438 05/16/2005 16:01:38.240 SEV=8 IKEDBG/81 RPT=18863 172.16.172.1570 SENDING Message (msgid=0) with payloads : HDR + ID (5) + CERT (6) total length : 1549
Now that you are comfortable with the event log for a successful phase 1 negotiation of a Main Mode Remote Access VPN connection, work through the following steps to troubleshoot the digital certificate issues:
If the VPN client does not send Main Mode Message 1, there is a problem with the certificate on the client. Check the validity dates, and make sure the personal certificate ties back to a trusted root. If the certificate is stored on the local system, ensure that it can be read by VPN client software.
If the tunnel establishment fails in Main Mode Message 1 and 2, there may be a problem with active IKE proposals on the Concentrator. Be sure there is an active proposal that supports certificates.
If the Concentrator does not receive Main Mode Message 5 from the VPN client, it could be because of IKE fragmentation. The certificate sent from the VPN client might be big enough to cause the packet to be fragmented.
One of the following reasons could cause the Concentrator to receive the Main Mode message 5, but fails:
- - Group Matching Be sure your rules are defined properly so that the VPN Concentrator can perform group matching properly. Raise the CERT and IKEDBG Event Classes to level 19 to see group Matching information.
- - The group may not have the right IPsec SA associated with it You must have right IPsec SA associated with the group. The default logging gives you this information.
- - The Client certificate does not validate If the VPN client certificate does not validate, be sure the certificates matched to the group are from the same Root CA as the certificates from the VPN client. If there is a tiered CA structure, set the client to Send CA Certificate Chain. If the certificate fails due to certificate chain issues, you might receive the following message on VPN Concentrator:
Unable to complete certificate chain, reason = Incomplete certificate chain
- - CRL checking Failure Be sure the search strings and URLs are defined correctly. For LDAP, take a sniffer trace to analyze the response from the LDAP server. The Cert Event Class shows the DN you are sending. For http, the Client Event Class will add detail to the exchange. If the search is all right, be sure the cert has not been revoked (Cert Event Class can be used to find details).
- - After CRL checking, the concentrator validates its own certificate before sending it to the client If this validation fails, default logging provides you the information. To correct the problem, check and correct the time and date on the Concentrator.
Finally the Concentrator sends Main Mode Message 6 to the VPN ClientThis may fail at the VPN client primarily due to a chaining issue. Set the Certificate Transmission parameter in the IPsec SA to entire certificate chain for tiered CA structures.