| The problem areas of IPsec are divided into the following categories based on their implementations: Basic LAN-to-LAN troubleshooting GRE over IPsec troubleshooting PKI-based troubleshooting Remote Access VPN troubleshooting
Basic LAN-to-LAN Troubleshooting A basic LAN-to-LAN tunnel is the simplest form of IPsec tunnel in which private networks behind the tunnel end points communicate across the tunnel. Because this is the simplest form of IPsec Tunnel, the troubleshooting steps that are described in the sections that follow form the foundation knowledge required for the subsequent sections of this book. Before going into the details of troubleshooting steps for a basic LAN-to-LAN tunnel, it's extremely important to understand a successful tunnel establishment process and understand the debug information. This section starts with this discussion. Then the section goes into the details of tunnel establishment failure issues, and issues with sending traffic across the tunnel. The following section is organized as follows: Successful LAN-to-LAN tunnel establishment process Tunnel establishment fails due to Phase I failure Tunnel establishment fails due to Phase II failure Tunnel is established but unable to pass any traffic
Successful LAN-to-LAN Tunnel Establishment Process Before looking into the details of configuration and troubleshooting steps for a LAN-to-LAN tunnel, it is important to understand the sequence of events that occurs when traffic needs to pass through an IPsec tunnel in both inbound and outbound directions. The list that follows explains both inbound and outbound IPsec traffic through the router configured with IPsec: For outbound IPsec traffic (inside to outside) If using IPsec, check for the input access-list and then if allowed, decrypt the IPsec packets; otherwise drop the packets. Check the input access-list for decrypted traffic (Beginning with Version 12.3(8), this step is omitted). Check policy routing and then routing. Perform NAT and Crypto (check map and mark for encryption). Perform Outbound access control list (ACL) checking after Network Address Translation (NAT). If allowed and marked for encryption by the interesting traffic ACL, main/aggressive Mode starts to establish Phase I with a bidirectional IKE SA. Then Quick mode, which is Phase II negotiation, triggers and establishes IPsec SA, which is unidirectional; so it needs two SAs for both traffic directions. Then either with AH or ESP or both sides, send actual data over the tunnel.
For Inbound IPsec traffic (Outside to Inside) The sequence of events is the same as for outbound except that NAT occurs before policy routing or routing.
Figure 6-6 shows a typical LAN-to-LAN tunnel setup. The configuration and troubleshooting for the remainder of this section is based on Figure 6-5. Figure 6-6. Basic LAN-to-LAN Tunnel Between the Dhaka and Doha Routers 
Example 6-6 shows the Dhaka router's basic LAN-to-LAN tunnel configuration. Note that Example 6-6 shows only the IPsec configuration; the irrelevant configuration has been removed for clarity. Example 6-6. Configuration of the Dhaka Router for LAN-to-LAN Tunnel Shown in Figure 6-6 Dhaka# show running-config ! Define Phase 1 parameters under ISAKMP policy crypto isakmp policy 10 encryption 3des authentication pre-share crypto isakmp key cisco123 address 30.1.1.1 ! Transform-set defines IPSec encryption and authentication hashing algorithm for Phase ! II crypto ipsec transform-set myset esp-3des esp-md5-hmac ! ! Apply the transform-set and the ACL, peers and so on under crypto map crypto map to_Doha 10 ipsec-isakmp set peer 30.1.1.1 set transform-set myset match address 101 ! Interface that is connected to the private side of the network interface Ethernet0/0 ip address 10.1.1.1 255.255.255.0 ! ! crypto map is then applied to an outbound interface interface Ethernet1/0 ip address 20.1.1.1 255.255.255.0 crypto map to_Doha ! ip classless ! The route must exist for the other side's private network which is 10.1.2.0/24. The ! following default route defines includes this network. Also make sure that the next hop ! is on the direction of outgoing interface where crypto map is applied. ip route 0.0.0.0 0.0.0.0 20.1.1.100 ! Access-list defines interesting VPN traffic access-list 101 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255 Dhaka#
|
Example 6-7 shows the Doha router's basic LAN-to-LAN tunnel configuration. Example 6-7. Configuration of the Doha Router for LAN-to-LAN Tunnel Shown in Figure 6-6 Doha# show running-config ! Define Phase 1 parameters under ISAKMP policy crypto isakmp policy 10 encr 3des authentication pre-share crypto isakmp key cisco123 address 20.1.1.1 ! Transform-set defines IPSec encryption and authentication hashing algorithm for Phase ! II crypto ipsec transform-set myset esp-3des esp-md5-hmac ! ! Apply the transform-set and the ACL, peers and so on under crypto map crypto map to_Dhaka 10 ipsec-isakmp set peer 20.1.1.1 set transform-set myset match address 101 ! Interface that is connected to the private side of the network interface Ethernet0/0 ip address 10.1.2.1 255.255.255.0 ! ! crypto map is then applied to an outbound interface interface Ethernet1/0 ip address 30.1.1.1 255.255.255.0 crypto map to_Dhaka ! ip classless ! The route must exist for the other side's private network which is 10.1.2.0/24. The ! following default route defines includes this network. Also make sure that the next hop ! is on the direction of outgoing interface where crypto map is applied. ip route 0.0.0.0 0.0.0.0 30.1.1.100 ! Access-list defines interesting VPN traffic access-list 101 permit ip 10.1.2.0 0.0.0.255 10.1.1.0 0.0.0.255 Doha#
|
Now, before going into the details of debug command output, go through a packet flow. Take an example based on the LAN-to-LAN tunnel that has just been configured. Assume that host 10.1.1.10 on the LAN segment of the Dhaka router wants to communicate with a host 10.1.2.10 that is on the LAN segment of the Doha router. Also, assume that the connection is initiated from Dhaka to Doha. The following sequence of events takes place: 1. | Based on the route on Host with IP address 10.1.1.10, an IP packet is sent to the Dhaka router. Dhaka router will consult its routing table and based on its default route, it will forward the packet via the Ethernet 1/0 interface.
| | | 2. | Before the packet is transmitted out of this interface, as there is a crypto-map applied, the Dhaka router will consult the SPD to see if there is a policy match for this packet (Source:10.1.1.10, Destination: 10.1.2.1). The SPD is built based on the access-list defined for interesting traffic. As the access-list 101 includes the packet's source and destination address, the router will decide that the traffic needs to be IPsec protected.
| 3. | The next step is to see if an IKE or IPsec SA is already established to the IPsec peer. Because this is the first packet to this destination, there will be no SA existing in the SADB. All packets that match this policy can be queued or dropped until the IKE and IPsec SA are established. IOS IPsec drops all packets while waiting for IKE and IPsec SAs to be established. That's why if you ping, you will first see some one- or two-packet loss.
| 4. | IKE phase 1 negotiation is performed between the peers to establish the ISAKMP SA. The IKE phase negotiation is based on the ISAKMP policy. The establishment of ISAKMP can be verified with the show crypto isakmp sa command.
| 5. | Upon successful establishment of ISAKMP SA (Phase I), IPsec SA negotiation takes place. This can be verified with the show crypto ipsec sa command.
| 6. | After the IKE and IPsec SA are established, the original IP packet is encapsulated in IPsec tunnel mode using an ESP header, as this is what is configured in the transform-set. As Tunnel mode is used, a new IP header is added with the source IP address of 20.1.1.1 and destination IP address of 30.1.1.1. This packet is handed back to the usual IP forwarding routines to process the packet.
| 7. | The encrypted IPsec packet arrives into the Doha router, and the presence of an IPsec header in the IP packet indicates that it must be decrypted on the Doha router.
| 8. | The outer header destination address (30.1.1.1), the security protocol ESP, and the SPI in the ESP header are used as indexes into the SADB to find the SA for this packet.
| 9. | Once there is a hit in the SADB, the packet is authenticated and then decrypted using the proper transforms.
| 10. | Once the packet is decrypted, the 5-tuple check of the policy corresponding to the packet is checked. The 5-tuple is the source address of the inner IP header, destination address of the inner IP header, source port of the inner header, destination port of the inner header, and the IP protocol ID. Querying the SPD validates the 5-tuple. The lifetime counter in bytes is decremented and the anti-replay window is updated.
The packet would have been dropped in the following cases:
- a. The SA was not in the SADB.
- b. The sequence number was on the left of the sliding window; therefore, the packet failed the replay check.
- c. Authentication failed.
- d. The length was incorrect.
- e. The SA lifetime had expired.
| | | 11. | If all goes well in the previous step, the packet is passed to the usual IP packet processing on the hub router.
|
Now that you understand the packet flows for the IPsec packet, it is time to explore the debug commands output to correlate the packet path as just discussed. Example 6-8 shows the debug output on the Dhaka router. Example 6-8. Dissection of Most Useful debug Commands Output Dhaka# show debug Cryptographic Subsystem: Crypto ISAKMP debugging is on Crypto Engine debugging is on Crypto IPsec debugging is on ! Traffic is initiated from host 10.1.1.10 to host 10.1.2.10 which matches the ACL ! defined for interesting traffic. 22:17:24.426: IPsec(sa_request): , ! In the following line, local is the Dhaka router (Initiator) IP address, and remote is ! the Doha router's (Responder) IP address. (key eng. msg.) OUTBOUND local= 20.1.1.1, remote= 30.1.1.1, ! local proxy is source network behind the Initiator (Dhaka Router) and remote proxy is ! the Destination Network behind the Responder (Doha Router). This information is driven ! by the access-list defined for the crypto map. local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), ! This information is based on the transform-set defined protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 3600s and 4608000kb, spi= 0x4579753B(1165587771), conn_id= 0, keysize= 0, flags= 0x400A ! Main Mode exchange begins here. The first two packets negotiate phase I SA parameters ISAKMP: received ke message (1/1) ISAKMP: local port 500, remote port 500 ISAKMP (0:1): Input = IKE_MESG_FROM_IPSec, IKE_SA_REQ_MM Old State = IKE_READY New State = IKE_I_MM1 ISAKMP (0:1): beginning Main Mode exchange 22:17:24: ISAKMP (0:1): sending packet to 30.1.1.1(I)MM_NO_STATE 22:17:24: ISAKMP (0:1): received packet from 30.1.1.1 (I) MM_NO_STATE 22:17:24: ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH Old State = IKE_I_MM1 New State = IKE_I_MM2 22:17:24: ISAKMP (0:1): processing SA payload. message ID = 0 22:17:24: ISAKMP (0:1): processing vendor id payload 22:17:24: ISAKMP (0:1): Checking ISAKMP transform 1 against priority 10 policy 22:17:24: ISAKMP: hash SHA 22:17:24: ISAKMP: default group 1 22:17:24: ISAKMP: auth pre-share 22:17:24: ISAKMP: life type in seconds 22:17:24: ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80 22:17:24: ISAKMP (0:1): atts are acceptable. Next payload is 0 22:17:24: ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE Old State = IKE_I_MM2 New State = IKE_I_MM2 ! The policy 10 on this router and the atts offered by the other side matched ! Third and fourth packets completes Diffie-Hellman exchange ISAKMP (0:1): sending packet to 30.1.1.1 (I) MM_SA_SETUP ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE Old State = IKE_I_MM2 New State = IKE_I_MM3 ISAKMP (0:1): received packet from 30.1.1.1 (I) MM_SA_SETUP ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH Old State = IKE_I_MM3 New State = IKE_I_MM4 ISAKMP (0:1): processing KE payload. message ID = 0 ISAKMP (0:1): processing NONCE payload. message ID = 0 ISAKMP (0:1): found peer pre-shared key matching 209.165.201.4 ISAKMP (0:1): SKEYID state generated ISAKMP (0:1): processing vendor id payload ! The fifth and sixth packets complete IKE authentication and Phase 1 SA is established ISAKMP (0:1): SA is doing pre-shared key authentication using id type ID_IPV4_ADDR . . . . . . ISAKMP (0:1): sending packet to 30.1.1.1 (I) MM_KEY_EXCH ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETEOld State = IKE_I_MM4 New State = IKE_I_MM5 ISAKMP (0:1): received packet from 30.1.1.1 (I) MM_KEY_EXCH ISAKMP (0:1): Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH Old State = IKE_I_MM5 New State = IKE_I_MM6 ISAKMP (0:1): processing ID payload. message ID = 0 ISAKMP (0:1): processing HASH payload. message ID = 0 ISAKMP (0:1): SA has been authenticated with 209.165.201.4 ISAKMP (0:1): Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE Old State = IKE_I_MM6 New State = IKE_P1_COMPLETE !IKE Quick Mode IPSec SA Negotiations ! Begin Quick Mode exchange. The IPSec SA is negotiated in QM ISAKMP (0:1): beginning Quick Mode exchange, M-ID of 843945273 ISAKMP (0:1): sending packet to 30.1.1.1 (I) QM_IDLE ISAKMP (0:1): Node 843945273, Input = IKE_MESG_INTERNAL, IKE_INIT_QM Old State = IKE_QM_READY New State = IKE_QM_I_QM1 ISAKMP (0:1): received packet from 30.1.1.1 (I) QM_IDLE ! IPSec SA proposal offered by far end will be checked against local crypto map ! configuration ISAKMP (0:1): processing HASH payload. message ID = 843945273 ISAKMP (0:1): processing SA payload. message ID = 843945273 ISAKMP (0:1): Checking IPSec proposal 1 ISAKMP: transform 1, ESP_3DES ISAKMP: attributes in transform: ISAKMP: encaps is 1 ISAKMP: SA life type in seconds ISAKMP: SA life duration (basic) of 3600 ISAKMP: SA life type in kilobytes ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0 ISAKMP: authenticator is HMAC-MD5 ISAKMP (0:1): atts are acceptable. IPSec(validate_proposal_request): proposal part #1, (key eng. msg.) INBOUND local= 20.1.1.1, remote= 30.1.1.1, local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 0s and 0kb, spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x2 ! IKE Quick Mode SA Creation Phase ! Two IPSec SAs have been negotiated, an incoming SA with the SPI generated by initiator ! and an outbound SA with the SPIs proposed by the remote end ISAKMP (0:1): Creating IPsec SAs inbound SA from 30.1.1.1 to 20.1.1.1(proxy 10.1.2.0 to 10.1.1.0) has spi 0x8EAB0B22 and conn_id 2000 and flags 2 lifetime of 3600 seconds lifetime of 4608000 kilobytes outbound SA from 20.1.1.1 to 30.1.1.1 (proxy 10.1.1.0 to 10.1.2.0) has spi -1910720646 and conn_id 2001 and flags A lifetime of 3600 seconds lifetime of 4608000 kilobytes ! The IPsec SA info negotiated by IKE will be populated into router's SADB 22:17:25: IPsec(key_engine): got a queue event... 22:17:25: IPsec(initialize_sas): , (key eng. msg.) INBOUND local= 20.1.1.1, remote= 30.1.1.1, local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 3600s and 4608000kb, spi= 0x4579753B(1165587771), conn_id= 2000, keysize= 0, flags=0x2 22:17:25: IPsec(initialize_sas): , (key eng. msg.) OUTBOUND local= 20.1.1.1, remote= 30.1.1.1, local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 3600s and 4608000kb, spi= 0x8E1CB77A(2384246650), conn_id= 2001, keysize= 0, flags= 0xA ! IKE Quick Mode Phase 2 is completed ! IPsec SA created in SADB,sent out last packet with commit bit set; IPsec tunnel ! established IPsec(create_sa): sa created, (sa) sa_dest= 20.1.1.1, sa_prot= 50, sa_spi= 0x4579753B(1165587771), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2000 IPsec(create_sa): sa created, (sa) sa_dest= 30.1.1.1, sa_prot= 50, sa_spi= 0x8E1CB77A(2384246650), sa_trans= esp-3des esp-md5-hmac , sa_conn_id= 2001 ISAKMP (0:1): sending packet to 30.1.1.1 (I) QM_IDLE ISAKMP (0:1): Node 843945273, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH Old State = IKE_QM_I_QM1 New State = IKE_QM_PHASE2_COMPLETE Dhaka#
|
Once you are comfortable with the packet paths, and can read and understand the debug output, go through the next few sections to find how to troubleshoot the IPsec VPN issues. Tunnel Establishment Fails at Phase I As discussed before, IPsec is established in two phasesPhase I and Phase II. So, the tunnel establishment might fail in either phase. You should execute show commands as discussed in the "Diagnostic Tools and Commands" section. First identify if it is a Phase I or Phase II failure. If the show crypto isakmp sa shows anything other than the QM_IDLE state as shown in Example 6-1, the problem is with Phase I. Run the debug commands as discussed in the "Diagnostic Tools and Commands" section and compare the output as shown in the preceding section for a successful tunnel negotiation. Work through the following steps to isolate any issues pertaining to Phase I tunnel establishment: Step 1. | Correct connectivity problems between the peers.
You must make sure that tunnel end points (Dhaka and Doha routers) can ping to each other's public interfaces where the crypto map is applied. If they are unable to ping to each other, make sure you have the route defined for both (usually the defined default gateway if these are gateway routers). Remember that some ISPs may block the ICMP packet. Therefore ping may fail even with the proper connectivity. For IPsec tunnel to establish, you must have UDP/500 (ISAKMP) open in both directions of tunnel end points. A quick way to verify that would be to use the debug ip udp command, which will show if the initiator or the responder is getting any ISAKMP request or response or not.
| | | Step 2. | Avoid configuration mismatches between the tunnel end points.
If the IP connectivity is tested and shown to be without problems, you must ensure the configuration on the tunnel end points. For Phase I, the ISAKMP policy must match, which gets exchanged in packet 1 and 2 of Phase I in Main Mode. Packet 1 sends all the configured ISAKMP proposals on the Initiator to the responder. Responder must match at least one ISKAMP policy as a whole to be able to successfully build up the Phase I. Packet 2 carries either a matching proposal or a failure message. If there is no match, the initiator tries the default policy of 65535, and if that does not match either, it fails ISAKMP negotiation. This occurrence outputs the debug message shown in Example 6-9, which shows the default policy on the router.
Example 6-9. Sample Output of debug Commands when ISAKMP Fails with Policy Mismatch Dhaka# debug crypto isakmp 3d01h: ISAKMP (0:1): processing SA payload. message ID = 0 3d01h: ISAKMP (0:1): found peer pre-shared key matching 209.165.200.227 ISAKMP (0:1): Checking ISAKMP transform 1 against priority 1 policy ISAKMP: encryption 3DES-CBC ISAKMP: hash MD5 ISAKMP: default group 1 ISAKMP: auth pre-share ISAKMP: life type in seconds ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80 ISAKMP (0:1): Hash algorithm offered does not match policy! ISAKMP (0:1): atts are not acceptable. Next payload is 0 ISAKMP (0:1): Checking ISAKMP transform 1 against priority 65535 policy ISAKMP: encryption 3DES-CBC ISAKMP: hash MD5 ISAKMP: default group 1 ISAKMP: auth pre-share ISAKMP: life type in seconds ISAKMP: life duration (VPI) of 0x0 0x1 0x51 0x80 ISAKMP (0:1): Encryption algorithm offered does not match policy! ISAKMP (0:1): atts are not acceptable. Next payload is 0 ISAKMP (0:1): no offers accepted! ISAKMP (0:1): phase 1 SA not acceptable! Router#
|
To verify the ISAKMP policy, you need to execute show crypto isakmp policy on both routers, and correct the policy mismatch issue. Be sure that at least one of the policies matches in its entirety.
| | | Step 3. | Check for IKE Packets on the Initiator with the debug command.
If you run debug crypto isakmp, you should see some IKE packets on the initiator. If there are no IKE packets, go through the following list to correct the problem:
- - Check to see if Internet Security Association Management Protocol (ISKAMP) is turned off ISAKMP is enabled by default on Cisco routers. If you turned it off, the ISKAMP packet would not be generated. The quickest way to verify this is by executing show crypto isakmp sa as shown below:
Dhaka# show crypto isakmp sa ISAKMP is turned off Dhaka# To correct the problem, enable ISAKMP with the following command: Dhaka(Config)# crypto isakmp enable - - Find out if the Peer IP address configured correctly If the peer address is not configured correctly, or if the certificate is used but the router has not generated RSA keys, authenticated the CA, and obtained a certificate IKE, Phase I negotiation is not initiated and the following messages are displayed with the debug as shown in Example 6-10.
Example 6-10. Peer Address Is Not Configured Correctly Dhaka# debug crypto isakmp *Jul 5 22:44:45.760: ISAKMP: Looking for a matching key for 30.1.1.1 in default *Jul 5 22:44:45.760: ISAKMP: Looking for a matching key for 30.1.1.1 in test *Jul 5 22:44:45.760: ISAKMP (0:2): No pre-shared key with 30.1.1.1! *Jul 5 22:44:45.760: ISAKMP (0:2): No Cert or pre-shared address key. *Jul 5 22:44:45.760: ISAKMP (0:2): construct_initial_message: Can not start Mai Dhaka#
|
- - Check and correct the Routing Problem On the Initiator If ISAKMP is turned on and you are still unable to see IKE packets, execute show ip route host|network_address to verify that the route not only exists for the tunnel end point, but also for the actual destination network through the same interface as the crypto map applied. Otherwise, IKE will not be triggered. Usually, routers have default routes for both the remote side private network and the remote tunnel IP address.
- - Verify Crypto Map and make sure it is applied on the correct interface If the crypto map is not applied, or applied on the wrong interface, ISAKMP will not be triggered, and therefore you will not see any IKE packets. The best way to find out if the crypto map is applied to the correct interface is to execute the show crypto map tag crypto_map_name command on both initiator and responder. The crypto map needs to be applied on the outgoing interface of the router. The outgoing interface is the interface to which the routing table is pointing to reach the remote side private network. Additionally, if you don't want to use the outside interface's IP as the local ID for VPN tunnel, use the command crypto map <name> local-address <interface>, to specify the correct interface. If there are physical and logical interfaces involved in carrying outgoing traffic, the crypto map should be applied to both; however, this restriction has been removed in the latest Cisco IOS. A misconfigured crypto map results in the error shown in Example 6-11.
Example 6-11. Sample Output of IPsec debug Commands with the Crypto Map Applied on the Wrong Interface Router# show debug Cryptographic Subsystem: Crypto ISAKMP debugging is on Crypto Engine debugging is on Crypto IPsec debugging is on . . . ! Following line is an indication that a wrong local IP address is selected. IPsec(validate_proposal): invalid local address 209.165.201.4 ISAKMP (0:4): atts not acceptable. Next payload is 0 ISAKMP (0:4): phase 2 SA not acceptable! Router#
|
- - Check to see if the Interesting Access-list is configured correctly The crypto access-list defines the interesting traffictraffic that should go through the tunnel. If your VPN traffic is not included in the crypto ACL, IKE negotiation does not occur. Ideally, the crypto ACL on both sides of the tunnel should be mirror images of each other. For example, if you want to send traffic from 10.1.1.0 network behind the Dhaka router (Figure 6-6) to 10.1.2.0 network behind the Doha router, your access-list on the Dhaka router should be access-list 101 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255 and the exact reflection of this ACL should be access-list permit ip 10.1.2.0 0.0.0.255 10.1.1.0 0.0.0.255 on the Doha router. If the access-lists are not the reflection of each other, then you need to be sure that the responder has an ACL which is a superset of the ACL defined in the initiator. Otherwise, the tunnel establishment will fail. Here is an example to illustrate this point. Assume that Dhaka is the initiator, and Doha is the responder. If Dhaka has an ACL access-list 101 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255 and Doha has a superset ACL of access-list permit ip 10.1.0.0 0.0.255.255 10.1.0.0 0.0.255.255, the tunnel will be established if Dhaka initiates the tunnel. However, if Doha initiates the tunnel, it will fail, because the ACL on Doha is beyond the scope of the ACL on Dhaka.
| | | Step 4. | Verify IKE packets on the responder with debug commands.
If Step 3 is verified successfully but Phase I is still not established, verify that the responder is seeing the IKE packets. The following list outlines some of the common steps to follow to prevent the responder from receiving the IKE packets:
- Perform Step 1 to ensure that the IP connectivity is established. - Verify the peer IP address to ensure that the initiator is not sending IKE Phase I requests to the wrong IP. - IKE Phase I and Phase II negotiation occurs using UDP/500. If there is a firewall between the tunnel end points, or if the responder has an incoming ACL on the interface where crypto map is applied, make sure that UDP/500 is allowed. See Example 6-12 for the debug messages that are shown on the Dhaka router, which initiated the IKE messages, and shows retransmit, which is an indication that the receiver, the Doha router, is not getting the traffic. Example 6-12. IKE Negotiation Messages on Initiator the Dhaka Router when ISKMP Is Blocked by a Firewall or by the Interface ACL on the Doha Router Dhaka# debug crypto isakmp *Jul 5 22:50:42.144: ISAKMP (0:1): retransmitting phase 1 MM_NO_STATE... *Jul 5 22:50:42.144: ISAKMP (0:1): incrementing error counter on sa: retransmit phase 1 *Jul 5 22:50:42.144: ISAKMP (0:1): retransmitting phase 1 MM_NO_STATE *Jul 5 22:50:42.144: ISAKMP (0:1): sending packet to 30.1.1.1 my_port 500 peer_ port 500 (I) MM_NO_STATE *Jul 5 22:50:52.144: ISAKMP (0:1): retransmitting phase 1 MM_NO_STATE... *Jul 5 22:50:52.144: ISAKMP (0:1): incrementing error counter on sa: retransmit phase 1 *Jul 5 22:50:52.144: ISAKMP (0:1): retransmitting phase 1 MM_NO_STATE *Jul 5 22:50:52.144: ISAKMP (0:1): sending packet to 30.1.1.1 my_port 500 peer_ port 500 (I) MM_NO_STATE
|
Open up UDP/500 on both the firewall (in both directions of traffic) and the inbound ACL on the interface where the crypto map is applied to resolve this issue.
| Step 5. | Avoid authentication failure.
Authentication failure is one of the many reasons IKE phase I establishment might fail. If the pre-shared key is used for authentication, you must ensure that you have defined the same pre-shared key on both peers; otherwise, the IKE Phase I will fail, resulting in the debug messages shown in Example 6-13.
Example 6-13. Sample Output of the debug Commands Output with Pre-shared Key Mismatch Dhaka# debug crypto isakmp ISAKMP (62): SKEYID state generated ISAKMP (62); processing vendor id payload ISAKMP (62): speaking to another IOS box! ISAKMP: reserved not zero on ID payload! ! Following messages indicates an authentication failure %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 209.165.200.227 failed its sanity check or is malformed Router#
|
If the pre-shared key is not the same, correct the problem. If the pre-shared key is the same on both sides, remove and add the pre-shared key back again on the routers to see if the problem is corrected. If this also fails, there could be a bug. If you are using certificates for IKE authentication, refer to the section entitled "Public Key Infrastructure (PKI) Troubleshooting" for more details.
|
Tunnel Establishment Fails at Phase II After Phase I is established, two peers negotiate with each other to create the IPsec SA. If Phase II negotiation fails, IPsec SA is not established. The following are some of the steps you can follow to troubleshoot Phase II failure: Step 1. | Check for IPsec transform set mismatches between peers.
If there is a mismatch on IPsec Transform set between the peers, quick mode negotiation fails. You need to verify IPsec SA negotiation with the show crypto ipsec sa command. Example 6-14 shows the output of debug crypto isakmp on the initiator router, which is failing in quick mode due to a transform set mismatch resulting in the SA not acceptable message.
Example 6-14. IKE Quick Mode Fails with Transform Set Mismatch Dhaka# debug crypto isakmp Dhaka# debug crypto ipsec ISAKMP (0:2): Checking IPsec proposal 1 ISAKMP: transform 1, ESP_3DES ISAKMP: attributes in transform: ISAKMP: encaps is 1 ISAKMP: SA life type in seconds ISAKMP: SA life duration (basic) of 3600 ISAKMP: SA life type in kilobytes ISAKMP: SA life duration (VPI) of 0x0 0x46 0x50 0x0 ! Transform set mismatch as shown in the following message IPsec(validate_proposal): transform proposal (prot 3, trans 3, hmac_alg 0) not supported ISAKMP (0:2): atts not acceptable. Next payload is 0 ISAKMP (0:2): SA not acceptable! Dhaka#
|
| | | Step 2. | Resolve asymmetric crypto access-lists issues.
If the crypto map access-lists are not the mirror image of each other on both peers, Quick Mode might fail, or one side can initiate quick mode successfully, but the other side cannot. If you have the Dhaka router crypto access-list configured with source address 10.10.0.0/16 and destination address of 20.20.0.0/16, and the Doha router configured with source 20.20.20.0/24 and destination of 10.10.10.0/24, the Dhaka router cannot initiate successful quick mode because Dhaka's access-list address range is outside the scope of the address ranger on Doha's. Therefore, Doha rejects Dhaka's IPsec SA proposal. However, Doha can successfully initiate quick mode, as Dhaka's access-list addresses include addresses in Doha's. Example 6-15 shows the debug output of quick mode failure with asymmetric crypto access-lists.
Example 6-15. Output of debug Commands When Configured with the Wrong Access List Dhaka# debug crypto isakmp Dhaka# debug crypto ipsec Cryptographic Subsystem: Crypto ISAKMP debugging is on Crypto Engine debugging is on Crypto IPsec debugging is on . . . 1w6d: IPSec(validate_proposal_request): proposal part #1, (key eng. msg.) INBOUND local= 209.165.201.4, remote= 209.165.200.227, local_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 0s and 0kb, spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4 1w6d: IPsec(validate_transform_proposal): proxy identities not supported 1w6d: ISAKMP (0:2): IPSec policy invalidated proposal 1w6d: ISAKMP (0:2): phase 2 SA not acceptable!
|
Example 6-16 shows a properly configured mirror image access-list for two peers.
Example 6-16. A Sample of a Properly Configured Reflected Access List Dhaka(config)# access-list 101 permit ip 10.10.10.0 0.0.0.255 20.20.20.0 0.0.0.255 Doha(config)# access-list 101 permit ip 20.20.20.0 0.0.0.255 10.10.10.0 0.0.0.255
|
| | | Step 3. | Avoid overlapping access-lists.
If you have multiple peers on a router, make sure that the matching address access lists for each of the peers are mutually exclusive of the matching address access lists for the other peers. If you don't do that, the router chooses the wrong crypto map to try to establish a tunnel with one of the other peers. See Example 6-17 for a sample output of this behavior.
Example 6-17. Output of the IPsec debug Command Showing a Problem with Overlapping Access Lists Dhaka# debug crypto isakmp Dhaka# debug crypto ipsec ! Only showing the relevant debugs output IPsec(validate_proposal_request): proposal part #1, (key eng. msg.) INBOUND local= 209.165.200.227, remote= 209.165.202.149, local_proxy= 10.1.1.0/255.255.255.0/0/0 (type=4), remote_proxy= 10.1.2.0/255.255.255.0/0/0 (type=4), protocol= ESP, transform= esp-3des esp-md5-hmac , lifedur= 0s and 0kb, spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x4 IPsec(validate_transform_proposal): peer address 209.165.202.149 not found ISAKMP (0:2): IPsec policy invalidated proposal ISAKMP (0:2): phase 2 SA not acceptable! Access list for 209.165.201.4: Access-list 100 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255 Access-list 100 permit ip 10.1.1.0 0.0.0.255 10.1.5.0 0.0.0.255 Access list for 209.165.202.149: Access-list 110 permit ip 10.1.1.0 0.0.0.255 10.1.2.0 0.0.0.255 Dhaka#
|
After you go through Steps 1 through 3, make sure that Phase II is established executing the show crypto ipsec sa command, as shown in Example 6-2.
|
Tunnel Is Established but Unable To Pass Traffic Even if a tunnel is established, as verified in the preceding section, traffic still might not be able to flow for several reasons. The following steps will help in troubleshooting the problem with data passing across the tunnel: 1. | Routing issues
If either or both sides of encryption or decryption counters are not incrementing with show crypto ipsec sa command output, there might be an issue with routing. Be sure that you have a route defined for the remote side private network that is pointing to the gateway of the router through the interface where the crypto map is applied. To verify that the routing is taking place properly, you might want to run debug ip packet <acl> detailed. Example 6-18 shows how to run debug ip packet <acl> detailed for traffic originating from 10.1.1.10 behind the Dhaka router going to FTP server behind the Doha router, which has an IP address of 10.1.2.10 across the tunnel.
Example 6-18. Illustration of How to Use the debug ip packet <acl> detailed Command Dhaka(config)# access-list 101 permit tcp host 10.1.1.10 host 10.1.2.10 eq 21 Dhaka(config)# access-list 101 permit tcp host 10.1.2.10 eq 21 host 10.1.1.10 Dhaka(config)# exit Dhaka# debug ip packet 101 detail
|
Be sure to run in Process Switching so you can see the details of the packet. You will need to be careful when running this in a busy system.
| 2. | Firewall issues
If there is any firewall between the tunnel endpoints, or if there is an access-list in the egress interface of tunnel end points, be sure to open up protocol AH (IP/51) and Encapsulating Security Payload (ESP) (IP/50). For ISAKMP, you need to open up port UDP/500. If NAT-T is configured, UDP/4500 port also needs to be open. For more discussion on this, refer to the section entitled, "Firewall with IPsec Issues" under "Common Problems and Resolutions."
| | | 3. | Network Address Translation (NAT) issues
If you do not have a firewall blocking any of the necessary ports, check to see if there is any NAT device between the tunnel end points. You might also have NAT turned on at tunnel end points. To get around the problem, refer to the "NAT with IPsec Issues" subsection under the section entitled, "Common Problems and Resolutions."
| 4. | Hardware Encryption card problem
If you are running the Hardware Encryption card, and you are having problems passing traffic across the tunnel, as a temporary work-around, you might want to turn off the Hardware Encryption card. If turning off the Hardware Encryption card resolves the issue, the problem could be with the hardware, or you might be running into a bug with the Hardware Encryption card on the code.
| 5. | Interface problem
Different interfaces use different switching and encapsulation codes. If you are using serial, and it fails, try your setup on simple Ethernet or other interfaces.
| 6. | Switching path problem
Different switching methods use completely different code paths with completely different sets of problems. If CEF switching does not work, try regular fast-switching or vice versa. Turn off fast-switching and see if the problem goes away. Even if the problem does not disappear with the "process switched," it is much easier to debug IPsec problems with process switching, as you can actually get some useful packet debugs. With switching path problems, you might run into issues such as packet drops, latency, and reliability problems.
|
GRE over IPsec As mentioned at the beginning of this chapter, IPsec only encapsulates unicast IP traffic, which might be less than desirable in networks in which multicast or non-IP traffic needs to be transferred securely across the IPsec tunnel. With GRE over IPsec, this is possible. The following list outlines the reasons why GRE over IPsec is used: To pass multicast and broadcast traffic across the tunnel securely. To pass non-IP traffic (for instance, International Packet Exchange (IPX) across the tunnel securely. To have resiliency (high availability). For more details on this, refer to the section entitled, "Best Practices." To help in saving memory and CPU of the router. This is because, for every ACL entry, two IPsec SAs are created. Hence, if regular IPsec is configured, the number of ACLs to include the interesting traffic becomes high, which increases the number of IPsec SAs on the router. These ACEs can be replaced by a single ACL for GRE traffic, which ends up creating only two IPsec SAs. This saves memory on the router.
Configuration Steps To integrate GRE over IPsec, you need to configure the tunnel interface on both sides, and force the private network to go through the tunnel using dynamic routing protocol (you also can use static routing if you desire). When the actual packet arrives at the router, the router forwards the packet to the tunnel interface, and then encapsulates the packet with a GRE header and tries to send the packet to the outgoing interface where a crypto map is configured. So, the crypto map is triggered and encapsulates the packet with an IPsec header, as the interesting traffic ACL includes the GRE source and destination addresses. Example 6-19 shows a basic GRE over IPsec configuration on the Dhaka router, based on Figure 6-6. Basic LAN-to-LAN tunnel configuration on the Dhaka router as shown in Example 6-7 is replaced by the GRE over IPsec configuration as shown in the following example. Example 6-19. Dhaka Router Configuration for a Basic GRE Over IPsec Dhaka# show running-config ! Phase 1 policy is defined here for phase 1 of IPsec tunnel build up process crypto isakmp policy 1 encryption 3des authentication pre-share crypto isakmp key cisco123 address 30.1.1.1 ! Phase II attributed for IPsec tunnel is defined by transform-set crypto ipsec transform-set myset esp-3des esp-md5-hmac ! ! Phase II information is applied to the crypto map crypto map GRE_to_Doha 10 ipsec-isakmp set peer 30.1.1.1 set transform-set myset match address 101 crypto map GRE local-address Ethernet 1/0 ! Interface Ethernet 0/0 is connected to the private side of the network interface Ethernet0/0 ip address 10.1.1.1 255.255.255.0 ! ! Tunnel Interface Creation & crypto map applied interface Tunnel0 ip address 1.1.1.1 255.255.255.252 ip mtu 1440 tunnel source Ethernet 1/0 tunnel destination 30.1.1.1 ! Crypto map is applied to the tunnel interface. This is required if you are running ! IOS version prior to 12.2(13)T. Beginning with IOS version 12.2(13)T, it is ! strongly discouraged to apply the crypto map under the tunnel interface; only ! crypto map on the physical interface is required. crypto map GRE_to_Doha ! crypto map is then applied to an outbound interface interface Ethernet1/0 ip address 20.1.1.1 255.255.255.0 crypto map GRE_to_Doha !EIGRP turned to get GRE, and regular traffic updates to the routing table router eigrp 10 network 1.1.1.0 0.0.0.3 network 10.1.1.0 0.0.0.255 no auto-summary no eigrp log-neighbor-changes ip classless ! The following default route will allow Dhaka router to the talk to the Doha Router ip route 0.0.0.0 0.0.0.0 20.1.1.100 ! Access-list defines GRE traffic as interesting traffic access-list 101 permit gre host 20.1.1.1 host 30.1.1.1 Dhaka#
|
Troubleshooting Steps You can run the debug tunnel command to troubleshoot the GRE tunnel issue. It is strongly recommended to ensure that you can build up the GRE tunnel over IP before you turn on IPsec for the GRE tunnel. While working with GRE over IPsec, remember the following points: If you are running an IOS version earlier than 12.2(13)T, you need to apply the crypto map on both the tunnel interfaces and the physical interfaces to correctly invoke switching paths to achieve IPsec. Beginning with IOS Version 12.2(13)T, or 12.3 and later, the configured IPsec crypto map needs to be applied to the physical interface only, and is no longer required to be applied on the GRE tunnel interface. You can still apply the crypto map on both physical and tunnel interfaces, but it is strongly recommended to apply the crypto map on the physical interface only. Specify GRE traffic as IPsec interesting traffic. On the Dhaka router, the interesting traffic for the GRE traffic should be defined with the following access-list: Dhaka(config)# access-list 101 permit gre host 20.1.1.1 host 30.1.1.1 Make sure that the source and destination addresses are the source and destination addresses of the GRE tunnel interface. In this case, the source is 20.1.1.1, which is the actual physical interface IP address. The destination address is the remote peer IP address, which is 30.1.1.1. Static or Dynamic routing protocol needs to be configured for the remote private network pointing to the tunnel interface as next hop. GRE tunnel should be encrypted by the IPsec tunnel. Take appropriate precautions to avoid the recursive routing problem. Otherwise, the tunnel interface will flap (going up and down). This situation arises, when based on the routing table, the router sends the GRE packets to its own interface instead of sending them to the physical interface. The message you get on the router is as follows: Tunnel0 temporarily disabled due to recursive routing. Keep the following list in mind to avoid the recursive routing loop: - Use different routing protocols or separate routing protocol identifiers for GRE traffic and actual traffic. - Keep tunnel IP address and actual IP network address ranges distinct. - For the tunnel interface IP address, don't use unnumbered to loopback interface when the loopback's IP address resides in the ISP address space. - Use static routes for the end point of GRE tunnel interfaces. - Use route filtering so that the router does not point to the tunnel interface for its own routes.
Public Key Infrastructure (PKI) Troubleshooting So far, this chapter has discussed IKE authentication using pre-shared keys. This section looks into some of the issues that you might run into when using digital certificates as a means of IKE authentication. The digital certificate is issued by a certificate authority (CA) server. The CA should issue the certificate at the time you install the device. It is not done for each IKE exchange. That is, after a device obtains a certificate, it does not need to contact the CA during the connection setup process unless you configure the device for Certificate Revocation List (CRL) Verification. This setup avoids the reliability problems associated with a centralized key server. Configuration Steps The Cisco Certificate Enrollment Protocol (CEP) manages the certificate life-cycle process. This protocol supports operations such as certificate enrollment, certificate revocation, and certificate access, allowing Cisco routers to participate within a public key infrastructure (PKI). The general procedure has four steps as follows: Step 1. | The first step is to generate an RSA key pair on the router. The generated keys are saved in the private configuration in non-volatile random access memory (NVRAM). This private configuration is never displayed to the user or backed up to another device. A single CA must be defined for the router using its fully qualified domain name (FQDN) or IP address, the CA's HTTP common gateway interface (CGI) script path, and optionally, the certificate revocation list (CRL) query URL.
| Step 2. | The second step of the general procedure is to retrieve the CA certificate. To authenticate the certificate of another device, the router must query the CA to obtain the CA certificate containing the CA public key. Because the router has no means of validating the CA certificate automatically, the key should be authenticated manually by contacting the CA administrator and comparing the fingerprints of the certificate. The validated certificate is stored in the configuration, and the public key from the CA certificate is added to the RSA public key chain.
| | | Step 3. | The third step of the general procedure is to request that the CA generate a certificate for the key pair of the router by sending an enrollment request. A dialog is begun in which the user-defined certificate attributes are entered (a challenge password, a user addressable name, and optionally, the router serial number and IP addresses). The CA operator may cease to manually validate the request, and the fingerprints can be compared.
| Step 4. | The fourth and last step of the general procedure is to request receipt of the certificate. Because there might be a significant delay, the certificate is retrieved asynchronously.
|
Work through the following steps to enroll the certificate with the CA server: Step 1. | Define the hostname and domain-name.
This is required, because the router assigns FQDN to the RSA key and Certificates.
Router(config)# hostname Dhaka Dhaka(config)# ip domain-name cisco.com ! Hence the fully qualified domain name is FQDN= Dhaka.cisco.com Dhaka(config)#
| Step 2. | Set the router clock.
| Step 3. | Generate the RSA key pair.
RSA key pair is used to sign and encrypt IKE key management messages. By default, a general purpose RSA key pair is generated. You can specify that usage keys be generated, which will generate a separate RSA key pair for signing and encryption. The syntax for generating the RSA key pair is as follows:
crypto key generate rsa <general-keys> <usage-keys > <word>
In the Dhaka router, the RSA key is generated as follows:
Dhaka(config)# crypto key generate rsa The name for the keys will be: Dhaka.cisco.com ! The following message is because this RSA keys already exist on the router % You already have RSA keys defined for Dhaka.cisco.com. % Do you really want to replace them? [yes/no]: yes Choose the size of the key modulus in the range of 360 to 2048 for your General Purpose Keys. Choosing a key modulus greater than 512 may take a few minutes. ! If you don't define the modulus, 512 will be used. How many bits in the modulus [512]: Generating RSA keys ... [OK] Dhaka(config)#
| | | Step 4. | Define the Certificate Authority (CA).
Define the communication parameters between the router and CA with the following parameters:
- - Define enrollment URL specifies the CA location. Enrollment URL is the minimum configuration requirement.
- - Define Registration Authority provides interface between CA and end device (router), and publishes a CRL. This is the default enrollment mode, even without RA mode specified.
- - Define Certificate Revocation List URL Specifies the URL of the CDP (CRL Distribution Point), and off-box cert retrieval location.
- - Define enrollment retry-count This defines how many times a router will resend a certificate request. Default is 10, range is 1 to 100.
- - Define enrollment retry period This is the wait period between certificate request retries in minutes. Default uses back-off.
The following configuration is a setup on the Dhaka router for the CA server located at 20.1.1.100:
Dhaka(config)# crypto ca identity vpn Dhaka(ca-identity)# enrollment url http://20.1.1.100:80 Dhaka(ca-identity)# query url ldap://20.1.1.100 Dhaka(ca-identity)# enrollment retry count 20 Dhaka(ca-identity)# enrollment retry period 5 Dhaka(ca-identity)# enrollment mode ra Dhaka(ca-identity)#
| Step 5. | Authenticate the CA.
You need to authenticate the CA to obtain the CA's self-signed certificate, which contains the CA's public key as follows for the Dhaka Router:
Dhaka(config)# crypto ca authenticate vpn Certificate has the following attributes: ! This finger prints should matched with the published finger print of the CA ! server Fingerprint: 1FCDG2C9 2DEDA6DC 4819D44C B4CFF5F2 % Do you accept this certificate? [yes/no]: yes Dhaka(config)#
Manually authenticate the CA's public key by comparing the CA's certificate fingerprint as shown previously.
| | | Step 6. | Enroll with the CA.
You need to enroll the router with the CA to obtain a signed certificate for each of the router's RSA key pairs. A general purpose RSA key which is generated by default has one RSA key pair and therefore will need one certificate. However, special usage RSA keys have two RSA key pairs and will need two certificates. The Dhaka router has generated a general purpose key, so it will get one certificate from the CA server:
Dhaka(config)# crypto ca enroll vpn % Start certificate enrollment .. Dhaka(config)#
|
Troubleshooting Steps The PKI problem can be categorized as follows: More detailed discussions follow in the next sections. Certificate Enrollment Process Failure The certificate enrollment process may fail for two reasons, which are discussed in order: | | 1. | Incorrect CA URL for CA Server
You need to know the correct enrollment URL from CA administration to authenticate the CA and register with the CA server. You also need to find out from CA administration if the registration authority (RA) is used. Also find out how many certificates you are supposed to obtain. For CA mode only, you get the CA root certificate and the router identity certificate. However, for RA mode, you get the CA root certificate, the RA signature certificate, the RA encryption certificate, and the router identity certificate.
Example 6-20 shows a sample URL for the Microsoft CA server.
Example 6-20. URL (Identity) for Microsoft CA Dhaka(config)# crypto ca trustpoint DhakaPKI Dhaka(ca-trustpoint)# enrollment mode ra Dhaka(ca-trustpoint)# enrollment url http://10.1.1.10/certsrv/mscep/mscep.dll Dhaka(ca-trustpoint)# crl query ldap://10.1.1.11 Dhaka(ca-trustpoint)#
|
Example 6-21 shows a sample URL for the Entrust CA server.
Example 6-21. URL (Identity) for Entrust CA Dhaka(config)# crypto ca trustpoint DhakaPKI Dhaka(ca-trustpoint)# enrollment mode ra Dhaka(ca-trustpoint)# enrollment url http://10.1.1.12 Dhaka(ca-trustpoint)# crl query ldap://10.1.1.12 Dhaka(ca-trustpoint)#
|
Example 6-22 shows a sample URL for the Verisign CA server.
Example 6-22. URL (Identity) for Verisign Testdrive CA Dhaka(config)# crypto ca trustpoint DhakaPKI Dhaka(ca-trustpoint)# enrollment url http://testdriveIPsec.verisign.com Dhaka(ca-trustpoint)# crl option Dhaka(ca-trustpoint)#
|
| 2. | HTTP (TCP/80 Port) is blocked between the router and the CA server
The CA or the registration authority (RA) server should be accessible (TCP/80) from the router. If there is any firewall blocking between the router and CA or RA server, the certificate enrollment process fails, resulting in the message shown in Example 6-23.
Example 6-23. Sample Output When the Router Is Unable to Query the Servers Dhaka# debug crypto pki message Dhaka# debug crypto pki transaction CRYPTO_PKI: socket connect error CRYPTO_PKI: 0, failed to open http connection CRYPTO_PKI: 65535, failed to send out the pki message !or a Failed to query CA certificate message Dhaka#
|
|
Certificate Enrollment Is Fine but IKE Authentication Still Fails IKE authentication occurs on the fifth and sixth packets of IKE negotiation. In the fifth packet, the responder receives the ID payload, cert payload, and signature payload from the initiator. The responder verifies if the initiator's ISAKMP identity in the ID payload matches the identity in the certificate. It performs CRL checking, lifetime checking, and verifies the integrity of the certificate using the public key. It performs an IKE authentication integrity check by recalculating the hash in the signature payload. Checking PKI enrollment and certificate lifetime required multiple commands, the user had to correlate output: show crypto pki certificates show crypto pki timers show crypto pki trustpoints
Beginning with IOS Version 12.3(11)T, you can check configured trustpoints, enrollment status, and certificate lifetime with just the following command: show crypto pki trustpoint status
In addition to show commands, you need to run the following debug commands to troubleshoot PKI problem: debug crypto pki messages debug crypto pki transactions
Example 6-24 shows the debug output of a successful IKE negotiation with a certificate on the responder (the Doha router). Example 6-24. A Successful Authentication on the Doha Router (Responder) Doha# debug crypto isakmp Doha# debug crypto pki message Doha# debug crypto pki transaction ISAKMP (0:1): processing ID payload. message ID = 0 ISAKMP (0:1): processing CERT payload. message ID = 0 ISAKMP (0:1): processing a CT_X509_SIGNATURE cert CRYPTO_PKI: status = 0: poll CRL ldap search: server=10.1.1.11, base=CN = CRL1, OU = dohavpn, O = cisco, C = bangladesh, attribute=: scope=2, filter=cn=CRL1 CRYPTO_PKI: ldap_bind() succeeded. CRYPTO_PKI: set CRL update timer with delay: 66494 CRYPTO_PKI: the current router time: 10:00:28 UTC Apr 18 2004 CRYPTO_PKI: the last CRL update time: 03:28:42 UTC Apr 18 2004 CRYPTO_PKI: the next CRL update time: 04:28:42 UTC Apr 19 2004 CRYPTO_PKI: transaction GetCRL completed CRYPTO_PKI: Certificate verified, chain status= 1 ISAKMP (0:1): OU = dohavpn ISAKMP (0:1): processing SIG payload. message ID = 0 ISAKMP (1): sa->peer.name = , sa->peer_id.id.id_fqdn.fqdn = Doha.dohavpn.com ISAKMP (0:1): SA has been authenticated with 20.1.1.1 Doha#
|
IKE with a digital signature may fail for one of the following reasons: 1. | ISAKMP identity mismatch
The router assigns an FQDN to the keys and certificates used by IPsec. During IKE or Phase 1 negotiation the router checks the FQDN in the certificate. Therefore you have to use ISAKMP identity as hostname, instead of address on both of the routers. You will see the following message if there is an ISAKMP identity mismatch:
Dhaka# 3d15h: ISAKMP (0:162): processing a CT_X509_SIGNATURE cert 3d15h: %CRYPTO-6-IKMP_NO_ID_CERT_ADDR_MATCH: ID of 30.1.1.1 (type 1) an certificate addr with 30.1.1.1 Dhaka#
| | | 2. | Time and Date Mismatch
If the date and time of the clock on the router does not fall between the start and end dates on the certificates and the next/last update of the CRL, the following errors will be seen during Phase 1 negotiation:
CRYPTO_PKI: New CRL Not Yet Valid (router time not synched to CA?) CRL published: 16:17:34 PST Jan 10 2002
During certificate authentication, you will see the following message:
% Error in receiving Certificate Authority certificate: status = FAIL, cert length = 0
Use the show clock command in conjunction with the show crypto ca cert command as shown in Example 6-25, and compare the output to verify the validity of the certificate.
Example 6-25. show crypto ca cert Output Dhaka# show crypto ca cert Certificate Status: Available Certificate Serial Number: 3C9CC580 Certificate Usage: General Purpose Issuer: OU = vpn O = cisco C = Bangladesh Subject: ! Following is a FQDN Name Name: Dhaka.dhakavpn.com OID.1.2.840.113549.1.9.2 =<16> Dhaka.dhakavpn.com ! Distinguished Name (DN) OU = dhakavpn O = cisco C = Bangladesh CRL Distribution Point: CN = CRL1, OU = dhakavpn, O = cisco, C = Bangladesh ! Following lines show the Certificate Lifetime. The start and end date should be ! within the time shown in the show clock output from the router. Validity Date: start date: 19:29:42 UTC Mar 24 2004 end date: 19:59:42 UTC Mar 24 2006 Associated Trustpoint: DhakaPKI CA Certificate ... RA KeyEncipher Certificate ... RA Signature Certificate ... Dhaka#
|
If the certificate start and end dates are correct, make sure the router time is correct. This can be verified with the show clock command. You can correct the time with the following command:
Dhaka# clock set 20:10:01 19 march 2005
| 3. | Certificate lifetime issue
If the certificate lifetime expires, you need to re-enroll; otherwise, IKE negotiation fails. The re-enrollment process can be automated as shown in Example 6-26.
Example 6-26. Turning on Auto-enrollment Dhaka(config)# crypto ca trustpoint DhakaPKI Dhaka(ca-trustpoint)# enrollment url http://20.1.1.100 Dhaka(ca-trustpoint)# enrollment mode ra Dhaka(ca-trustpoint)# crl query ldap://20.1.1.100 Dhaka(ca-trustpoint)# serial-number none Dhaka(ca-trustpoint)# ip-address none Dhaka(ca-trustpoint)# password revokeme Dhaka(ca-trustpoint)# auto-enroll
|
| 4. | The router does not have the CRL configured
If you do not specify the CRL optional command on the router, it will check for the CRL during Phase one negotiation. If the CRL is not present, you will see the following errors:
CRYPTO_PKI: status = 275: failed to insert CRL
To correct the problem, define the CRL as optional as shown below:
Dhaka(config)# crypto ca identity vpn Dhaka(ca-identity)# crl optional Dhaka(ca-identity)#
|
Remote Access Client VPN Connection Remote Access VPN connections can be implemented with the IPsec protocol in conjunction with the AAA on the router. This section details the following topics: Configuration steps Troubleshooting steps
Configuration Steps Figure 6-7 depicts a typical Remote Access VPN connection on the Dhaka router with DNS and WINS Server. Figure 6-7. A Basic Remote Access VPN Setup on Dhaka Router 
Example 6-27 shows a Remote Access VPN configuration on the Dhaka router with the comments followed by the commands. Example 6-27. Configuration of Remote Access VPN !aaa commands enable user Authentication and Group Authorization with the local user ! database aaa new-model aaa authentication login userauthen local aaa authorization network groupauthor local ! This username and password should be used for Extended Authentication for the Remote ! Access VPN Connection username cisco password 0 cisco123 ! ! Phase 1 Policy with all parameters are defined here crypto isakmp policy 3 encr 3des authentication pre-share group 2 ! ! The Group name and password will be used on the VPN Clients are vpnclient/cisco123. ! Mode-configuration parameters that will be passed to the VPN clients are defined here. crypto isakmp client configuration group vpnclient key cisco123 dns 10.1.1.3 wins 10.1.1.2 domain cisco.com ! IP address to the VPN client will be assigned by this pool pool ippool ! This access-list will define the traffic that needs to encrypted and encapsulated by ! the IPsec tunnel. acl 100 ! Phase 2 of the IPsec tunnel negotiation is defined with the following command crypto IPsec transform-set myset esp-3des esp-sha-hmac ! ! A dynamic map is created to bind the transform set just created crypto dynamic-map dynmap 10 set transform-set myset ! ! This is the actual crypto map that will be applied to the interface. The following line ! turns on XAuth based on the AAA Authentication method defined in the beginning of the ! configuration using the local user database. crypto map clientmap client authentication list userauthen ! The following line turns on authorization crypto map clientmap isakmp authorization list groupauthor ! The following command tells the router to respond on the address assignment request crypto map clientmap client configuration address respond ! Dynamic map that was created earlier is applied to the actual crypto map crypto map clientmap 10 IPsec-isakmp dynamic dynmap ! This is the pool that was defined under the group configuration. The Pool is defined ! with the network 10.1.3.0/24 networks. Hence, when the VPN client tries to connect, it ! will be assigned one of the IP address listed in this pool. ip local pool ippool 10.1.3.1 10.1.3.254 ! ! This access-list is applied under the group configuration for split tunneling. As you ! can see from the following ACL, only traffic defined in this access-list will be ! encrypted and will go through the tunnel. access-list 100 permit ip 10.1.1.0 0.0.0.255 10.1.3.0 0.0.0.255 ! !crypto map is then applied to outbound interface interface FastEthernet1/0 ip address 20.1.1.1 255.255.255.0 crypto map clientmap ! rest of the irrelevant configuration is removed. Dhaka#
|
Troubleshooting Steps Troubleshooting a Remote Access VPN connection involves analyzing the log on both routers and on the VPN clients. Troubleshooting steps discussed for LAN-to-LAN are also applicable for the Remote Access VPN. This section examines a successful debug command output for a Remote Access VPN connection. Example 6-28 shows a Phase I negotiation for a Remote Access VPN in Aggressive mode: Example 6-28. Phase I Establishment for Aggressive Mode Dhaka# debug crypto isakmp Dhaka# debug crypto ipsec ! The following message indicates that this Router received an ISAKMP message from the ! VPN client on src port 500, dst port=500 ISAKMP (0:0): received packet from 200.1.1.1 (N) NEW SA ISAKMP: local port 500, remote port 500 . . . !Router is trying to match the received proposal # 1 with the configured proposal # 3 ISAKMP (0:10): Checking ISAKMP transform 1 against priority 3 policy ISAKMP: encryption 3DES-CBC ISAKMP: hash SHA ISAKMP: default group 2 ISAKMP: auth XAUTHInitPreShared ISAKMP: life type in seconds ISAKMP: life duration (VPI) of 0x0 0x20 0xC4 0x9B !Received Proposal is acceptable as shown in the following line ISAKMP (0:10): atts are acceptable. Next payload is 3 . . . !Since VPN client does Aggressive Mode, the new state is IKE_R_AM_AAA_AWAIT Old State = IKE_READY New State = IKE_R_AM_AAA_AWAIT Dhaka#
|
Example 6-29 shows x-auth completion on Phase I. Example 6-29. Showing X-Auth Completion for Remote Access VPN connection Dhaka# ! Router is requesting the VPN client for User Authentication as shown in the following ! line ISAKMP (0:10): Need XAUTH . . . ISAKMP/xauth: request attribute XAUTH_TYPE_V2 ISAKMP/xauth: request attribute XAUTH_MESSAGE_V2 ISAKMP/xauth: request attribute XAUTH_USER_NAME_V2 ISAKMP/xauth: request attribute XAUTH_USER_PASSWORD_V2 ! Router is receiving the X-Auth Attributes from the VPN Client . . . ISAKMP: Config payload REPLY ISAKMP/xauth: reply attribute XAUTH_TYPE_V2 unexpected ISAKMP/xauth: reply attribute XAUTH_USER_NAME_V2 ISAKMP/xauth: reply attribute XAUTH_USER_PASSWORD_V2 Dhaka#
|
Example 6-30 shows Mode Configuration for Phase I. Example 6-30. Showing Mode Configuration for Remote Access VPN Connection Dhaka# ! Received mode configuration request from the VPN client ISAKMP (0:10): checking request: ISAKMP: IP4_ADDRESS ISAKMP: IP4_NETMASK ISAKMP: IP4_DNS ISAKMP: IP4_NBNS ISAKMP: ADDRESS_EXPIRY ISAKMP: APPLICATION_VERSION ! Unknown Attr: is not an Error. It just means that router does not support this mode- ! config. Attribute requested by the VPN clientc ISAKMP: UNKNOWN Unknown Attr: 0x7000 . . . ! Router is sending the Mode-Configuration to the VPN client ISAKMP: Sending private address: 10.1.3.1 ISAKMP: Unknown Attr: IP4_NETMASK (0x2) ISAKMP: Sending IP4_DNS server address: 10.1.1.3 ISAKMP: Sending IP4_NBNS server address: 10.1.1.2 ISAKMP: Sending ADDRESS_EXPIRY seconds left to use the address: 86395 ISAKMP: Sending APPLICATION_VERSION string: Cisco Internetwork OperatingSystem Software IOS (tm) 7200 Software (C7200-IK9S-M), Version 12.2(8)T, RELEASE SOFTWARE (fc1) ISAKMP: Unknown Attr: UNKNOWN (0x7000) Dhaka#
|
Upon completion of Phase I negotiation, Phase II negotiation takes place as shown in Example 6-31. Example 6-31. IOS Showing Phase II negotiation of a Remote Access VPN Connection Dhaka# ! Router is checking and validating the IPsec proposals ISAKMP (0:11): Checking IPsec proposal 4 ISAKMP: transform 1, ESP_3DES ISAKMP: attributes in transform: ISAKMP: authenticator is HMAC-SHA ISAKMP: encaps is 1 ISAKMP: SA life type in seconds ISAKMP: SA life duration (VPI) of 0x0 0x20 0xC4 0x9B ISAKMP (0:11): atts are acceptable. Dhaka# . . . ! After validating the phase II, the IPsec SAs are created; One SA for inbound traffic ! and the other SA for the outbound traffic ISAKMP (0:11): Creating IPsec SAs inbound SA from 200.1.1.1 to 20.1.1.1 (proxy 10.1.3.1 to 20.1.1.1) has spi 0x962A493B and conn_id 2000 and flags 4 lifetime of 2147483 seconds outbound SA from 20.1.1.1 to 200.1.1.1 (proxy 20.1.1.1 to 10.1.3.1) has spi -2145675534 and conn_id 2001 and flags C lifetime of 2147483 seconds Dhaka#
|
To analyze the log on the VPN Client software, click on Log from the VPN Client window. Then click on Log Settings and set the logging level to High for all the classes. Then you can bring up Log Window by clicking on Log Window. Many troubleshooting techniques applied for the LAN-to-LAN tunnel also can be used for the Remote Access VPN connection. If Remote Access VPN connection fails, you need to analyze the debug on the Router, and the log on the Remote Access VPN client. Following are some of the important points to keep in mind while troubleshooting Remote Access VPN connection: VPN clients only propose Diffie-Hellman Groups 2 and 5. Configure DH group 2 or 5 (not 1) on the router. Configure isakmp identity hostname if rsa-sig is used as an IKE authentication method. AAA authorization must be enabled on the router, so that the router can accept and send mode-configuration attributes. If VPN Clients are behind the NAT/PAT device, configure NAT-T or the IPsec over TCP feature. If NAT-T or IPsec over TCP is configured, be sure the firewall between the VPN Client and the router is allowing UDP/4500 or TCP/10000.
|
