IPSec

In Chapter 7, we discussed many of the characteristics of IPSec. Before we examine how IPSec can be implemented optimally, let's review the three types of IPSec architectures:

• Host-to-host The entire connection between the client and the server is encrypted. This is comparable to the encryption that standard SSH or SSL connections provide.

• Host-to-gateway The entire connection is encrypted except for the portion between the gateway and the remote server. The gateway is usually located on the perimeter. Host-to-gateway provides protection similar to SSH or SSL tunneling from a host to a remote SSH or SSL server or similar to using an SSL proxy server.

• Gateway-to-gateway The connection between the two gateways is encrypted, but the connections from the client to the client-side gateway and from the server to the server-side gateway are unencrypted. Both gateways are typically located on their respective network perimeters. A gateway-to-gateway IPSec VPN provides similar encryption to SSH or SSL tunneling between a client gateway and a remote SSH or SSL server.

What's the difference between using SSH or SSL and using IPSec? Although SSH, SSL, and IPSec might provide similar encryption capabilities, they are different from a functional standpoint. For every additional application you use through SSH or SSL, you have to establish additional connections and tunnels. IPSec makes one connection from a client to the remote VPN gateway or host, and it tunnels all application traffic through that one connection. Certain IPSec architectures can also conceal IP addresses, which is a significant security consideration in some environments. From a perimeter defense perspective and from a usability perspective, implementing VPNs using IPSec instead of SSH or SSL has many advantages. Let's look at how IPSec-based VPNs fit into perimeter defenses.

IPSec Client Integration

Most operating systems in use today include native IPSec clients, although some still require a separate IPSec client program to be acquired, installed, and configured. Some organizations also choose to use IPSec clients other than those built in to their systems; this is most often done to take advantage of additional features offered by the clients or to achieve full interoperability with a certain IPSec gateway (that is, using the same vendor for both the IPSec clients and IPSec gateway). Besides the additional time and resources needed to deploy third-party IPSec clients, such software also modifies the operating system's networking functions, which can cause operational problems.

On Windows systems, most nonnative IPSec clients fall into one of two categories:

• Clients based on shim technologies actually add a new layer between the existing network adapter and the TCP/IP protocol that is normally bound directly to the network adapter. This new layer is responsible for processing all traffic and implementing IPSec for all appropriate traffic. Because the shim is part of the existing network configuration, no routing changes are necessary for it to work properly.

• Clients that create a new network adapter, in addition to existing network adapters. Because this IPSec-specific network adapter is separate from the regular network components, the host requires routing changes so that traffic that needs IPSec processing goes through the new adapter and non-IPSec traffic goes through the other adapter.

For UNIX-based IPSec clients, multiple implementation methods are available. IPSec support can be added directly to the kernel, added as a new device driver that is recompiled into the kernel, or added as a loadable kernel module. Examples of free UNIX-based IPSec clients include Openswan (http://www.openswan.org/) and strongSwan (http://www.strongswan.org/).

As already mentioned, IPSec clients sometimes require routing changes to be made on the host. This is particularly true when a client needs to contact internal hosts and external hosts; only the traffic destined for the internal hosts must be handled using IPSec, although all the traffic could be. This isn't just because of the way the client software works; often it is due to the organization's security policy. It might be desirable to route all traffic through the VPN connection and then permit the remote VPN gateway to route traffic to external or internal hosts as appropriate. Of course, this causes a significant performance hit as compared to allowing the VPN client to make direct requests to other external hosts without utilizing the VPN connection to do so.

There is one other important point to know regarding IPSec client software. If you are making a connection between a single external host and a VPN gateway, you should configure the client software so that other hosts cannot use the tunnel. If you want to connect a remote network to your VPN gateway, you have to configure the IPSec software on the client side as a gateway, of course, to pass through traffic only from the authorized hosts on that local network.

IPSec Server Integration

IPSec servers can be deployed to different types of devices. Chapter 12, "Fundamentals of Secure Perimeter Design," contains a good discussion of VPN basics. To quickly review, the three most commonly used systems are as follows:

• VPN concentrators These dedicated boxes are used solely for VPN functions. Besides handling the establishment, maintenance, and termination of VPN connections, they might also perform functions such as firewalling, packet filtering, and Network Address Translation (NAT). The advantage of using a concentrator is that it is a single-function device, dedicated to VPN.

• Firewalls Many firewalls also provide support for IPSec. Using a firewall for VPN functionality is possible as long as the VPN overhead does not adversely affect the firewall's other operations. This solution is generally less expensive than a dedicated VPN concentrator.

• Routers Some routers have IPSec capabilities. The advantage of this is that a VPN can be established between two IPSec-capable routers, which provides an inexpensive gateway-to-gateway VPN.

Note

Although most devices implement IPSec according to standards, some have proprietary IPSec implementations that deviate from standards. If a VPN server runs a proprietary IPSec implementation, its users might be required to use a particular IPSec client, particularly in order to take advantage of proprietary features.

IPSec Perimeter Defense Adjustments

IPSec requires some interesting changes to perimeter defenses. Encapsulating Security Payload (ESP) mode uses IP protocol 50, whereas Authentication Header (AH) mode uses IP protocol 51. The Internet Key Exchange (IKE) negotiation uses UDP port 500. However, IPSec implementation can result in other things as well. NAT is often incompatible with IPSec. Because AH mode makes authentication value calculations based on the entire packet, which includes the source and destination IP addresses, any NAT must occur before IPSec is used on the packets. ESP mode does not have the same problem because it does not include the entire header when it makes its authentication value calculations. In general, if you want to use NAT, you should use ESP to provide authentication instead of ESP. However, there are still cases where ESP and NAT do not work well together, such as when a NAT mapping times out, causing the port used by IKE to change.

This becomes more complicated when Port Address Translation (PAT) is used instead of NAT. PAT relies on the use of port numbers. Remember, these are TCP and UDP port numbers, stored within the TCP or UDP payload portion of the packets. If the payload is encrypted, the port numbers are encrypted too, and the PAT devices are unable to process the packets because they cannot access the port number. Even if the payload is not encrypted, the structure of IPSec packets is different from that of non-IPSec packets, and many PAT devices can't correctly parse the IPSec packet structure.

To resolve conflicts between IPSec and address translation, some IPSec implementations now support a feature called NAT Traversal (NAT-T). If both endpoints state during the IKE negotiation that they support NAT-T, they next check to see if a NAT or PAT device between them is altering either of their IP addresses or source ports. If address translation is in use, the endpoints move their IKE negotiations from UDP port 500 to 4500 and wrap all their ESP packets within UDP packets. Known as UDP encapsulation, this separates the IPSec information from the new outer UDP header, which is subsequently manipulated by the NAT or PAT device without any impact to the IPSec headers. Unfortunately, because standards for NAT-T are still not finalized, there may be interoperability problems between different types of endpoints.

Note

VPN passthrough refers to the concept of successfully allowing VPN traffic to go through a perimeter defense device, even when that device performs NAT, PAT, or other actions that could adversely affect VPN traffic. VPN passthrough is often achieved by allowing VPN traffic to bypass functions such as NAT.

Whether you are implementing IPSec services on a VPN concentrator, a firewall, or a router, you have several options on where to place the IPSec services. This is discussed in detail in Chapter 12. Likely places for standalone VPN servers are on your perimeter, such as your DMZ or a screened subnet, and in parallel with your Internet firewall. Your design decision also depends on which IPSec architecture you want to implement.

IPSec Architectures

As we discussed earlier in this chapter, three types of IPSec architectures exist. Each is appropriate to meet certain types of needs.

• Host-to-host is most appropriate when external users need to access a single host. This requires software installation and configuration on the target host as well as each client that needs to access that host. Of course, installation is not necessary if the host operating systems have built-in IPSec support.

• Host-to-gateway is the best option for remote access, when many external users need to access a variety of internal hosts and resources. Each user's system must be IPSec-enabled (if it is not already) and configured appropriately. On the remote network, only the gateway needs to be configured; no changes are necessary for each host on the network that the external users will contact.

• Gateway-to-gateway is most commonly used to provide VPNs between separate external networks, such as two business partners, or to provide VPN capabilities within a single network, also known as an intranet VPN. The biggest advantage of this architecture is that no changes need to be made to the VPN clients or the hosts they contact.