VPN Protocols

Most VPN systems use one of two main protocols: Point-to-Point Tunneling Protocol (PPTP) or IP Security (IPSec). IPSec is the more widely used protocol for VPNs. Other VPN systems use varying degrees of both protocols. Additionally, Microsoft and Cisco jointly developed another VPN protocol called Layer Two Tunneling Protocol (L2TP).

PPTP

Point-to-Point Tunneling Protocol was originally designed and developed by a consortium of companies to encapsulate non-TCP/IP protocols such as IPX over the Internet using Generic Routing Encapsulation (GRE). Security in the protocol was added later.

GENERIC ROUTING ENCAPSULATION

Many protocols are currently available that are designed to encapsulate or hide one protocol in another, normally IP. GRE is designed to be more generic (hence the name) than these other protocols. As such, however, it may not fit the need of specifically encapsulating protocol X over protocol Y; instead, it is designed to be a simple, general-purpose encapsulation protocol that reduces the overhead of providing encapsulation. RFC 2784 describes GRE in detail. GRE is supported by the Linux kernel, however.

PPTP is very popular in many corporate environments, particularly those that are Windows-centric. For the most part, PPTP relies on a Windows NT host and includes clients for all flavors of Windows from Windows 95 on up, as well as the Mac OS. However, PPTP is considered to be less secure than IPSec, which is one of the reasons many people have switched to IPSec.

PPTP has many security concerns, including these:

• Poor encryption Nonrandom keys are generated, session keys are a hash of user passwords, and key lengths are too short.

• Bad password management Static passwords are easily compromised.

• Vulnerability to attacks Server spoofing attacks make this system vulnerable partly because packet authentication is not implemented. PPTP also is vulnerable to denial-of-service (DoS) attacks.

PPTP is still a widely used VPN protocol despite the noted security flaws and weaknesses. Because of its wide use, two main PPTP efforts are ongoing for Linux at this time; one is a server and the other is a client. More on this later.

IPSec

IPSec was designed with security in mind and is considered the de facto standard for secure private communication across public networks such as the Internet. As mentioned previously, IPSec has been included in the upcoming IPv6 implementation, but it also can be used in the current IPv4 standard.

IPSec provides data integrity, authentication, and confidentiality. All IPSec services are at the IP layer and provide protection for IP and upper-layer protocols. These services are provided by two traffic security protocols, the Authentication Header (AH) and the Encapsulating Security Payload (ESP). IPSec uses a cryptographic key-management system through the Internet Key Exchange (IKE) protocol and a managed Security Association (SA) connection system.

IPSec offers many advantages compared to other secure network access methods. One of the biggest advantages is that IPSec can work in the background without the user even knowing what's happening.

AUTHENTICATION HEADER

Normal IP packets consist of a header and a payload. The header contains both source and destination IP addresses that are required for routing. The payload consists of information that may be confidential. Headers can be spoofed or altered using a "man in the middle" type of attack. The AH actually signs the outbound packet digitally, verifying the identity of source and destination addresses and the integrity of the payload data.

AH provides only authentication, not encryption, and can be configured in one of two ways: in transport mode or in tunnel mode. Transport mode really applies only to the host implementation and provides protection for the upper-level protocols as well as selected IP header fields. Using transport mode, the AH is inserted after the IP header and before the upper-layer protocol (TCP, UDP, ICMP, and so forth), or before other IPSec headers that may already have been inserted.

The AH in tunnel mode protects the entire IP packet, including the entire inner IP header. As in transport mode, the AH is inserted after the outer IP header of the packet.

The AH is inserted after the IP header. In IPv4 implementations, the IP header contains the protocol number 51 (AH). The AH is shown in Figure C.1.

Figure C.1. The AH header format.

All fields in AH format must always be present and are included in the Integrity Check Value (ICV) computation.

ENCAPSULATING SECURITY PAYLOAD

Using ESP guarantees the integrity and confidentiality of the data in the original message by means of a secure encryption of either the original payload by itself or the combination of the headers and payload of the original packet.

ESP can be used in transport mode or tunnel mode, like AH, to provide encryption and authentication. Transport mode is applicable only to host implementations. It provides protection for the upper-layer protocols, but not for the IP header. For tunnel mode, the ESP is inserted after the IP header and before any upper-layer protocol such as TCP and UDP, or before any other IPSec headers that may already be inserted. In the current IPv4 implementation of TCP/IP, the ESP is placed after the IP header but before the upper-layer protocol. This makes ESP compatible with nonIPSec-aware hardware.

ESP's tunnel mode may be used in either hosts or security gateways. You must use ESP in tunnel mode if you deploy a security gateway. In tunnel mode, the inner IP header carries the proper source and destination addresses, whereas the outer IP header may contain distinct IP addresses such as addresses of security gateways. ESP protects the entire packet in tunnel mode, including the inner IP header. The position of the ESP packet is similar to that of transport mode.

ESP can use a wide variety of encryption algorithms, including 3DES, DES, CAST128, and Blowfish, to provide security services.

TRANSPORT AND TUNNEL MODES

In transport mode, the IPSec gateway is the destination of the protected packetsa machine acts as its own gateway. In tunnel mode, an IPSec gateway provides protection for packets to and from other systems.

The ESP is inserted after the IP header. In IPv4 implementations, the IP header contains the protocol number 50 (ESP). Figure C.2 shows an example of an ESP.

Figure C.2. The ESP format.

INTERNET KEY EXCHANGE

The Internet Key Exchange, or IKE, is an important part of an IPSec VPN. IKE itself is a hybrid protocol and allows for negotiation and authentication of keyed material for security associations in a protected manner. IKE consists of the following components:

• ISAKMP A framework for authentication and key exchange. ISAKMP is designed to be independent of the key exchange so that it can support many different key exchanges.

• Oakley A series of key exchanges, known as modes, and details of the services provided by each, such as perfect forward secrecy for keys, identity protection, and authentication.

• SKEME A key exchange technique that provides anonymity, reputability, and quick key refreshment.

IKE uses parts of these protocols to obtain authenticated keying material for ISAKMP and other security associations, such as AH and ESP. ISAKMP/Oakley is used by many VPN vendors to perform key exchange between IPSec devices.

ISAKMP/Oakley works in two phases when hosts need to set up connections between them. In the first phase, the hosts establish a secure channel between themselves by creating a bidirectional ISAKMP security association. This channel can then be used for the next phase to negotiate the required IPSec SAs. After this is done, IPSec can provide its security services with the established SAs. When the security associations expire, ISAKMP/Oakley is again used to negotiate new SAs.

SECURITY ASSOCIATIONS

To have secure traffic, there must be two security associationsone for each direction. The security association is essentially a one-way channel negotiated by the higher-level IPSec system and used by the lower level.

A security association is defined by three things:

• The destination IP address

• The protocol (AH or ESP)

• The security parameter index (SPI)

You cannot use both AH and ESP unless you create two or more SAs at each end of the VPN network. An SA can be used in transport mode or tunnel mode. A transport-mode SA is a security association between two hosts. A tunnel-mode SA is a security association applied to an IP tunnel. If either end of an SA is a security gateway, the SA is a tunnel-mode security association. Security association between two security gateways is always the tunnel-mode SA, just like an SA between a host and a security gateway.