Examining VPN Protocols

PPTP and L2TP are the communication standards used to manage tunnels and encapsulate private data. Data traveling through a tunnel must also be encrypted to be a VPN connection. Windows Server 2003 includes both PPTP and L2TP tunneling protocols.

To establish a tunnel, both the tunnel client and tunnel server must be using the same tunneling protocol. Tunneling technology can be based on either a Layer 2 or Layer 3 tunneling protocol that corresponds to the Open System Interconnection (OSI) Reference Model. Layer 2 protocols correspond to the Data-link layer and use frames as their unit of exchange. PPTP and L2TP are Layer 2 tunneling protocols that encapsulate the payload in a PPP frame before it is sent across the Internet. Layer 3 protocols correspond to the Network layer and use packets. IPSec tunnel mode is a Layer 3 tunneling protocol that encapsulates IP packets in an additional IP header before sending them across the Internet.

Tunneling Within a Windows Server 2003 Networking Environment

For Layer 2 tunneling technologies, such as PPTP and L2TP, a tunnel is similar to a session; both of the tunnel endpoints must agree to the tunnel and must negotiate configuration variables , such as address assignment or encryption or compression parameters. In most cases, data transferred across the tunnel is sent using a datagram-based protocol. A tunnel maintenance protocol is used as the mechanism to manage the tunnel.

Layer 3 tunneling technologies generally assume that all the configuration issues are preconfigured, often by manual processes. For these protocols, there may be no tunnel maintenance phase. For Layer 2 protocols (PPTP and L2TP), however, a tunnel must be created, maintained , and then terminated .

After the tunnel is established, tunneled data can be sent. The tunnel client or server uses a tunnel data transfer protocol to prepare the data for transfer. For example, as illustrated in Figure 9.3, when the tunnel client sends a payload to the tunnel server, the tunnel client first appends a tunnel data transfer protocol header to the payload. The client then sends the resulting encapsulated payload across the internetwork, which routes it to the tunnel server. The tunnel server accepts the packets, removes the tunnel data transfer protocol header, and forwards the payload to the target network. Information sent between the tunnel server and tunnel client behaves similarly.

Figure 9.3. Tunneling the payload through a VPN connection.

Point-to-Point Tunneling Protocol

The Point-to-Point Tunneling Protocol (PPTP) is a Layer 2 protocol that encapsulates PPP frames in IP datagrams for transmission over the Internet. PPTP can be used for remote access and router-to-router VPN connections. It uses a TCP connection for tunnel maintenance and a modified version of Generic Routing Encapsulation (GRE) to encapsulate PPP frames for tunneled data.

Layer 2 Tunneling Protocol

Layer 2 Tunneling Protocol (L2TP) is a combination of the Point-to-Point Tunneling Protocol (PPTP) and Layer 2 Forwarding (L2F), a technology proposed by Cisco Systems, Inc. L2TP encapsulates PPP frames that are sent over IP, X.25, Frame Relay, and ATM networks. The payloads of encapsulated PPP frames can be encrypted and/or compressed. When sent over the Internet, L2TP frames are encapsulated as User Datagram Protocol (UDP) messages.

L2TP frames include L2TP connection maintenance messages and tunneled data. L2TP connection maintenance messages include only the L2TP header. L2TP tunneled data includes a PPP header and PPP payload. The PPP payload can be encrypted or compressed (or both) using standard PPP encryption and compression methods .

In Windows Server 2003, L2TP connections do not negotiate the use of PPP encryption through Microsoft Point-to-Point Encryption (MPPE). Instead, encryption is provided through the use of the IPSec Encapsulating Security Payload (ESP) header and trailer.

IP Security

IPSec was designed as an end-to-end mechanism for ensuring data security in IP-based communications. The IPSec architecture includes an authentication header to verify data integrity and an encapsulation security payload for both data integrity and data encryption. IPSec provides two important functions that ensure confidentiality: data encryption and data integrity. IPSec uses an authentication header (AH) to provide source authentication and integrity without encryption and the ESP to provide authentication and integrity along with encryption. With IPSec, only the sender and recipient know the security key. If the authentication data is valid, the recipient knows that the communication came from the sender and that it was not changed in transit.