Chapter 3: VPN Security

Virtual private network (VPN) connections are made across public networks such as the Internet. This is a simple statement, but it packs a lot of issues behind it. When going across a public network, there are several items that have to be dealt with to make sure that the security of your network is not compromised. For instance: How do I ensure that the person establishing a connection with my gateway is authorized to do so? How do make sure that there is no way for a hacker to capture the conversation and use it to gain information like user credentials and confidential information? How do I maintain control over what a VPN user accesses in the network once they have established communications? How do I know if their VPN client machine is not going to infect the network with a virus? Obviously, there are a lot of concerns if we are going to make a remote system “part of the network.” Therefore, attention must be paid to ensure that the VPN servers and the private data that is sent across a VPN connection are protected from malicious users. Security for Windows VPN connections is a combination of basic elements that are required (authentication, authorization, encryption, and packet filtering) and advanced features that provide additional protection (such as certificate- based authentication, network access quarantine control, and remote access account lockout).

Basic Elements of Windows VPN Security

In order for a VPN connection to be secure, it must provide the following:

  • Authentication security. Security credentials take the form of either a user name and password or a certificate. If you use the proper authentication security protocol (the different options are listed below), you can ensure that the confidential portions of the credentials (such as the password or the private key for a certificate) are never sent. Rather, the connecting VPN client provides proof of knowledge of the confidential credentials.

  • Authorization security. Authorization security ensures that the VPN client is allowed to make a VPN connection, and can provide a set of connection constraints such as maximum connection time, idle timeout, required authentication method, and so on. You can also apply IP filters based on a user’s Active Directory group membership so that the individual in question can only access the information that he is supposed to see. This allows for administrators to add extra security to remote-based users.

  • Encryption security. Before the data between a VPN client and VPN server is sent over the VPN connection, it is encrypted using an encryption algorithm and a secret key, which is known only to the VPN client and VPN server. Encryption provides data confidentiality; even if a copy of the packet is captured, it is not readable (except for the IP header) without the knowledge of the secret key. When using PPTP, the encryption is done with a password-based hash algorithm. When using L2TP/IPSec, certificates are used to set up an IPSec encrypted tunnel that all authentication and authorization processes can take place in. This is one of the advantages to using L2TP/IPSec–the entire transaction even before authentication happens occurs in an encrypted state.

  • Packet filtering. When you connect a VPN server to the Internet, the server and your private intranet are now exposed to attack. An Internet- based attacker can try to attack the VPN server by flooding it with various types of packets or try to access your intranet by using your VPN server as a router. To combat both of types of attacks, the Internet interface of the VPN server is configured with a series of IP packet filters that only allow VPN traffic. This is different than the internal IP filters that apply to a user’s authentication–this process makes sure that only authorized conversations will be accepted by the VPN server. This will ensure that Denial-of-Service attacks and internet hacks cannot affect operations.

Each of these basic elements of VPN security is discussed in further detail in the following sections.

Authentication Security

User authentication for both Point-to-Point Tunneling Protocol (PPTP) and Layer Two Tunneling Protocol (L2TP) connections is based on Point-to-Point Protocol (PPP) authentication protocols. Windows Server 2003 and Windows XP support the following PPP authentication protocols:

  • Password Authentication Protocol (PAP). PAP is a simple, clear-text authentication scheme. The NAS requests the user name and password, and PAP returns them in clear text (unencrypted). Obviously, this authentication scheme is not secure because a third party could capture the user’s name and password and use it to get subsequent access to the NAS and all of the resources provided by the NAS. PAP provides no protection against replay attacks or remote client impersonation once the user’s password is compromised.

  • Challenge-Handshake Authentication Protocol (CHAP). CHAP is an encrypted authentication mechanism that avoids transmission of the actual password on the connection. The NAS sends a challenge, which consists of a session ID and an arbitrary challenge string, to the remote client. The remote client must use the MD5 one-way hashing algorithm to return the user name and a hash of the challenge, session ID, and the client’s password. The user name is sent as plain text.

    CHAP is an improvement over PAP because the clear-text password is not sent over the link. Instead, the password is used to create a hash from the original challenge. The server knows the client’s clear-text password and can, therefore, replicate the operation and compare the result to the password sent in the client’s response. CHAP protects against replay attacks by using an arbitrary challenge string for each authentication attempt. CHAP protects against remote client impersonation by unpredictably sending repeated challenges to the remote client throughout the duration of the connection.

  • Microsoft Challenge-Handshake Authentication Protocol (MS- CHAP). MS-CHAP is an encrypted authentication mechanism very similar to CHAP. As in CHAP, the NAS sends a challenge, which consists of a session ID and an arbitrary challenge string, to the remote client. The remote client must return the user name and an encrypted form of the challenge string, the session ID, and the MD4-hashed password. This design, which uses the MD4 hash of the password, provides an additional level of security because it allows the server to store hashed passwords instead of clear-text passwords. MS-CHAP also provides additional error codes, including a password expired code, and additional encrypted client-server messages that permit users to change their passwords during the authentication process. In MS- CHAP, both the access client and the NAS independently generate an initial encryption key for subsequent data encryption by Microsoft Point-to-Point Encryption (MPPE). Therefore, MS-CHAP authentication is required to enable MPPE-based data encryption.

  • MS-CHAP version 2 (MS-CHAP v2). MS-CHAP v2 is an updated encrypted authentication mechanism that provides stronger security for the exchange of user name and password credentials and determination of encryption keys. With MS-CHAP v2, the NAS sends a challenge to the access client that consists of a session identifier and an arbitrary challenge string. The remote access client sends a response that contains the user name, an arbitrary peer challenge string, an encrypted form of the received challenge string, the peer challenge string, the session identifier, and the user’s password. The NAS checks the response from the client and sends back a response containing an indication of the success or failure of the connection attempt and an authenticated response based on the sent challenge string, the peer challenge string, the encrypted response of the client, and the user’s password. The remote access client verifies the authentication response and, if correct, uses the connection. If the authentication response is not correct, the remote access client terminates the connection.

    Using this process, MS-CHAP v2 provides mutual authentication—the NAS verifies that the access client has knowledge of the user’s password and the access client verifies that the NAS has knowledge of the user’s password. MS-CHAP v2 also determines two encryption keys, one for data sent and one for data received.

  • Extensible Authentication Protocol (EAP). EAP is a new PPP authentication protocol that allows for an arbitrary authentication method such as smart cards, token cards, or biometrics such as fingerprint scanners or retinal scanners. EAP is an IETF standard extension to PPP that allows for arbitrary authentication mechanisms for the validation of a PPP connection. EAP was designed to allow the dynamic addition of authentication plug-in modules at both the client and server ends of a connection. This allows vendors to supply a new authentication scheme at any time. EAP provides the highest flexibility in authentication uniqueness and variation.

    EAP is documented in RFC 2284 and is supported in Windows Server 2003, Windows XP, and Windows 2000. EAP allows for two-factor authentication– a practice that is highly recommended and considered a must by most security personnel. For example, a person’s username and password can be discovered by a hacker, and without two-factor authentication the hacker can now impersonate the user. If, on the other hand, the user is issued a smart card, the username and password is useless unless the physical card is in their possession as well, thus satisfying the two-factor parameters: “something you have and something you know.” EAP is a mandatory piece to making two-factor authentication work.

Authentication Security for PPTP Connections

Because encryption for PPTP connections is based on the use of MPPE, you must use an authentication protocol that generates MPPE keys as part of the authentication process: MS-CHAP, MS-CHAP v2, or EAP-TLS. The MPPE key is generated using the password in the MS-CHAP algorithms–therefore if you are using PPTP it is highly recommended that you apply “strong password” policies for the users. In the case of EAP-TLS, the user certificates can be used. EAP-TLS using smart cards or MS-CHAP v2 is highly recommended as they provide mutual authentication and are the most secure methods of exchanging credentials.

Authentication Security for L2TP/IPSec Connections

As stated earlier, authentication for L2TP/IPSec connections occurs at two different levels: the computer is authenticated, and then the user is authenticated. This allows for the IPSec tunnel to be established prior to the authentication phase of the connection being made, thus allowing for all communications to happen in an encrypted state.

Phase One: IPSec Computer Authentication

Mutual computer authentication of the VPN client and the VPN server is performed when you establish an IPSec ESP security association (SA) through the exchange of computer, also known as “machine”, certificates. IPSec Main Mode and Quick Mode negotiation occur, and IPSec SAs are established with an agreed encryption algorithm, a hash algorithm, and encryption keys. To use L2TP over IPSec, a computer “machine” certificate must be installed on both the VPN client and the VPN server.

Phase Two: L2TP User-Level Authentication

Now that the IPSec tunnel is established and everything is happening in an encrypted mode, the user attempting the L2TP connection is authenticated using PPP-based user authentication protocols such as EAP, MS-CHAP, CHAP, and PAP. Because IPSec encrypts the PPP connection establishment process, any PPP authentication method can be used–even PAP, which passes information in the clear, is now protected by IPSec. However, the use of MS-CHAP v2 or EAP-TLS is highly recommended because they provide mutual authentication. Since a certificate infrastructure was already required to provision the machine certificates, it should be simple to provision user certificates as well via auto-enrollment of Windows Server 2003 Active Directory and thus allow for the higher level of authentication with certificates.

Authorization Security

Authorization is the verification that the connection attempt is allowed. Authorization occurs after successful authentication. For a connection attempt to be accepted, the connection attempt must be both authenticated and authorized. It is possible for the connection attempt to be authenticated by using valid credentials, but not authorized. Usually this is because the Active Directory group that the individual belongs to does not have the right to VPN access of the network. Examples of this can be contractors or part-time employees who should not be accessing information unless they are being monitored by full-time employees. Another example can be with internal implementations for VPN where only members of Human Resources can access information on the protected network. In this case, the connection attempt is denied.

In the Windows Server 2003 family, authorization of VPN connections is determined by the dial-in properties on the user account and remote access policies. For more information, see Chapter 5, “Site-to-Site VPN Components and Design Points” or Chapter 8, “Remote Access VPN Components and Design Points,” depending on the VPN methods you are deploying.

Encryption Security

To ensure confidentiality of the data as it traverses the shared or public transit internetwork, it is encrypted by the sender and decrypted by the receiver. The encryption and decryption processes depend on both the sender and the receiver having knowledge of a common encryption key.

The length of the encryption key is an important security parameter. Cryptanalysis and computational techniques can be used to determine the encryption key, which require more computing power and computational time as the encryption key gets larger. Therefore, it is important to use the largest possible key size. Required key strengths are configured on the Encryption tab of the properties of profile for a remote access policy.

Encryption Security with MPPE

PPTP inherits the use of MPPE encryption from PPP (MPPE is also used for dial-up remote access or demand-dial connections), which uses the Rivest-Shamir-Adleman (RSA) RC4 stream cipher. MPPE is only available when either the EAP-TLS, MS- CHAP, or MS-CHAP v2 authentication protocols are used.

MPPE can use 40-bit, 56-bit, or 128-bit encryption keys. The 40-bit key provides backward compatibility with clients running Windows 98 or Windows Millennium Edition. By default, the highest key strength supported by the VPN client and VPN server is negotiated during the connection establishment process. If the VPN server requires a higher key strength than is supported by the VPN client, the connection attempt is rejected.

MPPE was originally designed for encryption across a point-to-point link, in which packets arrive in the same order in which they were sent with little packet loss. For this environment, the decryption of each packet depends on the decryption of the previous packet. For VPNs, however, IP datagrams sent across the Internet can arrive in a different order from the one in which they were sent, and a higher proportion of packets can be lost. Therefore, MPPE for VPN connections changes the encryption key for each packet. The decryption of each packet is independent of the previous packet. MPPE includes a sequence number in the MPPE header. If packets are lost or arrive out of order, the encryption keys are changed relative to the sequence number.

Encryption Security with L2TP/IPSec

Encryption is determined by the establishment of the Quick Mode or IPSec SA. The available encryption algorithms include the following:

  • Data Encryption Standard (DES) with a 56-bit key

  • Triple DES (3DES), which uses three 56-bit keys and is designed for high- security environments

Because IPSec was designed for IP internetworks where packets could be lost and arrive out of order, each IPSec packet is decrypted independent of other IPSec packets.

The initial encryption keys are derived from the IPSec main mode authentication process. For DES-encrypted connections, new encryption keys are generated after every 5 minutes or 250 megabytes of data transferred. For 3DES-encrypted connections, new encryption keys are generated after every hour or 2 gigabytes of data transferred. IPSec is an in-depth topic that we cannot possibly do justice to in this book, so for more in-depth information on IPSec and the encryption process go visit the following Web site:

Packet Filtering Security

A VPN server is an IP router, forwarding IP packets between interfaces (network adapters or logical interfaces). This makes it a very powerful entity on the network– it can propagate and route traffic to potentially anywhere on your intranet and out to the Internet if the proper precautions are not taken. To provide security, an IP router can allow or disallow the flow of very specific types of IP traffic. This capability, called IP packet filtering, provides a way for the network administrator to precisely define what IP traffic is received and sent by the router. IP packet filtering is an important element of connecting corporate intranets to public networks like the Internet.

IP packet filtering consists of creating a series of definitions called filters, which define for the router what types of traffic are allowed or disallowed on each interface. Filters can be set for incoming and outgoing traffic.

  • Input filters define what inbound traffic on that interface the router is allowed to route or process.

  • Output filters define what traffic the router is allowed to send from that interface.

Without a set of packet filters for PPTP and L2TP/IPSec traffic, the VPN server is vulnerable to various types of denial of service (DoS) attacks and, because the VPN server is a router, it might be possible for an attacker on the Internet to directly access private intranet resources. The filters will disallow any unauthorized communications and maintain the integrity of the internal resources by cutting off access to unauthorized sources.

PPTP Packet Filtering

A PPTP-based VPN server typically has two physical interfaces: one interface on the shared or public network like the Internet, and another on the private intranet. It also has a virtual interface connecting to all VPN clients–the virtual interface is created and maintained by the VPN server itself, so you don’t need to worry about setting this up. For the VPN server to forward traffic between VPN clients, IP forwarding must be enabled on all interfaces. However, enabling forwarding between the two physical interfaces causes the VPN server to route all IP traffic from the shared or public network to the intranet. To protect the intranet from all traffic not sent by a VPN client, PPTP packet filtering must be configured so that the VPN server only performs routing between VPN clients and the intranet and not between potentially malicious users on the shared or public network and the intranet. This is an important concept–there should be no reason for remote users to have to directly access each other’s client systems over the corporate network. Allowing such activity will 1) cause an undue amount of overhead on the VPN server, and 2) expose the internal resources to unwarranted access due to the enablement of IP forwarding. PPTP packet filtering can be configured on either the VPN server or on an intermediate firewall. If there is an intermediate firewall, you should configure PPTP packet filtering on both the VPN server and the intermediate firewall. For more information about PPTP packet filtering, see Appendix B.

L2TP/IPSec over IPSec Packet Filtering

Just as in PPTP-based VPN connections, the enabling of forwarding between the interfaces on the public or shared network and the intranet causes the VPN server to route all IP traffic from the shared or public network to the intranet. To protect the intranet from all traffic not sent by an L2TP/IPSec client, you must configure L2TP/IPSec packet filtering so that the VPN server only performs routing between VPN clients and the intranet and not between potentially malicious users on the shared or public network and the intranet.

L2TP/IPSec packet filtering can be configured on either the VPN server or on an intermediate firewall. If there is an intermediate firewall, you should configure L2TP/IPSec packet filtering on both the VPN server and the intermediate firewall. For more information about L2TP/IPSec packet filtering, see Appendix B.


Depending on your choices during the Routing and Remote Access Server Setup Wizard, the correct PPTP and L2TP/IPSec packet filters are automatically configured on the Internet interface.

Deploying Virtual Private Networks With Microsoft Windows Server 2003
Deploying Virtual Private Networks with Microsoft Windows Server 2003 (Technical Reference)
ISBN: 0735615764
EAN: 2147483647
Year: 2006
Pages: 128

Similar book on Amazon © 2008-2017.
If you may any questions please contact us: