Selecting Security Protocols

Table of contents:

When selecting security protocols, the first decision is which layer of the networking stack should be secured: the link layer, the network layer, the transport layer, the application layer, or some combination of all of them? 802.1X-based link-layer security and VPNs are used at different layers of the protocol stack, and are not mutually exclusive. Your choice of layer 2 security, layer 3 security, or both will depend on your goals and requirements. Both provide encryption to protect messages from eavesdroppers. Both can provide user authentication. 802.1X protects against many attacks by denying network access before authentication completes, and its authentication is much better thought-out than IPsec remote access authentication methods. 802.1X used in conjunction with improved Layer 2 encryption is likely to meet the security needs of many organizations. In simple terms, the security of an IPsec tunnel is greater than a WEP "session," even one rekeyed frequently using 802.1X protection.

Applying Security in the Protocol Stack

One of the biggest changes in the 802.11 landscape since the first edition of this book is the emergence of a reasonable security architecture in 802.11. First and foremost, the security protocols must keep the network secure. In wireless LANs, security is largely a function of authentication and encryption.

Compound binding vulnerabilities

Most traditional methods of user authentication are not secure. Over time, the resistance of authentication protocols to attack is increasing, but widely-deployed older methods will be used for many years to come. To secure older authentication protocols, it is common to provide some form of encrypted channel to protect the insecure method across an insecure network. The development of strong tunneling technologies such as the Transport Layer Security (TLS) and the IP Security protocols (IPsec) offers an attractive method for protecting older authentication protocols. Using a secure tunnel to protect a much weaker legacy method is referred to as compound authentication because the authentication is split into its major tasks of establishing a secure channel and performing user authentication. The most notable examples of compound authentication methods are the TLS tunneling methods used on wireless networks (TTLS and PEAP), as well as the eXtended Authentication (XAUTH) commonly used with IPsec. Designing cryptographically secure protocols is extremely demanding work, and it is easier and safer to use a widely deployed and analyzed protocol such as TLS or IPsec to provide the necessary cryptographic components.

In October 2002, however, research found that the design shared by most compound authentication methods is insufficient to prevent certain types of man-in-the-middle (MITM) attacks. One of the major problems is that the tunneled "inner" authentication method is not strongly associated with, or "bound to," the "outer" protective tunnel. Before exchanging the sensitive "inner" authentication data inside the tunnel, the protocols do not provide for a check that the authentication is occurring with the endpoint of the "outer" tunnel. The compound binding problem does not occur because of weaknesses within the tunnel protocols or inner methods, but rather in the way they are combined. Inner authentication protocols must prove that the endpoints of the inner tunnel use the endpoints of any previous outer tunnels.[*]

[*] IPsec's Quick Mode (phase 2) does this by including signature messages that rely on the results of the IKE (phase 1) exchange protecting it. XAUTH deliberately broke this model.

At this point, the vulnerabilities associated with weak compound bindings are largely theoretical. Successful use of these vulnerabilities would require active participation on the part of the attacker. To carry out a successful MITM attack, the attacker would need to insert a radio device into the target network, carry out the first phase of authentication with a legitimate authenticator, and then attract a client to steal credentials from. Success requires physical access to the network and facility, as well as numerous active radio transmissions.

XAUTH is no longer subject to active standardization efforts, and may not be fixed to address compound binding vulnerabilities. Compound binding problems in IPsec can be avoided through the use of digital certificates on every client. Wireless LAN protocols have not yet reached the final stage, and are in the process of being modified to address compound binding flaws.


An IPsec session probably provides superior encryption strength to any of the RC4-based frame encryption methods, and the age of the standards and most implementations can offer a high degree of trust, especially if static addressing and digital client certificates are used. If it has a weakness, it is that a network-layer protocol cannot provide security for the MAC layer. In most wireless networks secured by IPsec, attackers can easily obtain IP addresses and launch attacks against other network users or denial-of-service attacks against the VPN components because there is no authentication before the network protocol is enabled.

Security certifications

Depending on the environment, it may be necessary to use products that have passed security certification evaluations. As a rule, many government networks require the use of network products that have been evaluated. In the United States, this process is based on Federal Information Processing Standard (FIPS) 140-2; several major goverments have joined together to replace a patchwork of national standards with the Common Criteria evaluation. Great weight is often given to these security evaluations, but it is worth noting that many products that are certified under meaningful evaluation criteria often have to remove many ease-of-use features.

At this point, link-layer wireless security protocols cannot be given security FIPS 140-2 certification. Several components of the entire link-layer security system must be approved by the National Institute of Standards and Technology (NIST) before the entire system can be certified. For a current list of validated components, see The major components of a system that need to gain individual approval before the system can be approved are:

Encryption cipher

802.11i's CCMP uses AES, which is approved. RC4 is not approved and almost certainly never will be, which bars TKIP from certification.

Cipher mode of operation

The CCM mode, which is used by 802.11i's CCMP, was approved in NIST Special Publication 800-38C in May 2004.

Hash algorithms

The pseudorandom function that expands master keys into key hierarchies is based on HMAC-SHA1, which is approved.

Key wrap

Keys are sensitive material and need to be encrypted for transport across the network. RFC 3394 defines the NIST key wrap function, which has been approved.

Key derivation

In January 2005, NIST released implementation guidance ( According to the implementation guidance document, NIST is planning to release Special Publication 800-56, which will formally approve key derivation methods.

Even though there are no generally available commercial products that meet security standards, there is an interesting exception. Harris Corporation, a large government contractor (and, coincidentally, the initial developer of the 802.11 PRISM chipset), sells an 802.11 system called SecNet11 that uses a National Security Agency-certified encryption scheme. Buyers must be given approval by the NSA. The price is, not surprisingly, extremely high: over $3,000 per wireless LAN interface, and $1,600 per AP.

Network support

Aside from security itself, other factors influence the decision over how to apply security in the network stack. One of the largest nonsecurity considerations is related to client software. Supporting software is trying, even at the best of times. Working with components built in to the operating system may be a significant advantage. 802.1X-based systems have the advantage, now that 802.1X supplicants are built in to the recent versions of major operating systems. IPsec clients are generally specific to the vendor of termination devices. In both cases, client configuration is a one-time event. 802.1X clients may have slightly better control over when they need to run, provided the corporate SSID is different from other networks. If IPsec is already widely deployed, it may be much easier initially to treat the wireless LAN as a remote access network, especially since that will not require user retraining. Before choosing IPsec, however, ensure that there is enough capacity on the VPN termination device to handle LAN-speed access from the new users.

Support for mobility is likely to be an important consideration. To prevent IPsec tunnels from being torn down, some arrangement must be made to provide IP mobility across the network. Some refer to this task as the creation of a "mobile VPN." IP mobility can be provided by the network itself through the use of a single VLAN for all APs, or through overlay tunnels from every AP to a VPN server, or even 802.1X-based dynamic VLAN assignment. With the ability of 802.11i-compliant devices to move security contexts between APs, it probably has a slight edge for networks with highly mobile users.

Support for network applications may also lead towards a preference for one protocol. Non-IP protocols are not supported by IPsec. Networks that still use IPX or AppleTalk may not have a choice. IP multicast is not supported by IPsec, although it is supported by link-layer security solutions as a series of multicast frames. IPsec is also likely to be too slow for applications that require low latency, such as voice transmission; I am also unaware of any existing phone that supports IPsec, or any plans to introduce an IPsec 802.11 phone.

Different applications may also require protection over different domains. In many cases, it is enough to protect communications over only the most vulnerable part of the path, and radio link encryption is sufficient. In some cases, it will be necessary to use site-to-site encryption to provide security all the way to the border of another network. The distance over which the data needs protection is a function of the security of the network past the radio. If the access points are connected to the same backbone as the destination traffic, radio link encryption is probably sufficient. If the traffic must travel across untrusted networks after leaving the radio, some form of network-or higher-layer encryption is required.

Table 22-2 summarizes many of the factors that come into play when deciding where to implement authentication in your network.

Table 22-2. Summary of major factors in authentication layer decision

Link layer (802.1X)

Network layer (VPN)

Integrated operating system components

May already be deployed with existing capacity on VPN concentrator

Protection of layer 2 network from attack

More familiar for users; less retraining is required

Better mobility architecture

Protects beyond the radio link

Support for IP multicast and non-IP protocols

Security certifications require VPN; e.g., FIPS

Fast handoff support for voice



Choose Authentication

The choice of authentication technology often depends on the way it will interface with the existing user database. As a first step, identify where the user accounts currently live. Are they in a Windows NT Domain or an Active Directory? Do they live in an LDAP database, or a Kerberos realm? Once you know where to find the users, the choice of system to authenticate them is usually straightforward.

Choosing an EAP method

Practically speaking, it is necessary to use an EAP method based on TLS to provide strong cryptographic keys for frame encryption. This constraint leaves us with three choices: EAP-TLS, PEAP, and TTLS. EAP-TLS is difficult to use because it requires certificates for every client. Without a PKI already installed, using EAP-TLS may be considered a practical impossibility. PEAP and TTLS are quite similar, so the question becomes which one to use. Although the protocols are similar, they support different authentication methods inside the encrypted tunnel. Let the interface to existing user databases make the decision about the authentication method. How the user credentials are accessed for verification determines your EAP method. Three of the authentication methods will work with most authentication databases, with other methods available to cover the remaining few cases.

PAP, the Password Authentication Protocol, transmits the username and password without any encryption. It simply submits a username and a password to the authentication server, and the authentication server compares the submitted password to a stored copy. Without any encryption, PAP should not be used across a network connection that does not provide privacy protection. (When PAP is used on wireless LANs, it is used in an encrypted channel, and therefore safely.) PAP is not an EAP method, and is therefore not supported by PEAP. The most common use of PAP is with user databases that one-way encrypt passwords, such as LDAP directories or the Unix /etc/password file. Rather than store the password itself, or the password in some recoverable form, some systems store a one-way hash of the password such as the SHA-1 or MD5 hash. When a client user or program wishes to prove its identity, the password is processed by the one-way hash, and the resulting hash is compared to stored hash. PAP is commonly used with token cards because the alphanumeric code from the token can be submitted along with a user name. Some supplicants have a "TTLS/Token Card" method that is TTLS/PAP, but without any caching of the password, since token codes expire. Because nearly every legacy authentication database supports password authentication, TTLS/PAP is like Type O blood. If in doubt, it is generally a safe bet.

Microsoft CHAP version 2 (MS-CHAP-V2) performs a challenge/response handshake. The client and server share a secret, and challenge each other to perform computations on challenge values with the secret. The secret is the MD4 hash of the user password. Although it is not reversible, the hash value is used as the secret, which makes it a "password equivalent." MS-CHAP-V2 is widely supported by Microsoft clients, and is commonly supported and used as an inner authentication method with PEAP. If you have an existing authentication database stored in a Windows domain or Active Directory, MS-CHAP-V2 is a logical choice for the inner authentication method. It can either be used "as-is" within TTLS, or it can be used as an EAP method with additional header information (EAP-MSCHAP-V2).

The third common inner authentication method is EAP-Generic Token Card (EAP-GTC). EAP-GTC was originally designed to take a username and token code. It does not provide any handshaking or privacy protection. Given its initial intent of shuttling a nonreusable code to a server, the lack of privacy protection is understandable. However, it has seen recent adoption as an EAP method that allows the submission of a username and associated authentication credentials. No EAP method allows the submission of username and password pairs directly, but there is nothing in EAP-GTC that prevents it from being used as a password submission protocol. Nothing, other than implementation, prevents EAP-GTC from being used as a PAP-like authentication method. As an EAP method, EAP-GTC can be used with either TTLS or PAP.

Several other inner authentication methods are available, including the Challenge Handshake Authentication Protocol (CHAP), Microsoft CHAP, version 1 (MS-CHAP), and EAP-MD5 Challenge (EAP-MD5). CHAP and Microsoft CHAP are only used by older authentication databases. They should only be used when the existing authentication database does not support one of the newer methods. Both can only be used with TTLS, since neither is an EAP method. EAP-MD5 does not provide security services. It is used for 802.1X on wired networks, but almost never on wireless because of the lack of eavesdropping protection.

It is likely that one of the three inner authentication methods shown in Table 22-3 will work with your user database. Based on the inner authentication method, choose either PEAP or TTLS. The choice of inner method may also depend on the TTLS or PEAP implementation that you consider. The standards allow EAP-GTC to be used within PEAP, but it is not supported by the built-in Windows supplicant.

Table 22-3. Summary of common EAP methods

Inner authentication method

Outer EAP method

Type of account or user database



TTLS only

One-way hash systems (most LDAP directories, Unix password format), token cards

Probably the second most common inner authentication method. Most universal; generally works with any database.



Windows user accounts (NT domains, Active Directory) or anything that stores an MD4 hash

Most common form of inner authentication due to the prevalence of Windows


TTLS or Cisco PEAP

Same as PAP

Only supported by Cisco supplicant on Windows

After selecting both the EAP method and inner authentication protocol, build the required interface. To work with a set of Windows user accounts, the RADIUS server must access the MD4 hashes of the user passwords. If made a member of a domain, Microsoft's Internet Authentication Server can pull user credentials out of a domain system without additional configuration. Third-party RADIUS servers may need to be installed on domain controllers to access the user account information. Any domain that is trusted can also be accessed through the domain protocols.

LDAP directories are increasingly popular, especially at large sites that require scalability. LDAP directories may be accessed in multiple ways. Some directories may offer RADIUS frontends. Wireless LAN devices can speak directly to the RADIUS frontend, or to the directory's RADIUS server through an alternative RADIUS server performing RADIUS proxy functions. Many RADIUS servers are also capable of working as an LDAP client. When provided with directory credentials, the RADIUS server is able to bind to the directory and validate user credentials directly. The advantage of binding to the directory is that it is usually possible to secure directory access through SSL, which further protects the authentication back-end connections. Less functional RADIUS servers may interface with LDAP by attempting to bind to the directory using the LDAP credentials supplied by the client and checking for success. However, by performing a bind and looking only for success, the RADIUS server is unable to access any user metadata from the directory. If the directory stores privilege information that can be used by the wireless LAN, it is desirable to have the RADIUS server bind to the directory so that the user authorization information can be provided to the wireless devices.

Kerberos servers are popular in many academic environments, and a few RADIUS servers can check user credentials against Kerberos. These RADIUS servers attempt to obtain a ticket from the Kerberos server; if the ticket request is successful, the user is allowed access. However, Kerberos tickets are bound to IP addresses, and are not useful for client authorization.

Authentication architecture

To be successful, an authentication system must interface with the existing user accounts. In many cases, the user account system is small and confined, and the main problem the network administrator has is determining how to build the interface between the wireless LAN and the user account database. Many organizations are large enough, or perhaps divided enough, for the responsibility for maintaining user accounts to be distributed across geographic or administrative boundaries. Users are not members of rival departmental armies, however, and may need to move across geographic or administrative boundaries in the course of work. The end goal of the network design should be to facilitate user mobility and productivity across boundaries. (If you like, call it the Schengen design criterion for distributed wireless LANs.) This design is often found on university campuses, where individual schools, departments, or facilities may run their own networks.

Figure 22-1 shows a portion of a made-up university campus, with five networks: the campus library, a central IT services department that handles networking for common areas and departments that do not need to run their own networks, a school of engineering that prides itself on running an independent network, the campus medical center, and the business school. Each of these departments maintains its own user database, in whatever form they prefer. For wireless LAN access, each of the databases is provided through a RADIUS frontend. In the figure, the user database and RADIUS frontend are shown symbolically.

In designing the authentication architecture, the goal is to provide an authentication facility that allows a single user account to be used across all campus networks. For example, anybody entering the library, whether student or staff, should be able to use the library's network based on the credentials on their home network. If the RADIUS servers in each department have trust relationships and are able to refer authentication requests to each other, only one account is necessary. Accounts are assigned to RADIUS realms, which allow the routing of requests to a defined server. The library's RADIUS server knows that requests for user@med must go to the medical center's RADIUS server. If that server accepts the user, access is granted.

Trust relationships between networks are a key component of the architecture. Just as in the cellular world, to get network connectivity anywhere, it is necessary to have a roaming agreement between your "home" network and the network you are "visiting." Network managers depend on the trust because they are granting access to users authenticated by a third party. In a campus environment, the roaming agreeement might be informal. On a larger scale, formal, legally binding agreements may be required. EduRoam, a consortium of European universities, is building a RADIUS mesh on a continental scale, and does require signed formal agreements to join the mesh.

Accounting may also be helpful in this type of environment. Campus libraries devote budget dollars to the acquisition of scholarly material, not to the building of

Figure 22-1. Mesh authentication architecture

advanced network services. Through accounting records, a net provider of service (such as the library) can establish that network services are being consumed by users from other networks, which may assist in funding network expansion where service is useful, as opposed to where there is money.

Choose Encryption

After selecting an authentication method, it is usually straightforward to choose a link-layer encryption method because you will be picking the strongest protocol supported on most hardware. Configuration is almost identical between the protocols, and the functionality from the user perspective is identical.

Most of what can be said about static WEP has already been said. Do not use it unless you have no alternative. Nearly all general-purpose devices now support better protocols, leaving only application-specific devices like 802.11 phones or barcode scanners in security limbo. Depending on your security requirements, it may be


After identifying users and validating identity, it may be desirable to further restrict network access. Restrictions may be enforced in many different ways, but two of the typical policy tools are packet filtering through access control lists, and VLAN assignment. In some ways, the latter is a handy way of applying basic rules to user groups by taking advantage of existing packet filters between VLANs on your network.

Often, the user account will contain some piece of metadata that indicates what sort of authorization should be applied. That identifier is often indicative of an organizational role. Roles must be translated into concrete action, such as "put all users from Group A on to VLAN A" or "apply Filter B to devices in Group B."

Identity and role live with the user account, and that data is typically translated into restrictions by both the RADIUS server and the wireless LAN device itself. Based on the role, the RADIUS server may pass some set of restricting attributes. RADIUS attributes may also trigger further, more detailed actions from the wireless LAN device. As an example, the RADIUS server may specify that a filter should be applied to traffic from a particular device, but it is almost certainly up to the wireless LAN hardware to define the specifics of that filter.

necessary to shut down deployments of telephony or automation if the security risks cannot be adequately addressed through other means, or if a risk analysis shows that the threat is large.

Using WEP with dynamically derived keys is significantly better because a frequent rekeying interval can mitigate many of the attacks against WEP discussed in Chapter 5. Dynamic WEP should be considered the minimum acceptable level of security for user payload data on general-purpose computing devices. In many cases, manufacturers had to patch drivers, but most drivers from mid-to-late 2002 onwards have support for dynamic keys.

Because of the lingering flaws in WEP, even when used with dynamic keys, TKIP is the target level of security for most networks being built as this book was written. Hardware support requirements for TKIP are identical to WEP, so nearly every 802.11 device has the required hardware. Software support or TKIP is also required, but is increasingly common. TKIP and WPA are built in to the most recent versions of Windows (XP) and MacOS (10.3) WPA is not yet widely available on Linux because the driver framework does not yet have the required level of integration with the supplicant software, although the enhancements are expected in the future.

Most organizations should target TKIP for deployment while making plans for migration to enhanced protocols. Consider dynamic WEP a minimum requirement.

When security is of the utmost importance in link-layer encryption, CCMP is the clear choice. TKIP requires protocol "countermeasures" to detect keys under attack and mitigate the threat. Countermeasures are an admission that there is a fundamental limit on the strength of the protocol. Security does come at a cost, however. CCMP has the most stringent hardware and software support requirements of any of the link-layer encryption protocols. It is only available for use on wireless interfaces that can provide AES hardware assistance. Nearly all recent cards do, but many older cards do not. If you have a significant base of older cards without AES hardware support, the upgrade to AES-capable hardware may be an expensive and daunting challenge.

CCMP is likely to offer the best security. However, it also has the most stringent hardware and software requirements. Many laptop manufacturers choose the cheapest chipsets available, and thus may not support CCMP without external cards.


Multiple SSID support

The 802.11 security protocol architecture allows the simultaneous use of multiple cryptographic protocols on the same AP. Clients are free to use any supported protocol. Encryption of unicast frames is typically done using the strongest protocol supported by a given system, while encryption of frames to group addresses uses the strongest protocol supported by all systems. If an AP has a group of associated stations that support WPA, but there is one station that only supports dynamic WEP, the group key will be dynamic WEP.

The standards do not limit what may be supported, but individual implementations may have problems mixing cryptographic types. One of the most common limitations is that most client software does not support using a CCMP key for unicast frames while simultaneously using a non-CCMP encryption type (TKIP, dynamic WEP, or static WEP) for group frames. Driver support for this type of mix may come in the future. At the time this book was written, supporting CCMP was generally done by providing an additional SSID. One supports the RC4-based encryption protocols, while a second network supports only CCMP. Unfortunately, using two SSIDs does increase the risk of user confusion because the SSID to attach to is a user-controlled parameter.

Client software limitations prevent simultaneous use of CCMP and RC4-based encryption on the same SSID. To use both at the same time, you must use an 802.11 AP that supports multiple SSIDs.

Introduction to Wireless Networking

Overview of 802.11 Networks

11 MAC Fundamentals

11 Framing in Detail

Wired Equivalent Privacy (WEP)

User Authentication with 802.1X

11i: Robust Security Networks, TKIP, and CCMP

Management Operations

Contention-Free Service with the PCF

Physical Layer Overview

The Frequency-Hopping (FH) PHY

The Direct Sequence PHYs: DSSS and HR/DSSS (802.11b)

11a and 802.11j: 5-GHz OFDM PHY

11g: The Extended-Rate PHY (ERP)

A Peek Ahead at 802.11n: MIMO-OFDM

11 Hardware

Using 802.11 on Windows

11 on the Macintosh

Using 802.11 on Linux

Using 802.11 Access Points

Logical Wireless Network Architecture

Security Architecture

Site Planning and Project Management

11 Network Analysis

11 Performance Tuning

Conclusions and Predictions

802.11 Wireless Networks The Definitive Guide
802.11 Wireless Networks: The Definitive Guide, Second Edition
ISBN: 0596100523
EAN: 2147483647
Year: 2003
Pages: 179
Authors: Matthew Gast © 2008-2020.
If you may any questions please contact us: