6.6 Certificate-based authentication

Certificate-based authentication is based on the SSL and TLS protocols. SSL stands for the Secure Sockets Layer protocol. It is a security protocol operating at the transport layer of the OSI networking stack. SSL was initially developed by Netscape. The latest SSL specification (3.0) can be down- loaded from http://wp.netscape.com/eng/ssl3. In 1999 SSL was standardized by the IETF in RFC2246 under the name Transport Layer Security (TLS) protocol (more information is available at http://www.ietf.org/rfc/ rfc2246.txt). TLS is also referred to as SSL version 3.1. A great book revealing all the nuts and bolts of both protocols is Eric Rescorla’s book, SSL and TLS: Designing and Building Secure Systems (Addison-Wesley, 2001).

SSL/TLS can provide the following security services:

• Server authentication: SSL/TLS uses X.509 server certificates to authenticate the Web server.

• Data confidentiality and integrity services: SSL/TLS always provides channel encryption services.

• Optional client authentication: SSL/TLS uses X.509 client certificates to authenticate the clients.

The last one is the most interesting one in the context of this chapter on IIS authentication methods. Contrary to the methods discussed earlier, certificate-based client authentication does not use the HTTP AUTH headers. Note that SSL can provide these authentication, confidentiality, and integrity services to a wide range of application-level protocols: not only HTTP but also including SMTP, NNTP, and so forth.

The SSL/TLS protocols are built on symmetric and asymmetric cryptographic protocols (also known as public key crypto) and X.509 certificates. For more information on X.509 certificates, see Chapter 13 on public key infrastructures (PKI).

When users connect to Web pages that are secured using SSL/TLS, they must use a secure URL that begins with https://. The use of https tells the browser it should try to establish a secure SSL connection with the Web server. By default these secure connections are made over port 443. In Internet Explorer a small lock symbol shows up at the bottom of the screen to show the user that they are connected over a secure SSL/TLS connection (illustrated in Figure 6.16). If you move over the lock symbol with your mouse pointer, you can see the SSL encryption strength that has been used. If you double-click the lock symbol, you will see the properties of the Web server’s SSL/TLS certificate. If something is wrong with the Web server’s certificate, the lock symbol will be covered with an exclamation mark.

Figure 6.16: Internet Explorer SSL/TLS lock symbol.

In short, this is how an SSL/TLS connection is set up between a browser and a Web server:

• The browser connects to the Web server using a secure URL (https://).

• The Web server sends the browser the server certificate (containing its public key).

• The browser and Web server negotiate the encryption level to be used.

• The browser encrypts a session key (to be used for SSL channel encryption) with the Web server’s public key and sends the encrypted blob to the Web server.

• The Web server decrypts the encrypted blob using its private key.

• The browser and Web server exchange data—the data are secured using the session key.

6.6.1 SSL setup

The use of SSL/TLS requires X.509 certificates. You always need a server certificate, and if you also want client certificate-based authentication, you will also need client certificates. Setting up SSL in IIS 6.0 typically includes the following steps:

• Generate a server certificate request file.

• Submit the request file.

• Generate the server certificate.

• Install the server certificate on the Web site.

• Configure SSL on the Web site.

• Optionally, generate, acquire, and install client certificates.

• Install the issuing CA’s certificate in the browser’s trusted root certificate store.

In this chapter, only SSL-specific topics are discussed; for general information about certificates, public key infrastructures (PKIs), and the Windows Server 2003 PKI software, see Chapters 13, 14, and 15.

The easiest way to generate a server certificate request file is to use the IIS Web server certificate wizard (illustrated in Figure 6.17)—that guides you through the request file generation process. You can start the wizard from the ISM: right-click the Web site on which you want to set up SSL, select properties, then in the directory security tab—in the secure commu- nications section—select Server Certificate (as illustrated in Figure 6.18).

Figure 6.17: SSL Web server certificate wizard.

Figure 6.18: Starting the Web server certificate wizard.

In the wizard you have the option to send the request immediately to an online CA or generate the request offline. The first option will not only generate and send the certificate request file; it will also automatically install the certificate. This option will only work if you have an operational Windows enterprise CA. When you choose to generate the request offline, you will have to submit the request to the CA and install the certificate on the Web server all by yourself.

A very important step in the certificate request file generation wizard is the specification of the Web site’s common name. You must make sure that the common name you enter in the wizard matches the name that the browser uses in the URL to securely connect to your Web site. This is because the common name appears in the server certificate, and if the name in the certificate does not match with the name in the URL, the browser will generate an SSL error. Microsoft supports the use of wildcards in the certificate’s common name.

Using Wildcards in the Common Name of SSL Server Certificates

Microsoft supports the use of wildcards in the common name of SSL server certificates. Using wildcards you can reuse the same certificate for different physical Web sites. This may be an interesting option for Web farms where only the machine is different between the different Web farm members.

The IIS SSL/TLS provider supports the following wildcards (the examples are all matching www.hp.com):

• *.hp.com

• w*.hp.com

• *w.hp.com

• www.h*.com

• www.hp.*

  Microsoft’s support for wildcards is in line with RFC 2595 and is documented in the Microsoft Knowledge Base article at http://support.microsoft.com/default.aspx?scid=kb;enus;258858.

Once you have generated the certificate request file, you must use it to generate the actual certificate. SSL/TLS server certificates can be generated in different ways:

• They can be generated by an internal Certification Authority (CA), which is for example using the Windows Server 2003 PKI software.

• They can be generated by an external commercial CA (e.g., Verisign).

• You can generate self-signed certificates using the IIS 6.0 Resource

Kit utility SelfSSL.

SelfSSL is a command-line tool allowing you to generate self-signed SSL/TLS certificates—in other words, you do not need a CA or PKI infrastructure. When you run SelfSSL, it will not only generate a self-signed certificate, but it will also install it on the Web server. If you use it with the /T switch, it will also add the self-signed certificate to the list of trusted certificates in the local machine’s certificate store.

If you did not submit the request file to an online Windows enterprise CA or if you did not use the SelfSSL tool, you will have to manually submit the request to the CA that will issue the server certificate. If the CA you are using is an internal Windows CA, the easiest way to do this is to use the CA’s Web interface. To do so, connect to the CA’s Web site using the /certsrv">http://<servername>/certsrv URL. On the welcome page, select “request a certificate,” then “submit an advanced certificate request,” and finally, select “submit a certificate request by using a base64-encoded CMC or PKCS#10 file, or submit a renewal request by using a base64-encoded PKCS#7 file.” The next Web page will then allow you to paste the base64- encoded content of the certificate request file in the page (the part that starts with --BEGIN NEW CERTIFICATE REQUEST-- and ends withEND NEW CERTIFICATE REQUEST--). If you are using an external commercial CA, consult the CA’s operational procedures.

Once you get the server certificate back from the CA (again, this step is not needed if you used an operational Windows enterprise CA or SelfSSL), you can install it on your Web server. To do so, you can use once more the Web Server Certificate Wizard or the IIScertdeploy.vbs script that comes with the IIS 6.0 Resource Kit. If you use the wizard, this time select the “Process the pending request and install the certificate” option.

SSL/TLS can be configured from the Directory Security tab, using the Edit… button in the Secure Communications section (see also Figure6.11). This button is only enabled if you successfully installed a server certificate. The configuration screen is shown in Figure 6.19. In the configuration screen you can:

• Enable/disable SSL for a Web site.

• Require 128-bit encryption, which provides extra protection for Web sites that deal with sensitive personal and financial information.

• Set client certificate-based authentication options. Both the “Accept client certificates” and “Require client certificates” options require the deployment of certificates to your browser clients. As with server certificates, SSL/TLS client certificates can be acquired from an internal CA or from an external commercial CA.

• Enable certificate mapping (explained in the next section).

• Enable certificate trust lists (CTLs) (explained later in this chapter).

If you want to configure the port that is used for SSL communications, this can be done from the Web site tab in a Web site’s properties (the default port is 443). The Advanced button also allows you to add multiple SSL identities for the site using different ports.

Client SSL/TLS certificates can also be requested from an internal or external CA. If you are using a Windows Server 2003–rooted PKI, users can request certificates using their Certificates MMC snap-in or the CA’s Web interface (accessible using /certsrv">http://<servername>/certsrv). Administrators can also automatically enroll users for SSL/TLS certificates using a GPO setting. More details on user certificate enrollment can be found in Chapter15. To add the server and client certificate’s issuing CA’s certificate to the user browser’s trusted root certificate store, you can use different methods (these are explained in greater detail in Chapter 14):

Figure 6.19: Configuring SSL/TLS.

• Use the IE Deployment Kit to create an install kit with CA’s certificate added to the trusted root domains.

• Put the CA’s certificate on a publicly accessible Web site where users can download and choose to trust it.

• Distribute the CA’s certificate using a GPO setting.

An interesting property of the SSL implementation in Internet Explorer (IE) 6.0 and later is that users can manually clear the SSL cache from the IE GUI. This means that the client-side SSL authentication certificates can be removed from the browser cache. Normally certificates remain in the cache until the computer is restarted. To clear the SSL cache use the Clear SSL State pushbutton in the Internet Options\Content tab.

6.6.2 Certificate mapping

In order to facilitate Web server access control enforcement, SSL client authentication certificates can be mapped to Windows security identities. This feature is called certificate mapping. It allows you to apply authorization settings that are defined for Windows security identities to users who have authenticated to IIS using a certificate.

Certificate mapping can be defined in the IIS metabase or in the Active Directory (AD). In the latter case we’re using a service that is known as the “Windows directory service mapper.” AD-based mapping is an interesting option if you have multiple Web servers that all need to have certificate mappings defined. Instead of defining the mappings on every individual Web server, you can define them once in the central AD repository.

IIS metabase-based mapping can be defined from the ISM secure communications dialog box (illustrated in Figure 6.19) and is only available if you have checked the “enable client certificate mapping” checkbox.

Certificate mapping defined in the metabase can be set up in one of two modes: one-to-one mapping and many-to-one mapping.

• When using one-to-one mapping, IIS looks at the complete content of the client certificate to map it to a Windows security identity.

• Many-to-one certificate mapping is based on rules (as illustrated in Figure 6.20). In this case IIS looks at particular attributes of the client certificate (as defined in the rules) to map it to a Windows security identity.

  
  Figure 6.20: Setting up a many- to-one certificate mapping rule in the ISM.

AD-based mapping can be defined from the Users and Computers MMC snap-in: Use the Name mappings… option in an account object’s context menu—this option is only available if the snap-in is in Advanced Features viewing mode. AD-based mapping only allows for one-to-one mapping. AD-based mapping or the Windows directory service mapper is enabled from the properties of the Web Sites container in the ISM (as illustrated in Figure 6.21).

Figure 6.21: Enabling the Windows directory service mapper.

6.6.3 Certificate validation

During the SSL/TLS protocol exchanges, both the browser and the Web server must validate each other’s certificates (for the server side this is only true if also client-side certificate authentication has been enabled). SSL/ TLS certificate validation includes the following checks (illustrated in Figure 6.22): an X.509 digital signature check, a trust check, a time check, a revocation check, and a formatting check. The certificate validation process is explained in greater detail in Chapter 15.

Figure 6.22: Certificate validation process.

Every X.509 certificate includes a digital signature that is validated when the certificate is used. The digital signature check includes:

• An integrity check: A check that the certificate has not been tampered with

• An authentication check: A cryptographic check to determine that the

Web site owns the private key that is associated with the public key stored in the certificate

The trust check validates whether the issuer of the Web site’s certificate is trusted. Chapter 14 explains in greater detail how Windows and its users and services (including IIS) determine whether or not a certificate is trustworthy. One feature that is worth pointing out here is the IIS support for Certificate Trust Lists (CTLs). A CTL is a signed list of trustworthy CA certificates that can be generated by the Web server administrator. A CTL can be configured using the CTL wizard that can be started from the secure communications dialog box (see also Figure 6.15). If the certificate’s issuer is not trustworthy, IIS will generate a 403.16 HTTP error. On the browser side a security alert will be generated stating that “The security certificate was issued by a company you have not chosen to trust. View the certificate to determine whether you want to trust the certifying authority” (see Figure 6.23(a)).

Figure 6.23: (a) Browser-side certificate trust error, and (b) browser-side certificate time and name error.

The time check validates whether the certificate has expired. EveryX.509 certificate has a limited lifetime. If the certificate has expired, IIS will generate a 403.17 HTTP error. On the browser side a security alert will be generated stating that “The security certificate has expired or is not yet valid” (see Figure 6.23(b)).

The revocation check validates whether the certificate has been revoked. By default, this check is not performed by Internet Explorer, but rather it is performed by IIS. In IIS revocation checking can be controlled using the certcheckmode metabase parameter (value 1 means revocation checking is disabled/value 0 means revocation checking is enabled). If the certificate has been revoked or if the CRL is not available, IIS generates a 403.13 HTTP error. On the browser side a security alert like the one shown in Figure 6.24 will be generated if the CRL is not available. Also, on the browser side revocation checking is controlled using the “check for server certificate revocation” entry in the browser’s Internet options (illustrated in Figure 6.25).

Figure 6.24: Browser-side SSL/TLS revocation check error.

Figure 6.25: Browser-side SSL/TLS certificate revocation checking option.

The formatting check validates whether the content of an X.509 certificate conforms to the X.509 certificate format specification. The check also validates whether the common name in the certificate matches the name of the Web site—as entered in the URL. If the certificate is ill-formed or if the common names do not match, IIS will generate a 403.16 HTTP error. On the browser side a security alert will be generated stating that “The name of the security certificate is invalid or does not match the name of the site” (illustrated in Figure 6.23(b)).

6.6.4 Deployment considerations

In most enterprise environments you must consider the following deployment issues when setting up SSL/TLS for securing Web resources:

• SSL/TLS cannot be used in combination with HTTP host headers.

• SSL/TLS can have a performance impact on the Web server side.

• The combination of SSL/TLS and HTTP load balancing.

• SSL/TLS and HTTP proxy and/or firewall combined operation. This issue is discussed in greater detail in the next section.

When using https, you cannot use HTTP host headers to differentiate between different Web sites on the same Web server. This is because the SSL connection is set up before the HTTP connection and the HTTP information is sent in an encrypted format over the wire.

The asymmetric cryptographic operations behind the certificate-based authentication in SSL/TLS can impact the Web server performance. To deal with this performance issue, you can do three things:

1. Use hardware crypto accelerators devices. These devices offload the main system processor by using a dedicated processor for the cryptographic operations. Table 6.1 gives some example devices (this is a nonexhaustive list).

   Table 6.1: SSL/TLS Crypto Accelerator Devices

   Vendor	Device	More Information At:
   Broadcom	CryptoNetX SSL800 CryptoNetX SSL1600 CryptoNetX SSL4000	http://www.broadcom.com
   F5	Big IP SSL 400 Big IP SSL 800	http://www.f5.com/f5products/bigip/SSL400800
   HP (Compaq)	Atalla AXL 300	http://h18000.www1.hp.com/products/servers/security/axl300
   Intel	Netstructure	http://www.intel.com/support/netstructure/index.htm
   nCipher	nFast 800 nFast 300	http://www.ncipher.com/nfast/index.html
   Rainbow	Cryptoswift PCI	http://www.rainbow.com/products/cryptoswift/PCI.asp

2. Limit the Web page size. Limiting the Web page size reduces the amount of data that need to be cryptographically processed.

3. Reuse cached SSL sessions to limit the number of SSL negotiations. You can fine-tune the SSL caching behavior by using the registry parameters in Table 6.2. They are located in the HKEY_LOCAL_MACHINE\system\currentcontrolset\control\ securityproviders\schannel registry key.^[8]

   Table 6.2: SChannel Caching Registry Parameters

   Registry Parameter	Data Type	Default Value	Meaning
   MaximumCacheSize	REG_DWORD	10,000	Maximum number of SSL sessions to maintain the cache
   ClientCacheTime	REG_DWORD	10 hours	Time in milliseconds to expire a client-side cache element
   ServerCacheTime	REG_DWORD	10 hours	Time in milliseconds to expire a server-side cache element

When using SSL in a Web load balancing environment (e.g., in a Web farm setup), you must make sure that your load balancing solution supports sticky sessions (also known as server affinity). Sticky sessions assure that the HTTP connection always returns to the same Web server in a server farm during a user Web connection sequence. This is key when the Web server maintains user session data.

6.6.5 SSL in proxy and firewall environments

SSL can be configured in different ways when dealing with HTTP proxies: The different approaches can be categorized as either SSL tunneling or SSL bridging. HTTP proxies are used in almost every perimeter security solution that includes application proxy-based firewalls and/or HTTP proxies.

An SSL tunneling setup provides true end-to-end SSL: The SSL tunnel starts on the browser and ends on the Web server. It is illustrated in Figure 6.26. It requires an SSL authentication certificate on the Web server and optionally an SSL client certificate on the client side. The problem with SSL bridging is that it breaks the role of an HTTP proxy. An HTTP proxy typically inspects the content of an HTTP request before it lets the request through. When using SSL the HTTP request cannot be inspected because the HTTP content is encrypted by the SSL tunnel. This is because an SSL tunnel is always set up before the HTTP application-level traffic reaches the destination host. SSL operates on the session level of the TCP/IP networking stack, while HTTP operates on the application level.

Figure 6.26: SSL and HTTP proxy approaches: SSL tunneling.

Does this mean that it is impossible to let an SSL tunnel flow through a firewall or HTTP proxy? No. To enable SSL tunnels to flow through HTTP proxies, SSL-enabled applications can use the HTTP CONNECT method. This method tells the proxy to ignore the content of an SSL session and to simply forward the SSL packets to the destination host (in this case a Web server). The HTTP CONNECT method is defined in RFC 2817 (available from the IETF Web site at http://www.ietf.org). It is important to stress that from a pure security point of view, SSL tunneling is not the best approach; it basically punches holes into your firewalls.

The alternative to SSL tunneling is SSL bridging. Using SSL bridging takes away the security concern expressed earlier for SSL tunneling. SSL bridging basically means that the SSL tunnel is started or terminated on the HTTP proxy. As a consequence, there is no more end-to-end SSL tunnel setup. SSL bridging can be set up in different ways:

• Using a single SSL tunnel that starts on the client side and terminates on the HTTP proxy. This approach is illustrated in Figure 6.27. It requires an SSL certificate for the HTTP proxy and optionally an SSL certificate for the client side (if strong client-side authentication is required).

  
  Figure 6.27: SSL and HTTP proxy approaches: SSL bridging (single tunnel terminated on proxy).

• Using a single SSL tunnel that starts on the HTTP proxy and terminates on the Web server. This approach is illustrated in Figure 6.28. It requires an SSL certificate for the Web server and optionally an SSL certificate for the HTTP proxy (if strong client-side authentication is required).

  
  Figure 6.28: SSL and HTTP proxy approaches: SSL bridging (single tunnel terminated on Web server).

  
  Figure 6.29: SSL and HTTP proxy approaches: SSL bridging (two tunnels).

  Setting up SSL Bridging and Tunneling in an MS ISA Server Environment

  ISA Server is Microsoft’s follow-up product to Proxy Server 2.0. Unlike its predecessor, which was primarily a Web caching solution, ISA Server can serve as a full-blown firewall. In addition, ISA Server provides advanced packet filtering and application proxy firewall functions.

  To set up SSL bridging on ISA Server, you must create an ISA Server destination set and Web publishing rule. To facilitate the configuration of SSL bridging in an OWA environment, Microsoft provides an OWA Publishing Wizard in Feature Pack 1 for ISA Server (illustrated in Figure 6.30). ISA Server Feature Pack 1 can be downloaded from http://www.microsoft.com/downloads/details.aspx?FamilyID=2f92b02c-ac49-44df-af6c- 5be084b345f9&DisplayLang=en.

  
  Figure 6.30: Setting up SSL bridging using the OWA Publishing Wizard.

  To configure ISA Server for SSL tunneling you need to use ISA Server’s Server Publishing feature. Compared to setting up SSL bridging, setting up SSL tunneling is relatively easy, mainly because no SSL certificate must be set up for the ISA Server.

• Using two SSL tunnels: one starting on the client and terminating on the HTTP proxy and another one starting on the HTTP proxy and terminating on the Web server. This approach is illustrated in Figure 6.29. It requires an SSL certificate on the HTTP proxy and Web server and optionally an SSL certificate on the client side (if strong client-side authentication is required).

Table 6.3 shows the different SSL approaches and their advantages and disadvantages.

Table 6.3: SSL and HTTP Proxy Approaches

SSL Approach	Pros	Cons
SSL Tunneling	End-to-end SSL	Security hole on HTTP proxy and firewall level: the proxy cannot screen for exploit infected content anymore
SSL Bridging—Option 1: Single SSL tunnel, started on client and terminated on proxy	HTTP content inspection on HTTP proxy and firewall level (no security hole) Offloads SSL processing from Web server	No end-to-end SSL HTTP traffic goes in the clear between proxy and Web server
SSL Bridging—Option 2: Single SSL tunnel, started on proxy and terminated on Web server	HTTP content inspection on HTTP proxy and firewall level (no security hole) HTTP traffic is encrypted between proxy and Web server	No end-to-end SSL HTTP traffic goes in the clear between browser and HTTP proxy
SSL Bridging—Option 3: Two SSL tunnels	HTTP content inspection on HTTP proxy and firewall level (no security hole) HTTP traffic is encrypted both between browser and proxy and between proxy and Web server	No end-to-end SSL

^[8]These registry settings were available for IIS 4.0 and 5.0 and documented in the following KB article: http://support.microsoft.com/default.aspx?scid=kb;en-us;247658. At the time of writing, no update to the KB article was available for IIS 6.0.