10.10 Obtaining Certificate Revocation Lists with OpenSSL

10.10.1 Problem

You have a certificate that you want to verify, as well as the certificate that was used to issue it, but you need to check the issuing authority's CRL to make sure that the certificate has not been revoked. We cover how to use a CRL once you have it in Recipe 10.5 but how do you get it in the first place?

10.10.2 Solution

All CAs should publish a CRL for each certificate used for issuing certificates, but many do not seem to. In fact, most CAs make it very difficult to find the CRLs they do publish, so it is easy to come to the conclusion that they do not publish a CRL at all. It turns out that some CAs do not publish a CRL, but the most prominent of CAs all do. Unfortunately, the CAs that do make it easy to find their CRLs are in the minority. We have spent a sizable amount of time attempting to track down CRLs for each of the certificates we have listed in Recipe 10.3, as well as numerous others with which we had no success. We have also managed to find many CRLs for which we were unable to find matching issuing certificates, but we have omitted them here. Many of them can be found at http://www.openvalidation.org.

Note that many CAs require acceptance of a licensing agreement before you're allowed to download their CRLs. You should make sure to check the information that we provide here before you use it, to ensure that you have the legal right to use the data and that the CA has not changed the location of their URLs since this book went to press. We have found many certificates that contain cRLDistributionPoints extensions in them where the URL was no longer valid. It may be that the URLs are invalid because no CRL has ever been issued; however, to avoid any possible confusion, it would be better for these CAs to issue an empty CRL.

10.10.3 Discussion

To obtain a CRL, first check the certificate and its issuing certificate for a cRLDistributionPoints extension that contains a URI GeneralName. This extension is defined in RFC 3280, and it specifies a way for CAs to communicate the location of the CRL that corresponds to the certificate used to issue another certificate. Unfortunately, this extension is defined as being optional, and most root CAs do not use it.

#include <string.h> #include <unistd.h> #include <sys/time.h> #include <openssl/conf.h> #include <openssl/ocsp.h> #include <openssl/ssl.h> #include <openssl/x509v3.h>     typedef struct {   char          *name;   unsigned char *fingerprint;   unsigned int  fingerprint_length;   char          *crl_uri;   char          *ocsp_uri; } spc_cacert_t;     spc_cacert_t *spc_lookup_cacert(X509 *cert);     static char *get_distribution_point(X509 *cert) {   int                   extcount, i, j;   const char            *extstr;   CONF_VALUE            *nval;   unsigned char         *data;   X509_EXTENSION        *ext;   X509V3_EXT_METHOD     *meth;   STACK_OF(CONF_VALUE)  *val;       if ((extcount = X509_get_ext_count(cert)) > 0) {     for (i = 0; i < extcount; i++) {       ext = X509_get_ext(cert, i);       extstr = OBJ_nid2sn(OBJ_obj2nid(X509_EXTENSION_get_object(ext)));       if (strcasecmp(extstr, "crlDistributionPoints")) continue;           if (!(meth = X509V3_EXT_get(ext))) break;       data = ext->value->data;       val = meth->i2v(meth, meth->d2i(0, &data, ext->value->length), 0);       for (j = 0;  j < sk_CONF_VALUE_num(val);  j++) {         nval = sk_CONF_VALUE_value(val, j);         if (!strcasecmp(nval->name, "URI"))           return strdup(nval->value);       }     }   }   return 0; }     char *spc_getcert_crlurl(X509 *cert, X509 *issuer, int lookup_only) {   char          *uri;   spc_cacert_t  *cacert;       if (!lookup_only) {     if (cert && (uri = get_distribution_point(cert)) != 0) return uri;     if (issuer && (uri = get_distribution_point(issuer)) != 0) return uri;   }       /* Get the fingerprint of the cert's issuer, and look it up in a table */   if (issuer) {     if (!(cacert = spc_lookup_cacert(issuer))) return 0;     return (cacert->crl_uri ? strdup(cacert->crl_uri) : 0);   }   return 0; }

If neither the certificate we are checking nor the certificate's issuing certificate contains a cRLDistributionPoints extension that we can use, we will fall back to looking up the issuing certificate's fingerprint in a table that we have built from the information presented in Recipe 10.3:

static spc_cacert_t lookup_table[  ] = {   { "Equifax Secure Certificate Authority",     "\x67\xcb\x9d\xc0\x13\x24\x8a\x82\x9b\xb2\x17\x1e\xd1\x1b\xec\xd4", 16,     "http://crl.geotrust.com/crls/secureca.crl",   },   { "Equifax Secure Global eBusiness CA-1",     "\x8f\x5d\x77\x06\x27\xc4\x98\x3c\x5b\x93\x78\xe7\xd7\x7d\x9b\xcc", 16,     "http://crl.geotrust.com/crls/globalca1.crl",   },   { "Equifax Secure eBusiness CA-1",     "\x64\x9c\xef\x2e\x44\xfc\xc6\x8f\x52\x07\xd0\x51\x73\x8f\xcb\x3d", 16,     "http://crl.geotrust.com/crls/ebizca1.crl",   },   { "Equifax Secure eBusiness CA-2",     "\xaa\xbf\xbf\x64\x97\xda\x98\x1d\x6f\xc6\x08\x3a\x95\x70\x33\xca", 16,     "http://crl.geotrust.com/crls/ebiz.crl",   },   { "RSA Data Security Secure Server CA (VeriSign)",     "\x74\x7b\x82\x03\x43\xf0\x00\x9e\x6b\xb3\xec\x47\xbf\x85\xa5\x93", 16,     "http://crl.verisign.com/RSASecureServer.crl", "http://ocsp.verisign.com/",   },   { "Thawte Server CA",     "\xc5\x70\xc4\xa2\xed\x53\x78\x0c\xc8\x10\x53\x81\x64\xcb\xd0\x1d", 16,     "https://www.thawte.com/cgi/lifecycle/getcrl.crl?skeyid=%07%15%28mps%AA"     "%B2%8A%7C%0F%86%CE8%93%008%05%8A%B1",   },   { "TrustCenter Class 1 CA",     "\x8d\x26\xff\x2f\x31\x6d\x59x\29\xdd\xe6\x36\xa7\xe2\xce\x64\x25", 16,     "https://www.trustcenter.de:443/cgi-bin/CRL.cgi/TC_Class1.crl?Page=GetCrl"     "&crl=2",   },   { "TrustCenter Class 2 CA",     "\xb8\x16\x33\x4c\x4c\x4c\xf2\xd8\xd3\x4d\x06\xb4\xa6\x58\x40\x03", 16,     "https://www.trustcenter.de:443/cgi-bin/CRL.cgi/TC_Class2.crl?Page=GetCrl"     "&crl=3",   },   { "TrustCenter Class 3 CA",     "\x5f\x94\x4a\x73\x22\xb8\xf7\xd1\x31\xec\x59\x39\xf7\x8e\xfe\x6e", 16,     "https://www.trustcenter.de:443/cgi-bin/CRL.cgi/TC_Class3.crl?Page=GetCrl"     "&crl=4",   },   { "TrustCenter Class 4 CA",     "\x0e\xfa\x4b\xf7\xd7\x60\xcd\x65\xf7\xa7\x06\x88\x57\x98\x62\x39", 16,     "https://www.trustcenter.de:443/cgi-bin/CRL.cgi/TC_Class4.crl?Page=GetCrl"     "&crl=5",   },   { "The USERTRUST Network - UTN-UserFirst-Object",     "\xa7\xf2\xe4\x16\x06\x41\x11\x60\x30\x6b\x9c\xe3\xb4\x9c\xb0\xc9", 16,     "http://crl.usertrust.com/UTN-UserFirst-Object.crl",   },   { "The USERTRUST Network - UTN-UserFirst-Network Applications",     "\xbf\x60\x59\xa3\x5b\xba\xf6\xa7\x76\x42\xda\x6f\x1a\x7b\x50\xcf", 16,     "http://crl.usertrust.com/UTN-UserFirst-NetworkApplications.crl",   },   { "The USERTRUST Network - UTN-UserFirst-Hardware",     "\x4c\x56\x41\xe5\x0d\xbb\x2b\xe8\xca\xa3\xed\x18\x08\xad\x43\x39", 16,     "http://crl.usertrust.com/UTN-UserFirst-Hardware.crl",   },   { "The USERTRUST Network - UTN-UserFirst-Client Authentication and Email",     "\xd7\x34\x3d\xef\x1d\x27\x09\x28\xe1\x31\x02\x5b\x13\x2b\xdd\xf7", 16,     "http://crl.usertrust.com/UTN-UserFirst-ClientAuthenticationandEmail.crl",   },   { "The USERTRUST Network - UTN - DataCorp SGC",     "\xb3\xa5\x3e\x77\x21\x6d\xac\x4a\xc0\xc9\xfb\xd5\x41\x3d\xca\x06", 16,     "http://crl.usertrust.com/UTN-DataCorpSGC.crl",   },   { "ValiCert Class 1 Policy Validation Authority",     "\x65\x58\xab\x15\xad\x57\x6c\x1e\xa8\xa7\xb5\x69\xac\xbf\xff\xeb", 16,     "http://www.valicert.com/repository/ValiCert%20Calss%201%20Policy%20Val"     "idation%20Authority.crl",   },   { "VeriSign Class 1 Public PCA (2020-01-07)",     "\x51\x86\xe8\x1f\xbc\xb1\xc3\x71\xb5\x18\x10\xdb\x5f\xdc\xf6\x20", 16,     "http://crl.verisign.com/pca1.1.1.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 1 Public PCA (2028-08-01)",     "\x97\x60\xe8\x57\x5f\xd3\x50\x47\xe5\x43\x0c\x94\x36\x8a\xb0\x62", 16,     "http://crl.verisign.com/pca1.1.1.crl",     "http://ocsp.verisign.com/",   },   { "VeriSign Class 1 Public PCA G2 (2018-05-18)",     "\xf2\x7d\xe9\x54\xe4\xa3\x22\x0d\x76\x9f\xe7\x0b\xbb\xb3\x24\x2b", 16,     "http://crl.verisign.com/pca1-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 1 Public PCA G2 (2028-08-01)",     "\xdb\x23\x3d\xf9\x69\xfa\x4b\xb9\x95\x80\x44\x73\x5e\x7d\x41\x83", 16,     "http://crl.verisign.com/pca1-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 2 Public PCA (2004-01-07)",     "\xec\x40\x7d\x2b\x76\x52\x67\x05\x2c\xea\xf2\x3a\x4f\x65\xf0\xd8", 16,     "http://crl.verisign.com/pca2.1.1.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 2 Public PCA (2028-08-01)",     "\xb3\x9c\x25\xb1\xc3\x2e\x32\x53\x80\x15\x30\x9d\x4d\x02\x77\x3e", 16,     "http://crl.verisign.com/pca2.1.1.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 2 Public PCA G2 (2018-05-18)",     "\x74\xa8\x2c\x81\x43\x2b\x35\x60\x9b\x78\x05\x6b\x58\xf3\x65\x82", 16,     "http://crl.verisign.com/pca2-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 2 Public PCA G2 (2028-08-01)",     "\x2d\xbb\xe5\x25\xd3\xd1\x65\x82\x3a\xb7\x0e\xfa\xe6\xeb\xe2\xe1", 16,     "http://crl.verisign.com/pca2-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 3 Public PCA (2004-01-07)",     "\x78\x2a\x02\xdf\xdb\x2e\x14\xd5\xa7\x5f\x0a\xdf\xb6\x8e\x9c\x5d", 16,     "http://crl.verisign.com/pca3.1.1.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 3 Public PCA (2028-08-01)",     "\x10\xfc\x63\x5d\xf6\x26\x3e\x0d\xf3\x25\xbe\x5f\x79\xcd\x67\x67", 16,     "http://crl.verisign.com/pca3.1.1.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 3 Public PCA G2 (2018-05-18)",     "\xc4\x63\xab\x44\x20\x1c\x36\xe4\x37\xc0\x5f\x27\x9d\x0f\x6f\x6e", 16,     "http://crl.verisign.com/pca3-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Class 3 Public PCA G2 (2028-08-01)",     "\xa2\x33\x9b\x4c\x74\x78\x73\xd4\x6c\xe7\xc1\xf3\x8d\xcb\x5c\xe9", 16,     "http://crl.verisign.com/pca3-g2.crl", "http://ocsp.verisign.com/",   },   { "VeriSign Commercial Software Publishers CA",     "\xdd\x75\x3f\x56\xbf\xbb\xc5\xa1\x7a\x15\x53\xc6\x90\xf9\xfb\xcc", 16,     "http://crl.verisign.com/Class3SoftwarePublishers.crl",     "http://ocsp.verisign.com/",   },   { "VeriSign Individual Software Publishers CA",     "\x71\x1f\x0e\x21\xe7\xaa\xea\x32\x3a\x66\x23\xd3\xab\x50\xd6\x69", 16,     "http://crl.verisign.com/Class2SoftwarePublishers.crl",     "http://ocsp.verisign.com/",   },   { 0, 0, 0, 0, 0 }, };     spc_cacert_t *spc_lookup_cacert(X509 *cert) {   spc_cacert_t  *entry;   unsigned int  fingerprint_length;   unsigned char fingerprint[EVP_MAX_MD_SIZE];       fingerprint_length = EVP_MAX_MD_SIZE;   if (!X509_digest(cert, EVP_md5(  ), fingerprint, &fingerprint_length)) return 0;       for (entry = lookup_table;  entry->name;  entry++) {     if (entry->fingerprint_length != fingerprint_length) continue;     if (!memcmp(entry->fingerprint, fingerprint, fingerprint_length)) return entry;   }   return 0; }

Once we have the URL of the CRL we want, it is a simple matter to retrieve it using the HTTP protocol. OpenSSL does not provide even the simplest of HTTP clients, so we must speak the bare minimum ourselves to connect to the server and retrieve the data.

static void *retrieve_webdata(char *uri, int *datalen, spc_x509store_t *store) {   int     bytes, content_length = 0, headerlen = 0, sd, ssl;   BIO     *conn = 0;   SSL     *ssl_ptr;   char    buffer[1024];   char    *headers = 0, *host = 0, *path = 0, *port = 0, *tmp;   void    *data = 0;   fd_set  rmask, wmask;   SSL_CTX *ctx = 0;       *datalen = 0;   if (!OCSP_parse_url(uri, &host, &port, &path, &ssl)) goto end_error;   if (!(conn = spc_connect(host, atoi(port), ssl, store, &ctx))) goto end_error;       /* Send the request for the data */   BIO_printf(conn, "GET %s HTTP/1.0\r\nConnection: close\r\n\r\n", path);       /* Put the socket into non-blocking mode */   BIO_get_fd(conn, &sd);   BIO_socket_nbio(sd, 1);   if (ssl) {     BIO_get_ssl(conn, &ssl_ptr);     SSL_set_mode(ssl_ptr, SSL_MODE_ENABLE_PARTIAL_WRITE |                           SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER);   }       /* Loop reading data from the socket until we've got all of the headers */   for (;;) {     FD_ZERO(&rmask);     FD_SET(sd, &rmask);     FD_ZERO(&wmask);     if (BIO_should_write(conn)) FD_SET(sd, &wmask);     if (select(FD_SETSIZE, &rmask, &wmask, 0, 0) <= 0) continue;     if (FD_ISSET(sd, &wmask)) BIO_write(conn, buffer, 0);     if (FD_ISSET(sd, &rmask)) {       if ((bytes = BIO_read(conn, buffer, sizeof(buffer))) <= 0) {         if (BIO_should_retry(conn)) continue;         goto end_error;       }       if (!(headers = (char *)realloc((tmp = headers), headerlen + bytes))) {         headers = tmp;         goto end_error;       }       memcpy(headers + headerlen, buffer, bytes);       headerlen += bytes;       if ((tmp = strstr(headers, "\r\n\r\n")) != 0) {         *(tmp + 2) = '\0';         *datalen = headerlen - ((tmp + 4) - headers);         headerlen -= (*datalen + 2);         if (*datalen > 0) {           if (!(data = (char *)malloc(*datalen))) goto end_error;           memcpy(data, tmp + 4, *datalen);         }         break;       }     }   }       /* Examine the headers to determine whether or not to continue.  If we are to    * continue, look for a content-length header to find out how much data we're    * going to get.  If there is no content-length header, we'll have to read    * until the remote server closes the connection.    */   if (!strncasecmp(headers, "HTTP/1.", 7)) {     if (!(tmp = strchr(headers, ' '))) goto end_error;     if (strncmp(tmp + 1, "200 ", 4) && strncmp(tmp + 1, "200\r\n", 5))       goto end_error;     for (tmp = strstr(headers, "\r\n");  tmp;  tmp = strstr(tmp + 2, "\r\n")) {       if (strncasecmp(tmp + 2, "content-length: ", 16)) continue;       content_length = atoi(tmp + 18);       break;     }   } else goto end_error;       /* Continuously read and accumulate data from the remote server.  Finish when    * we've read up to the content-length that we received.  If we didn't receive    * a content-length, read until the remote server closes the connection.    */   while (!content_length || *datalen < content_length) {     FD_ZERO(&rmask);     FD_SET(sd, &rmask);     FD_ZERO(&wmask);     if (BIO_should_write(conn)) FD_SET(sd, &wmask);     if (select(FD_SETSIZE, &rmask, &wmask, 0, 0) <= 0) continue;     if (FD_ISSET(sd, &wmask)) BIO_write(conn, buffer, 0);     if (FD_ISSET(sd, &rmask))       if ((bytes = BIO_read(conn, buffer, sizeof(buffer))) <= 0) {         if (BIO_should_retry(conn)) continue;         break;       }     if (!(data = realloc((tmp = data), *datalen + bytes))) {       data = tmp;       goto end_error;     }     memcpy((char *)data + *datalen, buffer, bytes);     *datalen += bytes;   }       if (content_length && *datalen != content_length) goto end_error;   goto end;     end_error:   if (data) { free(data);  data = 0;  *datalen = 0; } end:   if (headers) free(headers);   if (conn) BIO_free_all(conn);   if (host) OPENSSL_free(host);   if (port) OPENSSL_free(port);   if (path) OPENSSL_free(path);   if (ctx) SSL_CTX_free(ctx);   return data; }

With the data that has been retrieved from the server, we can create an OpenSSL X509_CRL object. We assume that the data retrieved from the server will be in DER format, which is the format returned by every server we have encountered (see Recipe 7.16). The DER format is more portable because not everyone supports PEM format. It is also a more compact format for transfer because it does not include any headers or base64 encoding. The OpenSSL function d2i_X509_CRL_bio( ) is used to create the X509_CRL object using a memory base BIO object created with BIO_new_mem_buf( ).

X509_CRL *spc_retrieve_crl(X509 *cert, X509 *issuer, spc_x509store_t *store) {   BIO       *bio = 0;   int       datalen, our_store;   char      *uri = 0, *uri2 = 0;   void      *data = 0;   X509_CRL  *crl = 0;        if ((our_store = (!store)) != 0) {     if (!(store = (spc_x509store_t *)malloc(sizeof(spc_x509store_t)))) return 0;     spc_init_x509store(store);      spc_x509store_addcert(store, issuer);   }      if (!(uri = spc_getcert_crlurl(cert, issuer, 0))) goto end;   if (!(data = retrieve_webdata(uri, &datalen, store))) {     uri2 = spc_getcert_crlurl(cert, issuer, 1);     if (!uri2 || !strcmp(uri, uri2)) goto end;     if (!(data = retrieve_webdata(uri2, &datalen, store))) goto end;   }       bio = BIO_new_mem_buf(data, datalen);   crl = d2i_X509_CRL_bio(bio, 0);      end:   if (bio) BIO_free(bio);   if (data) free(data);   if (uri) free(uri);   if (uri2) free(uri2);   if (store && our_store) {     spc_cleanup_x509store(store);     free(store);   }        return crl; }

In this recipe, we have used a number of functions from Recipe 9.1, Recipe 10.5, Recipe 10.7, and Recipe 10.8. These functions provide us with network connectivity and certificate verification. We will only need the latter if we need to connect to an SSL-enabled web server to retrieve the CRL, and it will all be handled by the network connectivity functions.

Note that we construct an X509_STORE object that contains any system-wide trusted certificates as well as the issuing certificate for which we're getting the CRL. For simplicity, we assume that an SSL-enabled server that is serving the CRL will present this same certificate. In practice, however, that is not always a safe assumption. Our testing indicates that this assumption frequently holds true, but there is a problem: if we are retrieving the CRL from an SSL-enabled server, we have to trust that the peer's certificate has not been revoked. Fortunately, this is a reasonably safe assumption for us to make here because if a CA's signing certificate has been revoked for some reason, there are much bigger problems.^[2]

^[2] If the CA's signing certificate has been revoked, it is still acceptable to trust the signature on the CRL if and only if the signing certificate is also in the list of revoked certificates. Unfortunately, if it is not, there is no way to know that the certificate has been revoked, so there is no choice but to accept it. If the CA's signing certificate has been revoked because of a compromise of the certificate's corresponding private key, the party responsible for the compromise could likely issue an invalid CRL. As you can see, this is a vicious circle and only serves to demonstrate the flaws in CRLs that we discuss in Recipe 10.1.

We have provided code here to retrieve CRLs using HTTP because it is simple to implement and is commonly used by CAs to distribute their CRLs; however, LDAP is also commonly used for CRL distribution. Unfortunately, owing to the complexity of the solution, we don't include a detailed discussion of that topic in this book.

LDAP is commonly used instead of the Directory Access Protocol (DAP) simply because it is less cumbersome. Unfortunately, it lacks some of the features that make storing CRLs and other PKI objects in a directory attractive. In particular, LDAP does not support location transparency and uses referrals instead, but few LDAP client implementations actually support referrals correctly. Because of the lack of location transparency, LDAP does not scale as well as DAP, and it makes it more difficult for CAs to interoperate.

From the standpoint of the client, using LDAP to retrieve CRLs adds complexity without much benefit over other, simpler protocols such as HTTP. We feel that it's important to be aware of how common the use of LDAP is, and we leave it to you to decide whether to include support for it in your own programs.

10.10.4 See Also

• RFC 3280: Internet X.509 Public Key Infrastructure Certificate and Certificate Revocation List (CRL) Profile

• Recipe 7.16, Recipe 9.1, Recipe 10.1, Recipe 10.3, Recipe 10.5, Recipe 10.7, Recipe 10.8