The History and Origins of LDAP

In the first part of this chapter we introduced the notion of online directories and made comparisons to a variety of other network-based services and protocols. Now that we have discussed the capabilities of directories in depth, it is time to discuss LDAP and the origin of general-purpose, standards-based online directories.

The Dawn of Standard Directories: X.500

It is impossible to discuss the origins of LDAP without first talking about X.500. In the mid-1980s, two separate standards bodies independently started work on directory services that could work across system, corporate, and international boundaries. The first body was the International Telegraph and Telephone Consultative Committee (CCITT), which later changed its name to the International Telecommunication Union (ITU). CCITT member organizations ( mainly telephone companies) wanted to create a white pages directory that could be used to look up telephone numbers and electronic mail addresses.

The other standards body was the International Organization for Standardization (ISO), which wanted a directory to serve as a name service for Open Systems Interconnection (OSI) networks and applications. (A name service provides information about network objects; the DNS is a familiar example.)

Eventually, the two independent directory specification efforts merged into one effort, and X.500 was born. The first X.500 standard was approved in late 1988 and published in early 1990 by CCITT; it was subsequently updated in 1993, 1997, and 2001 (work continues today). The specification itself references other ISO Standards and consists of a series of recommendations, including the following:

• X.501: The Models . Describes the concepts and models that underlie an X.500 directory service.

• X.509: Authentication Framework . Describes in detail how the authentication of directory clients and servers is handled in X.500.

• X.511: Abstract Service Definition . Describes in detail the functional services provided by X.500 directories (for example, search operations, modify operations, and so on).

• X.518: Procedures for Distributed Operation . Describes how directory operations that span multiple servers are handled, among other details.

• X.519: Protocol Specifications . Describes all the X.500 protocols, including Directory Access Protocol (DAP), Directory System Protocol (DSP), Directory Operational Binding Protocol (DOP), and Directory Information Shadowing Protocol (DISP).

• X.520: Selected Attribute Types . Defines attribute types used by X.500 itself, and some that are generally useful as well (such as the telephoneNumber attribute).

• X.521: Selected Object Classes . Defines object classes used by X.500 itself, and some that are generally useful as well (such as the person object class).

• X.525: Replication . Describes how directory content is replicated among X.500 servers.

• X.530: Systems Management . Describes how an X.500 directory service can be managed through the use of OSI systems management services and protocols, directory services and protocols, and local means.

As you can see, X.500 developers must absorb quite a few documents before they can begin to develop X.500 directory services software! The X.500 documents are published both online and in paper form by the ITU, which currently charges money to access them.

X.500 Innovations

At the time it was originally defined, X.500 had many novel qualities. First, it was one of the first truly general-purpose directory systems, and it was designed from the beginning to be extensible to serve the needs of a great variety of applications. Second, X.500 provided a rich search operation that supported many different kinds of queries. Third, X.500 was designed to be a highly distributed system in which the servers, data, and administrators could span the globe. Finally, X.500 was an open standard not controlled by any one vendor or tied to any specific operating system, networking technology, or application. The dream of the X.500 designers was that there would eventually be one fully interconnected global directory service: the Directory. Figure 1.10 shows the major components of an X.500 directory service.

Figure 1.10. Components of an X.500 Directory Service

X.500 directories rely on a large suite of protocols, including DAP, DSP, DOP, and DISP. Some of X.500's strengths are its information model (which is flexible and complete), its versatility, and its openness. As you will see, the LDAP designers adopted many of the best ideas from X.500 while removing unneeded complexity.

X.500 Flaws

In practice, X.500 directories suffered from some significant flaws, especially in their first decade of existence. The early implementations were buggy and did not perform or scale well. Those who tried to deploy distributed X.500 directories discovered some significant barriers to adoption, many of which were not specific to X.500 (such as difficulty obtaining and maintaining good data in a directory service). In addition, because of the complexity of the standards themselves and the associated difficulty of implementation, it has taken a long time for interoperability between different X.500 implementations to become possible.

Furthermore, X.500 was based on the OSI network protocols. The dream of the OSI designers was that OSI would eventually replace TCP/IP as the networking protocol suite of the future. Unfortunately for them, this never happened . The simplicity, speed, and low cost of TCP/IP proved to be an unbeatable combination. Recently, in fact, the proponents of X.500 have defined a mapping that allows X.500 to run directly over TCP/IP.

Finally, the origins of X.500 itself have to some extent prevented it from succeeding. The Internet grew rapidly in a bottom-up fashion as independent organizations deployed hosts and services at their own pace and without a lot of reliance on others. In contrast, X.500 was designed with the large public service providers in mind, implying a top-down deployment. Top-down deployment has proved to be impossible to achieve ”and it is certainly at odds with the prevailing Internet culture.

Early X.500 Implementations and Pilots

One of the best-known early X.500 implementations, called Quipu, was developed at University College, London. Its name refers to the complicated system of knot-tying that the Incan culture used for communication. Part of the reason for Quipu's early popularity was that it was freely available in source code form. Quipu is built on top of an OSI networking package called the ISO Development Environment (ISODE), which allows OSI protocols to run on top of TCP/IP and, thus, the Internet.

The availability of Quipu encouraged many organizations to experiment with X.500 directories. The Quipu implementation served as the basis for several European and United States “based pilot projects for X.500 white pages applications. Unfortunately, these worldwide X.500 directory pilots didn't grow as fast as their sponsors hoped, and interest has waned.

X.500 found its greatest success in large organizations that could afford the time and effort needed to deploy a large, complex directory service. Some early deployers of X.500 included the United Kingdom academic community, Boeing Corporation, the National Aeronautics and Space Administration (NASA), the University of Texas, and the University of Michigan. X.500 products are still available today, although all of them are accessible through the use of LDAP in addition to the X.500 protocols. Typically, an X.500 directory service is deployed in partnership with the X.500 software vendor. Most vendors provide extensive consulting services to help organizations deploy and integrate an X.500 directory into their existing environment.

The Creation and Rise of LDAP

The early adopters of X.500 found that DAP (X.500's directory client access protocol) was fairly complex and not well suited for or available on the desktop computers of the day (PCs and Macintoshes). DAP is large, complex, and difficult to implement, and most implementations perform poorly. Because most potential directory users had ordinary machines on their desks, the people deploying X.500 began to look for an approach that avoided the heavyweight DAP.

Forerunners of LDAP: DIXIE and DAS

In about 1990, two independent groups devised similar protocols that, compared to DAP, were simpler and easier for desktop computers to implement. Desktop clients used one of these new protocols to communicate with an intermediate server directly over TCP/IP, and the intermediate server in turn implemented X.500 DAP. Both groups brought their work to the IETF.

The two lightweight protocols for desktop computers were called Directory Assistance Service (DAS), defined in RFC 1202; and Directory Interface to X.500 Implemented Efficiently (DIXIE), defined in RFC 1249. Figure 1.11 shows the architecture of a directory system that uses a DIXIE client/client/server architecture.

Figure 1.11. DIXIE Provides a Front End to an X.500 Directory

Both protocols were successful; DIXIE, though, was quickly adopted as the preferred method to access X.500 directory systems. However, one shortcoming of the DAS and DIXIE protocols is that both were closely tied to a single X.500 implementation (Quipu).

The Creation of LDAP

After DIXIE and DAS showed the utility of a lighter-weight access protocol for X.500, the members of the OSI-DS Working Group within the IETF decided to join forces and produce a full-featured , lightweight directory access protocol for X.500 directories. Thus, LDAP was born.

The developers of the first LDAP specification were Wengyik Yeong, Steve Kille, Colin Robbins, and Tim Howes, who is one of the authors of this book. Note also that contributions were made by many people from the Internet and X.500 communities. The first LDAP specification was published as RFC 1487 in July 1993. The first version of LDAP to see widespread use was version 2 (LDAPv2); the final specification for LDAPv2 was published as RFC 1777.

LDAP Innovations

The LDAP developers simplified the heavyweight X.500 DAP protocol in four important areas:

1. Functionality . LDAP provides most of DAP's functionality at a much lower cost. Redundant operations and rarely used features of DAP were eliminated, simplifying the implementation of LDAP clients and servers.

   The IETF

   The Internet Engineering Task Force (IETF) is a large, open, international group of network researchers, designers, and operators who work on evolving the Internet architecture and enhancing its capabilities. The IETF was formally chartered in 1986, although it existed informally for some time before that. The IETF is focused primarily on protocol development and engineering for the Internet. The group is open to any interested individual; there are no membership requirements, and only a small fee is collected to cover the cost of meetings. The IETF holds face-to-face meetings three times each year.

   The technical work is done primarily within IETF working groups (although individual or vendor contributions are also accepted). The working groups have one or more chairpersons, and the group members meet three times each year at the regularly scheduled IETF meetings. However, most of the work happens outside the meetings on electronic mailing lists that each working group maintains. The working groups are organized into several large areas (routing, security, transport, applications, and so on). An early directory working group was the Open System Interconnection-Directory Services (OSI-DS) Working Group, which initially focused on the use of X.500-based directories on the Internet.

   Specifications produced by IETF members are first published in draft documents called, logically enough, Internet Drafts . When consensus is reached in a working group and a series of technical and administrative requirements are adequately met, documents may be published as Requests for Comments (RFCs). All IETF documents are freely available at no cost on the Internet. Note that there are several categories of RFCs, ranging from Informational to Experimental to Standards Track, and that few RFCs are intended to become official Internet Standards. As of early 2003, more than 3,400 RFCs had been published, but fewer than 70 had reached full Internet Standard status.

   The first IETF meeting, held in San Diego in January 1986, was attended by 21 network engineers . IETF meetings now have upwards of 2,000 attendees. As of early 2003, more than 140 active working groups were organized into 8 areas within the IETF. More information about the IETF can be found on its World Wide Web site at www.ietf.org.

2. Data representation . In LDAP, most data elements are carried as simple text strings, thereby simplifying implementations and increasing performance (but for efficiency the text strings are wrapped inside binary-encoded messages).

3. Encoding . A subset of the X.500 encoding rules is used to encode LDAP messages, thus simplifying implementation.

4. Transport . LDAP runs directly over TCP instead of requiring the unwieldy, multilayer OSI networking stack. Implementation is simplified, performance is increased, and the need for OSI is eliminated, thus easing the deployment of LDAP directories.

Early LDAP Implementations

At first, LDAP was used exclusively to provide a front end for X.500-based directory services. The first LDAP implementation was produced by the authors while at the University of Michigan. The U-M LDAP implementation, as it came to be known, was small and fast, and it ran on a wide variety of popular computing platforms. Most importantly, it included a simple, well-specified C language API that implemented all of the client side of LDAP and could be used to develop any kind of LDAP client application. The LDAPv2 client API eventually became a de facto standard and was published in RFC 1823. Figure 1.12 depicts the components of the early U-M LDAP software releases. The key server-side component is ldapd, which is an LDAP “to “X.500 DAP protocol translator.

Figure 1.12. Major Components of the Early U-M LDAP Releases

The first version of the U-M LDAP implementation was released publicly on the Internet in 1992 with a copyright notice that allowed unrestricted use of the software. A wide range of freely available and commercial LDAP client software soon followed, much of which was based on the University of Michigan implementation.

LDAP as a Standalone Directory Service

In early 1995, the LDAP group at U-M found itself on the brink of another revelation. When examining the directory access statistics for the U-M service, the group noticed that more than 99 percent of the X.500 directory access came through LDAP. This was true of most X.500 directories. Figure 1.13 shows the architecture prevalent at the time.

Figure 1.13. LDAP Directory System Architecture, circa 1995

Although LDAP had solved the directory client access problem, the server implementations were still large, complex, and difficult to deploy. The U-M LDAP group realized that by eliminating the intermediate ldapd server, they could greatly reduce the overall complexity of the system and greatly increase the performance. After all, ldapd spent all its time translating LDAP requests into DAP requests and recasting the DAP results returned from the X.500 servers as LDAP results.

In addition, the promise of a global, interconnected X.500 directory didn't seem to be achievable, so the need for an X.500 directory server at all was questioned. The concept of a standalone, or native, LDAP server was born. The new server was dubbed slapd (for standalone LDAP daemon ), and with the aid of a grant from the National Science Foundation, it was quickly implemented. Figure 1.14 shows the revised LDAP system architecture.

Figure 1.14. LDAP System Architecture after the Introduction of slapd

While keeping the best pieces of the X.500 models (see Chapter 2 for more information), LDAP had now broken free of the last bit of X.500 and could proudly stand on its own. At the same time, referrals were added to the LDAPv2 protocol as an experimental extension to support an interconnected mesh of slapd directory servers. These changes moved LDAP out of its role as a simple, useful protocol for accessing X.500 directories to a much broader role as the foundation for a complete directory service. The first release of the U-M LDAP software to include slapd was the U-M LDAP 3.2 release in December 1995.

LDAP Momentum

LDAP's commercial success was assured in April 1996 when Netscape headed a coalition of more than 40 prominent software companies that endorsed LDAP as the Internet directory service protocol of choice. Today, companies such as Netscape, Sun Microsystems, and Novell produce a complete set of enterprise software centered on an LDAP-based directory service, and all major network application software vendors support LDAP in their client and server products.

LDAP Version 3 Developed

In 1997, LDAP entered an even more mature phase in its evolution with the release of version 3 of the protocol (LDAPv3). Mark Wahl, now with Sun Microsystems, along with some of the original LDAP authors and a cast of hundreds from across the Internet, contributed to the LDAPv3 specification. LDAPv3 improves on LDAPv2 in the following important areas:

• Internationalization . The UTF-8 (UCS Transformation Format 8) character set is used everywhere it is appropriate. Because UTF-8 is an encoding of the universal Unicode character set, all the characters used in every language in the world can be stored and manipulated by LDAPv3 directory servers and clients.

• Referrals . A standard mechanism for returning referrals to other servers was added, which allows LDAP servers to be deployed in a bottom-up fashion, just as World Wide Web servers are.

• Security . Support for Simple Authentication and Security Layer (SASL) and Transport Layer Security (TLS) was added to increase LDAP's security and allow a richer set of applications to be supported.

• Extensibility . LDAPv3 can be extended to support new operations, and existing operations can be extended through the use of controls, allowing LDAP to be enhanced incrementally to meet future needs.

• Feature and schema discovery . All LDAPv3 servers publish the versions of the LDAP protocol they support, along with other useful information, in a special directory entry called the root DSE (DSA-Specific, or Directory Server “Specific, Entry). In addition, LDAPv3 servers publish their supported directory schemas as a set of LDAP attributes. These features help LDAP clients and servers work together more intelligently.

At the time of this writing, the LDAPv3 specification encompasses the following RFCs:

• RFC 2251: Lightweight Directory Access Protocol (v3) . Describes the LDAP protocol itself.

• RFC 2252: Lightweight Directory Access Protocol (v3): Attribute Syntax Definitions . Defines attribute syntaxes used by LDAP and its applications, including how these values are represented as simple strings.

• RFC 2253: Lightweight Directory Access Protocol (v3): UTF-8 String Representation of Distinguished Names . Describes how distinguished names are represented as simple strings with the UTF-8 character set.

• RFC 2254: The String Representation of LDAP Search Filters . Defines a scheme for representing LDAP search filters (which may be complex Boolean expressions involving multiple terms) as simple strings for use in LDAP URLs and APIs.

• RFC 2255: The LDAP URL Format . Defines a Uniform Resource Locator (URL) scheme to represent LDAP search requests and referrals.

• RFC 2256: A Summary of the X.500(96) User Schema for Use with LDAPv3 . Provides a list of useful directory schemas (attributes and object classes) that are pulled from the X.500 specifications.

• RFC 2829: Authentication Methods for LDAP . Describes the required and optional authentication mechanisms that LDAPv3 servers and clients must support.

• RFC 2830: Lightweight Directory Access Protocol (v3): Extension for Transport Layer Security . Describes how to use Transport Layer Security (TLS) with LDAPv3 for authentication and privacy.

• RFC 3377: Lightweight Directory Access Protocol (v3): Technical Specification . Specifies the set of RFCs that define LDAP version 3 and addresses the "IESG Note" regarding the lack of satisfactory authentication mechanisms that is attached to RFCs 2251 through 2256.

Like their X.500 counterparts, developers of LDAP directories must absorb quite a few documents before they can successfully develop LDAP software. Fortunately, all RFCs are freely available on the Internet, and most of the RFCs related to LDAP are fairly short and easy to read. Unfortunately, some of the LDAP documents assume that the reader has knowledge that can be obtained only by consultation of the X.500 standards. For example, to understand all the details of the LDAP naming model (which specifies how to refer to and arrange directory information), you would need to read the X.500 documents.

In late 1997, LDAPv3 was approved as a proposed Internet Standard (the first step on its way to becoming a full Internet Standard), and in January 1998, Netscape shipped the first commercial LDAPv3 server, Netscape Directory Server 3.0. All serious directory services software vendors support LDAPv3 in their products today.

Some of the available LDAP directory services products are modeled after the original University of Michigan slapd concept and are built around a standalone LDAP server that includes a high-performance embedded database (many such products use some of the U-M code). In other cases, vendors of X.500 or other directory services strongly embraced LDAP and made it their directory access and operational protocol of choice. Some examples of general-purpose LDAP directory services available at the time of this writing include

• Netscape Directory Server

• Sun ONE Directory Server

IETF Directory Services Working Groups

Over the years , several different IETF working groups have dealt with LDAP. All these groups have been chartered within the Applications or Operations areas of the IETF.

The first group to deal with LDAP was OSI-DS. Some work that focused on adapting X.500 for the Internet, along with all the early DAS, DIXIE, and LDAP work, was done within the confines of OSI-DS.

In 1993 the directory services efforts in the IETF were split into several groups. These groups included the Access, Searching, and Indexing of Directories (ASID) group and the Integrated Directory Services (IDS) group. The core LDAPv3 work was done within the ASID group; IDS focused mainly on directory services issues that were not specific to LDAP, such as security and privacy.

Eventually, the ASID and IDS groups were disbanded, and a series of new working groups was chartered to continue LDAP's evolution. The LDAPEXT Working Group focuses on standardizing important extensions to LDAPv3 itself, such as authentication methods and server-side sorting, and it is likely to complete its work soon. The LDAP Duplication/Replication/Update Protocol (LDUP) group is working on server-to-server and server-to-client replication standards for LDAP directory services. The LDAPv3 Revision (LDAPBIS) Working Group is focused on revising the core LDAPv3 RFCs to advance them from Proposed to Draft Internet Standard status (one step closer to full Internet Standard). Thanks to the simple but flexible extension mechanisms included in LDAPv3, no one expects an LDAP version 4 to be needed anytime soon.

More information about these IETF working groups can be found on the IETF's World Wide Web site at http://www.ietf.org./html. charters /wg-dir.html.

• Microsoft Active Directory

• IBM Directory Server

• Novell eDirectory

• Oracle Internet Directory

• OpenLDAP slapd server

These directory products are generally designed to support a variety of Internet, intranet, and extranet applications. They usually provide good integration with other services that are based on Internet protocols, such as SMTP-based messaging servers, HTTP-based Web servers, and the like. They provide good administrative tools, are relatively inexpensive, and aim to be easy to deploy and maintain. Note that because these products do not all share a common design center, some are more suitable to certain kinds of directory deployments than others. For example, whereas Netscape Directory Server is designed to support mainly large intranet and e-commerce applications, Active Directory is designed to support mainly the administration of large Windows networks.

LDAP: Protocol or Directory Service?

Some people object to use of the term LDAP directory service . Why? Because, the argument goes, LDAP is merely a directory access protocol and therefore we should talk about only LDAP-enabled directory services ”for example, an X.500 directory can be LDAP-enabled, and Novell eDirectory is LDAP-enabled. So why do we use the term LDAP directory service throughout this book?

Since the introduction of the University of Michigan standalone LDAP server (slapd), and with the specification of LDAPv3 by the IETF, directory servers that speak LDAP can stand on their own and need not be dependent on a full-blown implementation of the X.500 standards or any other directory specification other than the LDAP RFCs and the parts of X.500 that LDAP depends on. Also, the term LDAP has become commonly used to refer to servers; for example, people often say "LDAP server" just like they say "FTP server." Therefore, the natural label for a directory service based on a collection of LDAP servers is LDAP directory service . In fact, we can't think of a better term to describe a pure LDAP solution such as was pioneered by the University of Michigan and Netscape!

The Key Advantages of LDAP

Today it is clear that LDAP has moved to the front and center of the online directory services space, and much excitement and energy are being put into developing and deploying LDAP directories. LDAP emerged from the rest of the pack to dominate the directory services space and capture the interest of information technology professionals because of the following factors:

• LDAP is simple, yet versatile . LDAP supports a wide variety of directory-enabled applications that have widely varying needs.

• LDAP is ubiquitous . A good implementation of LDAP was developed and distributed freely on the Internet by researchers at the University of Michigan, thus providing much of LDAP's early momentum. Today, LDAP implementations are available for every major and most minor computing platforms in use.

• LDAP directories are inexpensive and easy to understand . Organizations that choose LDAP directories find them to be relatively inexpensive to deploy and maintain. LDAP directory systems are simple enough that an army of consultants is not needed to understand and deploy them.

• LDAP directory services simply work better . The high reliability, performance, and scalability of LDAP directory products, combined with their general-purpose design, allow them to meet the most important directory services needs.

• LDAP has a lot of " mindshare ." Shortly after Netscape and dozens of other software companies listened to requests from their customers and put their support behind LDAP, interest in LDAP exploded. The hunger for a single directory standard is being filled by LDAP, and momentum continues.

In short, LDAP is making the dream of a universal, general-purpose directory service a reality on the Internet and within organizations.