| One of the first tasks in choosing directory software is to develop evaluation criteria that take into account your current and future needs. Whether you have no idea what software you might use or you already have a specific set of products in mind, evaluation criteria are valuable . A good set of evaluation criteria helps you choose, justify, and feel comfortable with a collection of products. In this section we provide some advice for developing evaluation criteria to fit your own situation. We divide the criteria into the following areas: Each area is discussed in the following sections. As you read through these criteria, examine your own directory service needs and your design to see which factors are most important to you. Your goal is to create a list of requirements and questions you have about the products. You should assign a priority or weight to each item on the list so that you can make intelligent trade-offs later. Your list of evaluation criteria will serve as a tool for evaluating products and help you ask the right questions of vendors about what they can deliver. Keep in mind that the areas we cover in the following sections are not exhaustive and that there will be other factors you should consider as well. Use this information only as a starting point to develop your own set of evaluation criteria. And don't neglect the areas in which you have no immediate needs; be sure to take future requirements into account. Core Features When you're looking at the core features of LDAP software products, your main task is to make sure that the product can meet the needs of all the directory-enabled applications you plan to deploy now and in the future. This is not a small task, of course, but the work done in Chapter 6, Defining Your Directory Needs, and throughout the rest of Part II, Designing Your Directory Service, should provide all the information you need to create a list of the features that are important to you. Areas to consider include the following: -
Support for the hardware and software platforms you use now or plan to use in the future. -
Support for all the core LDAP operations and extensions needed by your applications. -
Support for alternative data formats and access protocols such as DSML that are needed by your applications. -
Replication features, including support for all the topologies your design uses, the capability to provide uninterrupted service to client applications even if one or more replicas fail, flexible scheduling policies, and so on. Chapter 11, Replication Design, includes detailed product criteria for replication. -
Support for distributed directories, if required by your design. Such support usually comes from LDAP referrals or server-to-server chaining of requests . -
Data import, maintenance, and backup features. -
The quality, availability, and extra cost of documentation and support. For example, will the vendor provide a dedicated account representative and a dedicated technical contact for you? And what kind of response time will the vendor guarantee? Management Features You will need to manage both the directory service itself and its contents. When examining the management features of software, focus on your most critical needs and look for flexibility to meet future requirements. Areas to consider include the following: -
Simplicity and flexibility of installation scripts and procedures. -
Availability of good tools for configuring, monitoring, and maintaining all aspects of the service itself. Look for tools that are easy to use for complicated tasks such as security, access control, and performance tuning. Often both graphical and command-line tools are desirable, depending on your situation and preferences. -
Availability of a variety of LDAP client interfaces that users and administrators can employ to find and manipulate directory content such as person entries, group entries, and any other data you plan to store in your directory. The best client interfaces can be customized to target specific needs. -
Scriptable administration and content manipulation tools that can be used to automate frequent tasks. -
The capability to manage user -specific constraints such as password policies, resource usage quotas, and other items. -
Remote administration capabilities. Reliability All directory services are expected to be reliable so as not to inconvenience applications and halt business processes. Specific requirements, of course, vary widely. For example, if your directory will serve primarily as an electronic phone book but everyone has paper copies of the same data available for use if your service is unavailable, reliability is not so critical. However, an extranet application used by customers and business partners to update their contact information requires a very reliable service. Here are some reliability issues to consider when creating your evaluation criteria: -
Support for 24x7 (continuous) operation. Features such as the capability to back up data and change the server configuration without affecting the running service can be important. Common configuration changes include extending the directory schema, adding indexes to improve performance, and making temporary changes needed during upgrade and maintenance activities (such as a change in the TCP port on which a server listens). -
A robust, transactional server data store that is self-repairing and recovers automatically from temporary failures such as power outages. -
Support for automated failover to minimize downtime in the event of software or hardware failures. With directory server software, such support can be provided as a by-product of support for third-party or built-in high-availability solutions, advanced replication features such as multimaster topologies and hot standbys, and client-side support for transparent failover to a standby server. -
Directory service monitoring tools. As discussed in Chapter 19, Monitoring, monitoring is essential for quickly identifying and correcting problems. Support for Simple Network Management Protocol (SNMP), server error logging facilities, and the availability of proprietary monitoring tools can all be helpful. Otherwise you need to write your own monitoring tools. (If you think you'll need to do this, make sure that appropriate hooks and SDKs are provided with the directory products you choose.) Performance and Scalability High performance and high scalability are important in many directory deployments. The directory design process described in Part II, Designing Your Directory Service, should help you expose all your performance and scalability requirements. Areas to consider include the following: -
Latency associated with directory operations. For example, how fast must the directory service respond to a search or modify request? -
Throughput of directory operations. How many operations can a directory server handle in a given time period? How fast can a client application or SDK send LDAP requests and process the responses? How many search, add, and modify requests can the server process per second? Be sure to take into account the type and mix of operations you expect to have in your environment. For example, server performance for searches is typically much greater than it is for adds and modifies; thus if your needs imply that there will be a lot of add and modify traffic, you should pay special attention to the write performance of the directory server software. -
The capability of the server to handle many simultaneous LDAP connections while providing good performance to all the clients . -
The capability to monitor and tune the performance of the directory server to meet your needs. This is important in determining the characteristics of the hardware you should purchase, as well as in optimally meeting the requirements of your most important directory-enabled applications. -
The capability of server and client software to maintain good performance as scale increases . Dimensions to consider include the number and size of entries, number and size of attribute values, number of access control rules, number of replicas, and number of directory partitions. -
The availability of performance information such as benchmark results and performance-focused documentation. If performance is important to you but a directory software vendor has little to say about it, you may need to look at other products or conduct your own benchmark tests. Directory Performance Testing To determine how well a directory product will perform when you deploy it, you can test its performance in a laboratory setting. Creating an accurate performance test that produces meaningful results is often difficult. To closely model your own directory data and directory-enabled applications, you may need to develop your own custom benchmarking tools. This can be a lot of work, but it may be worth tackling up front. Understanding the performance characteristics of your directory software will help you make informed decisions as you finalize your design and deployment plans. DirectoryMark, jointly developed by Mindcraft and Netscape, is an off-the-shelf tool for benchmarking LDAP directory servers. This tool is highly configurable, so you may be able to use it to simulate the client load you expect to impose on your servers. Connect to Mindcraft's Web site at http://www.mindcraft.com/benchmarks/dirmark for more information. | Security As discussed in Chapter 12, Privacy and Security Design, security requirements vary widely from one directory service deployment to another. Without exception, though, security should be an important part of your product evaluation criteria. Areas to consider include the following: -
Access control capabilities, including adequate flexibility and granularity to meet your privacy, security, and data management requirements. -
The capability to create administrative domains and delegate administration of directory content to independent administrators and end users as needed. Graphical interfaces that support delegation, such as Netscape's Delegated Administrator component, are also required. -
Auditing features to maintain a record of all access and changes to directory data. Ideally, access to the audit information itself should be controlled through access control and separation of administrative domains. -
The authentication methods supported by the client and server software (for example, LDAP basic authentication, various Simple Authentication and Security Layer [SASL] methods , and certificate-based authentication). -
The encryption capabilities for LDAP data streams and directory data stores, including support for Secure Sockets Layer (SSL) and Transport Layer Security (TLS). -
The capability to use SSL or TLS to protect replication updates in transit and to protect operations that cross server boundaries when server-to-server chaining is being used. -
Support for hardware acceleration of cryptographic algorithms. This is important for servers that must perform a lot of encryption or that routinely handle digital certificates. Standards Compliance To most people, standards are not interesting for their own sake; the products that adhere to standards are more interesting. Standards-compliant software is important to you because it provides increased flexibility, better interoperability, more customer choice, and a proven, well- understood core feature set. Standards documents are typically technical and difficult to understand. Although you probably do not need to read and understand all the standards documents, you should be aware of all the emerging and ratified standards so that you can ask software vendors whether they comply with them. Because no available product supports all the standards, you need to determine which standards are most critical to you. You can do that by understanding what each standard specifies and how it aligns with your own directory requirements. The most important group that creates standards for LDAP is the Internet Engineering Task Force (IETF), which is the standards-setting body for the Internet. IETF standards are published as a series of documents called Requests for Comments ( RFCs ). Note that not all RFCs are destined or even intended to become standards, and that specifications are first published as Internet Drafts. (See Chapter 1, Directory Services Overview and History, for more information on the IETF and how it operates.) Other important standards are produced by industry consortia, and some de facto standards are developed by the leading directory services vendors. Directory service standards that you should consider for your evaluation criteria list include the following: -
The core LDAP Internet protocol standards, including the complete set of LDAPv3 RFCs. The relevant documents are RFCs 2251 through 2256, 2829, 2830, and 3377. Because these standards are the building blocks on which all LDAP- related software is built, most products claim to support these RFCs. However, be sure to ask your software vendor exactly which aspects of these LDAP standards they do not support. -
Related Internet standards that LDAP has adopted, including SASL (which is defined in RFC 2222), SASL digest authentication (RFC 2831), and the Directory Services Monitoring Management Information Base (MIB; RFC 2605). You can use SASL to integrate with a specific authentication scheme. Support for the SNMP MIB for directory services helps you integrate your servers with off-the-shelf network management systems (NMSs). -
LDAPv3 extensions, such as those for dynamic entries (RFC 2589), the password modify extended operation (RFC 3062), storing vendor information in the root DSE (RFC 3045), server-side sorting of search results, Virtual List View, discovery of LDAP services using the Domain Name System (DNS), connectionless LDAP (over UDP), authentication response control, proxied authorization control, LDAP password policy, schema update procedures, standard access control, and standard replication protocols. At the time of this writing, many of these extensions are available only as Internet Drafts; they have not yet been published as RFCs. In general, you should examine each extension to see whether you need the feature it provides within your own directory deployment. -
The LDAP Data Interchange Format (LDIF), which defines a standard format for representing directory entries and changes to entries. LDIF is defined in RFC 2849. -
LDAP application programming interface (API) standards within SDKs and LDAP clients. Internet standard LDAP APIs for C/C++ and Java are being developed by the IETF. Note that several proprietary APIs also exist, such as Microsoft's Active Directory Services Interface (ADSI) and JavaSoft's Java Naming and Directory Interface (JNDI). However, the time and cost incurred in developing LDAP applications can be reduced through the use of standard APIs. See Chapters 21, Developing New Applications, and 22, Directory-Enabling Existing Applications, for more information on available APIs and SDKs. -
Schema standards, such as those being developed by the Directory Enabled Network (DEN) initiative. Good LDAP products can be extended to accommodate a wide variety of schemas. -
Security standards, such as X.509 for certificates and the SSL and TLS protocols. Security is an area in which standards are especially important, not just for basic interoperability, but also for easier evaluation of the security provided by the products. For example, all SSL version 3.0 implementations support a certain level of security, but for proprietary security schemes, you need to trust the software vendor to provide adequate capabilities. Security is an area that is not well understood, but standards reduce the learning curve and allow you to transfer knowledge from one product to another much more easily. -
XML directory standards, such as DSML, which is being defined within the confines of the OASIS Directory Services Technical Committee, and Security Assertion Markup Language (SAML), which is being defined by the OASIS XML-Based Security Services Technical Committee. -
X.500 directory service standards. Because LDAP depends to some extent on the X.500 standards for its naming, information, and operations models, the X.500 standards are relevant to all LDAP directories. X.500 also specifies standards for access control and replication that have been adopted by some LDAP vendors. More information on X.500 can be found in Chapter 1, Directory Services Overview and History. -
Any other standards or certification requirements, such as Y2K compliance, certification by an operating system or hardware vendor, or support for industry-leading third-party products such as high-availability and backup solutions. Interoperability Standards compliance is an important part of any product's interoperability story. However, if you require interoperability with specific products, you should take that into account when developing your evaluation criteria. Areas to consider include the following: -
Availability and support for directory-enabled applications that you plan to deploy. Some vendors, such as Sun, offer a complete line of LDAP-enabled Web and messaging products designed to work with their own directory service products. When working with products from two or more vendors, you probably should verify interoperability yourself. -
Availability of synchronization tools for data sources with which your directory service needs to interoperate . For example, you may have a requirement to mirror the data held in a cc:Mail address book in your LDAP directory; therefore, you would need to buy or build a cc:Mail-to-LDAP synchronization tool. See Chapter 23, Directory Coexistence, for more information on integrating your directory service with other data sources. -
Availability of metadirectory solutions to facilitate the joining and synchronization of disparate directories and data stores. See Chapter 23, Directory Coexistence, for more information on metadirectories . -
Proof of interoperability. This can come in the form of results from vendor-to-vendor tests performed in an ad hoc fashion or at an LDAP interoperability testing forum, such as the DirConnect and LDAP Plugfest events sponsored by the Open Group. Cost The most obvious product-related cost is that of the software itself, but you should also consider the total cost of buying, deploying, supporting, and maintaining your directory service and the applications that surround it. Chapter 15, Analyzing and Reducing Costs, covers cost analysis in depth, but some general areas to examine when you're creating your list of evaluation criteria include the following: -
Software costs . Be aware of the different cost structures under which the software may be sold ”per server installation, per CPU, per user, per entry, yearly lease, unlimited use, and so on. Also don't forget to take into account ongoing upgrade and support costs and the licensing costs associated with operating systems or any other software the directory software depends on. For example, at the time of this writing, Microsoft requires that you upgrade all your Active Directory domain controllers to Windows 2000 Server (or later versions) to take advantage of some Active Directory features. -
Hardware costs . Different directory service products require different hardware, some of which may be much more expensive than others. -
Deployment costs . The costs of hardware, personnel, software deployment, and so on. -
Maintenance costs . The costs of day-to-day tasks such as performing backups , maintaining the content of the directory servers, monitoring it for failures, and any other ongoing activities. Look for products that allow management tasks to be easily automated. -
Training costs . These include costs to train administrators of the system, end users, and directory content administrators. -
Support costs . Products that are easy to learn, use, and manage cost less to support. Keep in mind that some of your costs will be difficult to quantify, especially if you have not yet deployed your directory service. Talk to people in other organizations who have already deployed a similar product to gather more concrete information about costs. Flexibility and Extensibility It is unlikely that an existing product will be able to meet all your needs out of the box. For this reason, and because you can't anticipate all your future needs, it is important to choose directory products that are flexible and extensible. Areas to consider include the following: -
Configurability . Look for the capability to selectively enable and disable features, the capability to adapt the product to work well on your available hardware and operating systems, and the ease with which you can make configuration changes. For example, if you have remote offices in your organization, you may want to run some replicated servers in those offices on inexpensive machines that have moderate amounts of memory, CPU, and disk capacity. For your main campus, you may want to run the same directory server software but tune it to take advantage of large machines that have a lot of memory, CPU, and disk capacity. -
Flexibility . To support a variety of directory-enabled applications, you want to be able to extend directory schemas easily, add or remove indexes, and otherwise configure directory servers to provide optimal performance to support an application's queries. For example, if you bring a high-performance, directory-enabled application online in the future, it will probably come with its own schema extensions and may require the tuning of your servers to achieve optimal performance. -
Extensibility . You can extend some products by writing scripts or creating software plug-ins that alter the behavior of directory clients and servers. Netscape Directory Server, for example, provides a complete set of well-documented plug-in interfaces. This kind of extensibility can help when you need to support an unusual application or when you want to adapt the directory service to fit your organization's data maintenance policies. Other Considerations There are a few additional areas to examine as you construct your evaluation criteria. These issues are related primarily to the future of the product and the vendor that provides it. Consider each of the following topics: -
The future of the product . Is it evolving and moving in the direction you expect your organization to move? Is the product important enough to the organization that develops it that you can count on continued availability and support? -
Completeness of product line . A vendor that sells a complete line of directory-enabled applications and has made LDAP a key part of its corporate strategy may be able to meet more of your needs and serve as a good partner for all your directory services efforts. On the other hand, it is unlikely that you will buy all your directory products from only one vendor. -
Industry and developer mindshare . Look at how much support there is for a product outside the company or organization that provides it. A product that is used and supported by many third parties has a better chance of meeting the needs of most people. On the other hand, if your requirements are specialized, this factor isn't as important. -
The ability to leverage other products and services the vendor has to offer . For example, Netscape Directory Server promises easy integration of your intranet or extranet site with various AOL Time Warner online services and content. (AOL Time Warner is the parent company of Netscape, so this is not a surprise.) -
General vendor issues . Make sure that the vendor stands behind the products it sells. Determine whether the vendor has a viable business plan so that you can be assured that it will be around to support you. Examine the vendor's track record: What experience have other people had with the vendor? An Evaluation Criteria Example In this section we present a sample evaluation criteria list for directory server software. The sample criteria were developed for a fictitious company called On Time Package Delivery that is deploying a directory service for the first time. The focus of the deployment is support of directory-enabled intranet applications. The first two applications that are expected to come online are an electronic phone book and an e-mail delivery service. On Time currently has only 5,000 employees but expects to grow rapidly in the next few years . The sample criteria are presented as a series of tables that the On Time directory services planning team created using a spreadsheet program similar to Microsoft Excel. Each row in the spreadsheet lists a specific characteristic used to evaluate each candidate product. A description and a weight are provided for each item. The weight is a number from 1 (not very important) to 10 (extremely important) that captures the importance of the item. Items that are extremely critical ( must-have features ) are marked with an asterisk (*) so that you can spot them more easily when reviewing the results of the evaluation. A score of zero for a must-have feature would immediately eliminate a product from consideration. The right side of each table provides room to evaluate two products (A and B). Each product is given a rating for each characteristic. A score ranging from 0 (poor) to 100 (best) is calculated by multiplication of each item's weight ( W ) by a product's rating ( R ). Adding together all the scores produces an objective number that you can consult when making a final product choice. Table 13.2 shows On Time's evaluation criteria for core directory server features. As illustration of how the scoring system works, consider the first row in Table 13.2: support for all basic LDAP operations. This feature was given a weight of 7 (out of a possible 10) by On Time because it is fairly important to the company (On Time plans to make full use of its directory service eventually, including allowing employees to update their own contact information). Product A received a rating of 8 (out of a possible 10) because it supports all the basic LDAP operations but falls short on complete implementation of some of the added features of LDAPv3. The item score for Product A is calculated by multiplication of 7 (the weight) by 8 (the product rating) to arrive at 56. Product B received a rating of 3 because it supports only search operations. Product B's score is 21 (7 x 3). Table 13.2. On Time's Evaluation Criteria for Core Directory Server Features | | | Product A | Product B | | Feature | Weight ( W ): 0 “10 | Rating ( R A ): 0 “10 | Score ( W x R A ) | Rating ( R B ): 0 “10 | Score ( W x R B ) | | Basic LDAP operations | 7 | 8 | 56 | 3 | 21 | | LDAPv3 Virtual List View extension (for phonebook application) | 5 | 10 | 50 | 10 | 50 | | *Runs on Windows 2000 Server | 10 | 10 | 100 | 10 | 100 | | Runs on HP/UX 11.11 | 5 | 10 | 50 | | | | Multimaster replication | 4 | | | 9 | 36 | | *Single-master replication | 10 | 10 | 100 | 10 | 100 | | Quality of design and deployment documentation | 6 | 2 | 12 | 7 | 42 | | Quality of administration and configuration documentation | 8 | 4 | 32 | 8 | 64 | | Total score (out of 550) | | | 400 | | 413 | Tip For an objective numeric evaluation such as this to work, you need to ensure that the grading is done fairly. In practice, the same group of people must assign the ratings for each criterion, or you must spell out in great detail how the products are to be rated. If you can spare the resources, a good approach is to have two or more people rate each product independently and then compare the results to ensure that all parties basically agree.
Because On Time plans to develop its own phone book application and eventually create dozens of directory-enabled applications, flexibility and extensibility of the directory software are very important. Table 13.3 shows On Time's evaluation criteria for this area. The sample evaluation criteria we have presented are, of course, far from complete. When you develop your own evaluation criteria, you should include all the areas we discussed in the previous sections, such as security, interoperability, and cost. The evaluation criteria tables shown here could be improved by the addition of a column to record specific comments or notes about each product feature. Table 13.3. On Time's Criteria for Flexibility and Extensibility | | | Product A | Product B | | Criterion | Weight ( W ): 1 “10 | Rating ( R A ): 0 “10 | Score( W x R A ) | Rating ( R B ): 0 “10 | Score( W x R B ) | | Ease of configuration | 4 | 5 | 20 | 7 | 28 | | Flexibility of configuration | 7 | 9 | 63 | 5 | 35 | | Will run on low-end server machines | 7 | 7 | 49 | 6 | 42 | | *Can be optimized for new applications: tunable indexing | 9 | 8 | 72 | 5 | 45 | | *Can be optimized for new applications: extensible schemas | 10 | 8 | 80 | 4 | 40 | | Server-side plug-in APIs | 2 | 9 | 18 | 2 | 4 | | Total score (out of 390) | | | 302 | | 194 | |
