Understanding and Deploying LDAP Directory Services > 12. Choosing Directory Products > Evaluation Criteria for Directory Software |
Evaluation Criteria for Directory SoftwareOne of the first tasks in choosing directory software is to develop evaluation criteria that takes into account your current and future needs. Whether you have no idea what software you might use or you already have a specific set of products in mind, evaluation criteria are very valuable . A good set of evaluation criteria helps you decide on, justify, and feel comfortable with your product choices. In this section, we provide some advice for developing evaluation criteria to fit your own situation. We divide the criteria into the following areas:
Each area is discussed in the following sections. As you read through these criteria, examine your own directory service needs and design to see which factors are most important to you. Your goal is to create a list of requirements and questions you have about the products. You should assign a priority or weight to each item on the list so you can make intelligent trade-offs later. Your list of evaluation criteria will serve as a tool for evaluating products and help you ask the right questions of vendors about what they can deliver. Keep in mind that the areas we cover in the following sections are not exhaustive and that there will likely be other factors that you may want to consider as well. Use this information only as a starting point to develop your own set of evaluation criteria. And don't neglect the areas in which you have no immediate needs; be sure to take future needs into account. Core FeaturesWhen looking at the core features of LDAP software products, your main task is to ensure that the product can meet the needs of all the directory-enabled applications you plan to deploy now and in the future. This is not a small task, of course, but the work done in Chapter 5 and throughout Part II should provide all the information you need to create a list of the features that are important to you. Some areas to consider include:
Management FeaturesYou will need to manage both the directory service itself and its contents. When examining the management features of software, focus on your most critical needs and look for flexibility to meet future needs. Areas to consider include
ReliabilityAll directory services are expected to be reliable so as not to inconvenience applications and halt business processes. Specific requirements, of course, vary widely. For example, if your directory will primarily serve as an electronic phone book, but everyone has paper copies of the same data available for use if your service is unavailable, reliability is not too critical. However, extranet applications used by customers and business partners to update their contact information require great reliability. Some reliability issues to consider when creating your evaluation criteria include
Performance and ScalabilityHigh performance and high scalability are very important in many directory deployments. The directory design process described in Part II of this book should help you expose all your performance and scalability needs. Areas to consider include:
Directory Performance Testing To determine how well a directory product will perform when you deploy it, you can conduct performance testing in a laboratory setting. Creating an accurate performance test that produces meaningful results is often difficult. To closely model your own directory data and directory-enabled applications, you may find that you need to develop your own custom benchmarking tools. This can be a lot of work,but it may be worth tackling up front. Understanding the performance characteristics of your directory software will help you make informed decisions as you finalize your design and deployment plans. DirectoryMark, jointly developed by Mindcraft and Netscape, is an off-the-shelf tool for benchmarking LDAP directory servers. This tool is highly configurable, so you may be able to use it to simulate the client load you expect to impose on your servers. Connect to Mindcraft's Web site at http://www.mindcraft.com/benchmarks/dirmark/ for more information. SecurityAs discussed in Chapter 11, "Privacy and Security Design," security needs vary widely from one directory service deployment to another. Without exception, though, security should be an important part of your product evaluation criteria. Areas to consider include:
Standards ConformanceTo most people, standards are not interesting for their own sake; it's the products that adhere to standards that are more interesting. Standards-compliant software is important to you because it provides increased flexibility, better interoperability, more customer choice, and a proven, well- understood core feature set. Standards documents are typically very technical and hard to understand. Although you probably do not need to read and understand all the standards documents, you should be aware of all the emerging and ratified standards so that you can ask software vendors if they comply with them. Because no available product supports all the standards, you need to determine which standards are most critical to you. You can do that by understanding what each standard specifies and how it aligns with your own directory needs. The most important group that creates standards for LDAP is the Internet Engineering Task Force (IETF), which is the standards-setting body for the Internet. IETF standards are published as a series of documents called requests for comments (RFCs). Note that not all RFCs are destined or even intended to become standards, and that specifications are first published as Internet drafts. (See Chapter 2, "A Brief History of Directories," for more information on the IETF and how it operates.) Other important standards are produced by industry consortiums, and some de facto standards are developed by the leading directory services vendors. Some of the directory service standards you should consider for your evaluation criteria list include:
InteroperabilityStandards conformance is an important part of any product's interoperability story. However, if you require interoperability with specific products, you should take that into account when developing your evaluation criteria. Areas to consider include:
CostThe most obvious cost is that of the software itself, but you should also consider the total cost of buying, deploying, supporting, and maintaining your directory service and the applications that surround it. Chapter 14 covers cost analysis in depth, but some general areas to examine when creating your list of evaluation criteria include the following:
Keep in mind that some of your costs will be hard to quantify, especially if you have not yet deployed your directory service. Talk to people within other organizations who have already deployed a similar product to gather more concrete information about costs. Flexibility and ExtensibilityIt is unlikely that an existing product can meet all your needs out of the box. For this reason, and because you can't anticipate all your future needs, it is important to choose directory products that are flexible and extensible. Areas to consider include the following
Other ConsiderationsThere are a few additional areas to examine as you construct your evaluation criteria. These issues are primarily related to the future of the product and the vendor that provides it. Consider each of the following topics:
An Evaluation Criteria ExampleIn this section, we present an example evaluation criteria list for directory server software. The sample criteria were developed for a fictitious company called Airius Airlines that is deploying a directory service for the first time. The focus of the deployment is support of directory-enabled intranet applications. The first two applications that are expected to come online are an electronic phone book and an email delivery service. Airius currently has only 5,000 employees but expects to grow rapidly over the next few years . The sample criteria are presented as a series of tables that the Airius Airlines directory services planning team created using a spreadsheet program like Microsoft Excel. Each row in the spreadsheet lists a specific characteristic used to evaluate each candidate product. A description and a weight are provided for each item. The weight is a number from 1 (not very important) to 10 (extremely important) that captures the importance of the item. Items that are extremely critical ( must-have features) are marked with an asterisk (*) so that they can be spotted more easily when reviewing the results of the evaluation. The right side of each table provides room to evaluate two products (Product A and Product B). Each product is given a rating for each characteristic. A score ranging from 0 (poor) to 100 (best) is calculated by multiplying each item's weight by a product's rating. By adding all the scores, an objective number is produced that can be consulted when making a final product choice. Airius's evaluation criteria for core directory server features are shown in Table 12.2. Table 12.2. Sample evaluation criteria for core directory server features
To illustrate how the scoring system works, consider the first row in Table 12.2: support for all basic LDAP operations. This feature was given a weight of 7 (out of a possible 10) by Airius because it is fairly important to the company (it plans to eventually make full use of its directory service, including allowing employees to update their own contact information). Product A received a rating of 8 (out of a possible 10) because it does support all the basic LDAP operations but falls short on complete implementation of some of the added features of LDAPv3. The item score for Product A is calculated by multiplying 7 (the weight) by 8 (the product rating) to arrive at 56. Product B received a rating of 3 because it supports only search operations. Product B's score is 21 (7 times 3). Tip For an objective numeric evaluation such as this to work, you need to ensure that the grading is done fairly. In practice, the same group of people must assign the ratings for each criterion, or you must spell out in great detail how the products are to be rated. If you can spare the resources, a good approach is to have two or more people rate each product independently and then compare the results to ensure that all parties basically agree. Because Airius plans to develop its own phone book application and eventually create dozens of directory-enabled applications, flexibility and extensibility of the directory software are very important. Table 12.3 shows Airius's evaluation criteria for this area. Table 12.3. Sample criteria for flexibility and extensibility
The sample evaluation criteria we have presented for Airius Airlines are, of course, far from complete. When you develop your own evaluation criteria, you should include all the areas we discussed in the previous sections, such as security, interoperability, and cost. The evaluation criteria tables shown here could be improved by adding a column to record specific comments or notes about each product feature.
|
Index terms contained in this sectionaccess controldirectory software security administration delegating directory software security ADSI algorithm acceleration directory software security APIs standards directory software concordance authentication directory software security automated failover directory software backups data evaluating directory software benchmarks results directory software performance choosing directory software core features 2nd costs 2nd 3rd evaluation criteria 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th flexibility 2nd 3rd interoperability 2nd management features 2nd performance and scalability 2nd 3rd 4th 5th reliability 2nd security 2nd standards conformance 2nd 3rd 4th 5th 6th clients interfaces directory software configuring directory software connections simultaneous directory software performance content manipulation tools directory software continuous operattion directory software core features evaluating directory software 2nd distributed directory support hardware and software platform support replication support costs evaluating directory software 2nd 3rd deployment hardware license structures maintenance support training data server stores directory software DEN (Directory Enabled Networking) deployment directory software costs directories software core features 2nd costs 2nd 3rd evaluation criteria 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th flexibility 2nd 3rd interoperability 2nd management features 2nd performance and scalability 2nd 3rd 4th 5th reliability 2nd security 2nd standards conformance 2nd 3rd 4th 5th 6th Directory Enabled Networking, see DEN DirectoryMark testing directory performance distributed directories evaluating directory software documentation directory software evaluating encryption directory software security evaluating directory software 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th core features 2nd costs 2nd 3rd examples 2nd 3rd 4th flexibility 2nd 3rd interoperability 2nd management features 2nd performance and scalability 2nd 3rd 4th 5th reliability 2nd security 2nd standards conformance 2nd 3rd 4th 5th 6th extensions LDAPv3 directory software concordance flexibility evaluating directory software 2nd 3rd hardware algorithm acceleration directory software security directory software costs directory software platform support IETF IMC (Internet Mail Consortium) importing data evaluating directory software installing directory software interfaces client directory software Internet protocol standards directory software concordance Internet Mail Consortium (IMC) interoperability evaluating directory software 2nd application support metadirectories synchronization tools JNDI latency of operations directory software performance LDAPv3 extensions directory software concordance LDIF directory software concordance licensing directory software costs maintainence directory software maintenance directory software costs management features evaluating directory software 2nd client interfaces installation procedures maintenance and configuration tools remote administration scriptable administration and content tools user-specific restraints metadirectories directory software interoperability monitoring servers directory software montiring directory software performance evaluating directory software 2nd 3rd 4th 5th benchmark results latency of operations simultaneous connections testing 2nd throughput of operations tuning servers protocols Internet directory software concordance reliability evaluating directory software 2nd automated failover continuous operation server data stores server monitoring tools remote administration directory software replication directory software evaluating restraints user directory software RFCs SASL directory software concordance scalability evaluating directory software 2nd 3rd 4th 5th benchmark results latency of operations server and client software simultaneous connections throughput of operations tuning servers schema standards directory software concordance scriptable administration tools directory software SDKs within clients directory software concordance security evaluating directory software 2nd access control authentication methods delegating administration encryption hardware algorithm acceleration SSL and TLS standards directory software concordance servers data stores directory software monitoring directory software tuning directory software performance simultaneous connections directory sofware performance site licensing directory software costs SNMP SNMP MIB directory software concordance software directory core features 2nd evaluation criteria 2nd 3rd 4th 5th 6th 7th 8th 9th 10th 11th 12th 13th 14th 15th 16th 17th 18th 19th 20th flexibility 2nd 3rd interoperability 2nd management features 2nd 3rd 4th 5th performance and scalability 2nd 3rd 4th 5th reliability 2nd security 2nd standards conformance 2nd 3rd 4th 5th 6th platform support evaluating directory software SSL directory software security standards conformance evaluating directory software 2nd 3rd 4th 5th 6th LDAP Internet protocols LDAP SDK and client APIs LDAPv3 extensions LDIF SASL schema security SNMP MIB X.500 Y2K compliance support directory software costs synchronization directory software interoperability technical support directory software evaluating testing directory software DirectoryMark throughput directory software performance TLS directory software security training directory software costs tuning servers directory software performance users restraints directory software X.500 standards directory software concordance Y2K compliance standards directory software concordance |
2002, O'Reilly & Associates, Inc. |