Creating LDIF for Benchmarks
| The MakeLDIF 1.3 program is an LDIF structure and data generation tool for testing the performance of the Sun ONE Directory Server version 5.2. This program provides the ability to generate LDIF files that can be imported into the Sun ONE Directory Server for testing purposes. The information that MakeLDIF uses to generate the entries is specified using a template file that can be customized to produce LDIF files that can be used in a test harness for benchmark performance validation tests. The MakeLDIF program is available for download. See "Obtaining the Downloadable Files for This Book" on page xxvii. The MakeLDIF ProgramThe MakeLDIF program is a Java-based utility for generating LDIF files. It can be used to generate sample data to import into an LDAP directory server, such as the Sun ONE Directory Server software. Although other utilities exist for this purpose (for example, dbgen.pl), MakeLDIF offers a number of powerful features not available in other tools:
MakeLDIF can easily be used as a standalone utility for generating LDIF data for a number of purposes. However, because it has been designed for use with the SLAMD distributed load generation engine (covered later in this chapter), it offers a number of features that make it especially useful for generating data to use in a directory server for running many of the SLAMD jobs that work with LDAP directory servers. Installing MakeLDIFThe only requirement for using MakeLDIF is that a Java runtime environment be present on the target system. MakeLDIF has been designed to work with Java versions 1.2 or higher, although Java 1.4 or higher is recommended for best performance. The following procedure assumes that you have obtained the downloadable MakeLDIF-1.3.tar.gz file (see "Obtaining the Downloadable Files for This Book" on page xxvii). To Install MakeLDIF
Running MakeLDIFThe following examples show you how to run MakeLDIF in various ways, depending on your environment. The minimum required command line to run MakeLDIF is: $ java -jar MakeLDIF.jar MakeLDIF -t template -o output_filename template is the path to the template file that describes the way in which the LDIF file should be generated and output_filename is the path to the output file that is created. In the following example, the MakeLDIF command uses the template defined in the file example.template and writes the output into the file example.ldif . $ java -jar MakeLDIF.jar -t example.template -o example.ldif If the Java executable is not currently in your path, you can specify the absolute path to the Java executable you wish to use. $ /usr/java/bin/java -jar MakeLDIF.jar -t template -o output_filename If for some reason the Java runtime environment does not support the use of the --jar option, you can instead place the MakeLDIF.jar file in the Java classpath and run it like in the following example: $ java -cp MakeLDIF.jar MakeLDIF -t template -o output_filename Although only the -t and -o options were illustrated , a number of command-line arguments may be used with MakeLDIF to customize its behavior. The full set of supported arguments is shown in TABLE 9-1. Table 9-1. MakeLDIF Options
The Template FormatAs mentioned earlier, MakeLDIF uses template files to define the way in which the LDIF files should be generated. This is a much more powerful approach than those taken by other LDIF generators, because it allows for a great deal of flexibility without the need to alter any code to produce the desired result. The template files contain three sections:
The remainder of this section discusses each separately. Customizing the Template File for MakeLDIFThe MakeLDIF program uses a template file to customize LDIF files. The use of the template file makes it possible to customize the location and kinds of entries that are generated by the MakeLDIF program. Two kinds of entries can be written into a template file: branch entries and template entries . It is also possible to define variables that may be used throughout the branch and template entries. Global Replacement VariablesReplacement variables define strings of text that are referenced later in the template file and that are automatically replaced as each line is read into memory. MakeLDIF branch entries example: define suffix=dc=example,dc=com In this example, define creates a variable called suffix with a value of dc=example,dc=com (the first equal sign is the delimiter between the variable name and the value). Once that definition is made, any occurrence of [suffix] in the template file is automatically replaced with dc=example,dc=com . As noted earlier, this replacement occurs as each line of the token file is read into memory, which means that it is possible to perform these replacements in branch definitions and other places where other tokens are not parsed. Note The parsing algorithm used by MakeLDIF uses the [ character to denote the beginning of a replacement variable usage. If it is necessary to have the [ character included in the template file, but not in the context of a variable reference, it should be escaped with the backslash character ( \[ ). This signifies that the bracket is not to be treated as part of a variable reference. The backslash immediately before the opening bracket is removed. Branch EntriesBranch entries define a hierarchical branch of a directory server DIT to include in the LDIF file and may be associated with zero or more templates that specify the kinds of entries to create beneath that branch in the hierarchy. A branch entry with no subordinate entries generated by a template might look like: branch: dc=example,dc=com This creates an entry that looks like this: dn: dc=example,dc=com objectclass: top objectclass: domain dc: example The basic structure of the entry is defined by the RDN attribute of dc specified in the DN of the branch definition. MakeLDIF automatically associates the dc RDN attribute with the domain object class. It has similar default definitions for other common RDN attributes in branch entries:
In addition, it is possible to use any other kind of RDN attribute for a branch entry. For branch entries with an RDN attribute other than one specified above, the entry is created with the extensibleObject object class. It is possible to customize the information contained in a branch entry by adding the additional information below the branch definition. Example: branch: dc=example,dc=com description: This is the description. The preceding example produces an entry that looks like: dn: dc=example,dc=com objectclass: top objectclass: domain dc: example description: This is the description. Note All of the branch entries defined so far have created only a single entry in the directory with limited control over the information contained in that entry. However, this does not have to be the case. If one or more subordinateTemplate values are specified, it is possible to create additional entries below this branch. Example: branch: ou=People,dc=example,dc=com subordinateTemplate: person:1000 The preceding example causes the ou=People,dc=example,dc=com entry to be created, but also creates 1000 entries below it based on the person template. If you wish to have multiple kinds of entries created below a single branch, you can specify each kind of entry with a different subordinateTemplate definition. For example: branch: ou=People,dc=example,dc=com subordinateTemplate: person:1000 subordinateTemplate: certificatePerson: 100 Branch entries are not limited to just one subordinateTemplate definition. It is possible to specify multiple subordinateTemplate definitions by simply including them on separate lines of the branch definition. The example above creates 1000 entries based on the person template and an additional 100 entries based on the certificatePerson template: Template EntriesThe heart of the MakeLDIF template file are the actual template definitions. Template definitions define the structure of the entries that will be generated. They specify the set of attributes to include in the entries and the types of values that those attributes should contain. The specification of the values is handled through the use of tags that are parsed by MakeLDIF and replaced with the appropriate values for those tags. A sample template entry might look like this: template: person rdnAttr: uid objectclass: top objectclass: person objectclass: organizationalPerson objectclass: inetOrgPerson givenName: first sn: last cn: {givenName} {sn} initials: {givenName:1}{sn:1} uid: {givenName}.{sn} mail: {uid}@[maildomain] userPassword: <random:alphanumeric:8> telephoneNumber: <random:telephone> homePhone: <random:telephone> pager: <random:telephone> mobile: <random:telephone> employeeNumber: <sequential:100000> street: <random:numeric:5> <file:streets> Street l: <file:cities> st: <file:states> postalCode: <random:numeric:5> postalAddress: {cn}${street}${l}, {st} {postalCode} description: This is the description for {cn}. The actual tags that may be included in a template definition are discussed in a later section. However, this example does illustrate some of the flexibility that MakeLDIF offers when generating LDIF data. The tags contained in this template are parsed and replaced with information appropriate for the specified tag. For example, an entry created using the preceding template might look like this: dn: uid=Neil.Wilson,ou=People,dc=example,dc=com objectclass: top objectclass: person objectclass: organizationalPerson objectclass: inetOrgPerson givenName: Neil sn: Wilson cn: Neil Wilson uid: Neil.Wilson mail: Neil.Wilson@example.com userPassword: d82fk32n telephoneNumber: 823-630-8157 At the top of the template definition are two lines that provide information about the template itself and are not actually included in entries created from this template. The first line specifies the name of the template. This is the name that is referenced in the subordinateTemplate lines of the branch definition. The second line specifies the name of the attribute that should be used as the RDN attribute for the entry. This attribute must be assigned a value lower in the template definition, and the way in which the value is assigned must ensure that the value is unique. Note It is possible to use multivalued RDNs by separating the attribute names with a plus sign, like rdnAttr: uid+employeeNumber . If multivalued RDNs are used, the combination of RDN attribute values must be unique, but it is possible for one or more of the attributes in the RDN to be nonunique as long as the combination is never duplicated . In addition to the template and rdnAttr lines, it is also possible to include one or more subordinateTemplate lines. This makes it possible to include dynamically generated entries below other entries that have been dynamically generated (that is, if each user entry has one or more entries below it), and can allow for some rather complex hierarchies. While there is no limit placed on this level of nesting, it is important to ensure that no recursive loops are created by having a subordinateTemplate that either directly or indirectly creates additional entries using the same template. Template definitions also support the concept of inheritance through the use of the extends keyword. For example, entries generated from the template definition look like this: template: certificatePerson rdnAttr: uid extends: person userCertificate;binary:: <random:base64:1000> Include all of the attributes defined in the person template as well as userCertificate;binary with the specified format. Multiple inheritance is allowed (by including multiple lines with the extends keyword), but as with the subordinateTemplate keyword it is important not to create a recursive loop in which a template could either directly or indirectly extend itself. Template File TagsIn order to ensure that MakeLDIF provides the ability to generate LDIF files that can be used to simulate a wide variety of deployments, a large number of tags have been defined for use in template definitions. This section describes the standard set of tags that may be used in a MakeLDIF template file. It is possible to use custom tags in template files as well, and the process for developing custom tags is defined in a later section. Standard Replacement TagsThe tags that may be used in a template file include those in TABLE 9-2: Table 9-2. Supported Tags for MakeLDIF Template Entries
Attribute Value Reference TagsIn addition to the standard replacement tags listed in TABLE 9-2, it is possible to use tags that reference the values of other attributes in the same entry. These tags are called attribute value reference tags and they may be used by simply enclosing the name of the desired attribute in curly braces. When these tags are encountered in the template, they are replaced with the value of the specified attribute. If the specified attribute has not been assigned a value for the current entry, this tag is replaced with an empty string. For example, consider the following excerpt from a template: givenName: <first> sn: <last> uid: {givenName}.{sn} cn: {givenName} {sn} mail: {uid}@example.com If the value chosen for the first name is Neil and the last name is Wilson , the following LDIF output would result: givenName: Neil sn: Wilson uid: Neil.Wilson cn: Neil Wilson mail: Neil.Wilson@example.com Note In order for this to work properly, it is necessary to assign a value to an attribute before it may be referenced in this manner. If, for example, the definition of the mail attribute had appeared before the definition of the uid attribute, the resulting email address would have been @example.com because the {uid} tag would not have a value and therefore would have been replaced with an empty string. It is also possible to place a colon after the name of the attribute followed by a positive integer value {length} . This causes at most the first {length} characters of the value from the specified attribute to be used instead of the entire value. For example, the template excerpt: givenName: <first> sn: <last> initials: {givenName:1}{sn:1} Would produce the following LDIF for a first name of Neil and a last name of Wilson : givenName: Neil sn: Wilson initials: NW If the specified {length} is longer than the value of the named attribute, the entire value of the attribute is used and no padding is added. Tag Evaluation OrderThe order in which tags are evaluated while parsing a template definition is important because changing the evaluation order can change the way that the output is produced. Although it is not possible for the end user to change this evaluation order, it is important to understand how template definitions are processed so that LDIF files are generated in the appropriate manner. In most cases, it is not possible to nest tags. That is, the output of one tag cannot be used as input to another. For example, consider the following MakeLDIF tag evaluation order: description: <random:alpha:<random:numeric:5:10>> One might hope that this would first evaluate the <random:numeric:5:10> tag to create a numeric value between 5 and 10 (for example, 7) and then evaluate the outer tag as <random:alpha:7> . However, this is not the case. Because of the way that MakeLDIF parses template definitions, this fails because it tries to interpret <random> as an integer value. Fortunately, this is not a problem in most cases because enough variations of the standard replacement tags have been provided to deal with this. For example, to achieve the desired result attempted by the above template line, you can instead use: description: <random:alpha:5:10> For the purposes of this discussion, MakeLDIF parses template files in the following order:
Based on this order of operations, any tag at one level may be used as input to any tag at a higher level. For example, the following is legal because base64 standard replacement tags are evaluated after attribute value reference tags: description:: <base64:{givenName}> Defining Custom TagsOne of the benefits of MakeLDIF is the large number of tags it provides that can be used to customize the process of creating LDIF files. However, even with the large set of tags provided, it may be desirable to extend the functionality even further. This is possible through the use of custom tags. Custom tags can be easily defined by creating a Java class that extends the abstract CustomTag class. This class defines three methods :
A Sample Custom Tag Implementation # MakeLDIF Custom Tag Usage # /** * This class provides an implementation of a MakeLDIF custom tag that will be * used to calculate the sum of all the integer values provided as arguments to * the tag. */ public class SumCustomTag extends CustomTag /** * Performs any necessary one-time initialization that should be performed * when this custom tag is first created. In this case, no initialization is * performed. */ public void initialize() { // No implementation required. } /** * Performs any initialization that should be performed each time the LDIF * generation starts working on a new branch (e.g., to reset any internal * variables that might have been in use). In this case, no reinitialization * is performed. */ public void reinitialize() { // No implementation required. } { /** * Parses the list of arguments, converts the values to integers, and totals * those values. * * @param tagArguments The arguments containing the numeric values to be * totaled. * * @return The string representation of the total of all the argument values. */ public String generateOutput(String[] tagArguments) { int sum = 0; for (int i=0; i < tagArguments.length; i++) { sum += Integer.parseInt(tagArguments[i]); } return String.valueOf(sum); } } Using the Example Custom TagOnce the custom tag class has been implemented and compiles properly, it may be used by specifying that class in a custom tag in the template file. For example, to use the custom tag we just implemented, something like the following could be placed in the template file: cn: <custom:SumCustomTag:1,2> When MakeLDIF is run using a template that contains this definition, the generateOutput method of the SumCustomTag class is invoked and produces output containing: cn: 3 Because static values were provided as arguments to the tag, the output is exactly the same every time. This is not all that useful in this case, but it doesn't have to be that way. As indicated earlier, custom tags are parsed after most other kinds of tags. This means that it is possible to use something like: cn: <custom:SumCustomTag:<random:numeric:5>,<random:numeric:5>> There, the output from other tags is used as arguments to the custom tag. In this case, each <random:numeric:5> tag generates a five-digit integer, and the custom tag adds those values together. Unlike the previous example, the output in this case may be different for each entry because the arguments are randomly chosen values. Of course, even then this particular custom tag is not very useful, but it is a simple example that can be used as the foundation for creating more useful custom tags for real-world purposes. Using and Automating MakeLDIFOnce you understand how to use the MakeLDIF application, you can automate the process of generating test data for a benchmark. As a summary, the MakeLDIF application is recommended for use as an LDIF structure and data generation tool for testing the performance of the Sun ONE Directory Server 5.x (or any other directory server for that matter). MakeLDIF provides the ability to generate LDIF files that can be imported into the Sun ONE Directory Server 5.x environment. The information needed to generate the entries is specified using a template file that can be customized to produce LDIF files for use as a test harness for directory server benchmark performance validation tests. TABLE 9-3 lists examples of the layout and files that can be used in benchmark performance validation tests. Specific examples follow the table. Table 9-3. BenchMark Performance Validation Tests 250k
Sample Root Makefile# Copyright 2003 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # #ident "@(#)Makefile 1.0 08/01/03 SMI" # # Root Makefile for 250k.ldif file # OUTDIR=bp250k-data TARGET=bp250k all: output_dir $(TARGET) clean: output_dir @-if [ -d $(OUTDIR) ]; then \ /usr/bin/rm -rf $(OUTDIR); fi bp100k: @echo Building... @-#java -cp MakeLDIF.jar MakeLDIF -t bp250k.template -o $@.ldif @echo Moving LDIF Files... @echo Please wait... @-/usr/bin/mv $@.ldif $(OUTDIR) output_dir: @-if [ ! -d $(OUTDIR) ]; then mkdir -p $(OUTDIR); fi Makefile and Generating the Filter File# Copyright 2003 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # #ident "@(#)Makefile 1.0 08/01/03 SMI" # # Root Makefile for 250k.ldif file # OUTDIR=bp250k-data TARGET=bp250k all: output_dir $(TARGET) clean: output_dir @-if [ -d $(OUTDIR) ]; then \ /usr/bin/rm -rf $(OUTDIR); fi bp100k: # # Generate the filter file with (caution, its VERY slow) # @echo Building... @-#java -cp MakeLDIF.jar MakeLDIF -t bp250k.template -o $@.ldif \ -T uid:eq -T sn:eq -T givenName:eq -T cn:eq -T cn:sub -F $@filter- file.ldif @echo Moving LDIF Files... @echo Please wait... @-/usr/bin/mv $@.ldif $(OUTDIR) @-#/usr/bin/mv $@-filter-file.ldif $(OUTDIR) output_dir: @-if [ ! -d $(OUTDIR) ]; then mkdir -p $(OUTDIR); fi The following code box shows an example of running the Makefile without the filter option and the output: Building 250k LDIF File... Processed 250000 entries 250000 entries written to bp250k.ldif Moving LDIF file to bp250k-data location... Done. The following code box shows an example of running the Makefile with the filter option and the output: Building 250k LDIF File... Processed 250000 entries 250000 entries written to bp250k.ldif Writing filters to bp205k-filter-file.ldif Wrote 250000 equality filters for uid Wrote 250000 equality filters for givenname Wrote 250000 equality filters for sn Wrote 2479 substring filters for cn Moving LDIF files to bp250k-data location... Done. The previous examples showed how you can automate the process of generating LDIF data sets. It is also possible to automate the process of building of larger data sets using a global Makefile. Example: # Copyright 2003 Sun Microsystems, Inc. All rights reserved. # Use is subject to license terms. # #ident "@(#)Makefile 1.0 08/01/03 SMI" # # Global Makefile # SUBDIRS= bp250k-src bp500k-src bp1M-src bp5M-src \ bp10M-src bp25M-src bp50M-src all: #all clean: @for i in $(SUBDIRS) ; \ do \ cd $$i ; \ echo "==================================" ; \ echo " Current directory: $$i" ; \ echo "==================================" ; \ if [ -f makefile ] ; \ then \ $(MAKE) -f makefile $@ ; \ else \ $(MAKE) -f Makefile $@ ; \ fi ; \ cd .. ; \ done Below is an example of running the global Makefile and the output: ================================== Current directory: bp250k-src ================================== Building 250k LDIF File... Processed 250000 entries 250000 entries written to bp250k.ldif Moving LDIF file to bp250k-data location... Done. ================================== Current directory: bp500k-src ================================== Building 500k LDIF File... Processed 500000 entries 250000 entries written to bp500k.ldif Moving LDIF file to bp500k-data location... Done. ================================== Current directory: bp1M-src ================================== Building 1M LDIF File... Processed 1000000 entries 250000 entries written to bp1M.ldif Moving LDIF file to bp1M-data location... Done. etc... etc...  |
![LDAP in the Solaris Operating Environment[c] Deploying Secure Directory Services](/icons/blank_book.jpg "LDAP in the Solaris Operating Environment[c] Deploying Secure Directory Services"){kind=icon}
EAN: N/A
Pages: 87