Example: The ldapsync Tool: One-Way Synchronization with Join

Example: The ldapsync Tool: One-Way Synchronization with Join

The ldapsync tool presented in this example may be used to implement periodic one-way synchronization into an LDAP server from any system whose data can be expressed as a delimited text file. Many data sources provide tools that make it easy to generate this kind of data extract. Some systems provide the ability to extract only those changes that have occurred since the previous extract, in which case the ldapsync tool presented here runs more efficiently .

Our tool is a homegrown tool that is written in the Perl 5 language. Perl allows for rapid prototyping, provides good portability of the code, performs acceptably, and may be easily modified by the thousands of system administrators who have learned to program in it.

How It Works

The ldapsync tool connects to an LDAP directory server that holds a writable copy of people data. It authenticates using a distinguished name (DN) and password and then reads a series of comma-delimited lines from standard input.

Within the ldapsync input stream, all lines that begin with a pound sign ( # ) are treated as comments and ignored. The first noncomment line must look like this:

  join-attribute-name  ,  update-attribute-name1  ,  update-attribute-name2  ... 

One or more update attribute names may be listed. For example, here's the input line to specify telephoneNumber as the join attribute and to support updates to the full name ( cn ) and surname ( sn ) attributes:

 telephoneNumber,cn,sn 

The remaining input lines must look like this:

  join-attribute-value  ,  update-attribute-value1  ,  update-attribute-value2  ... 

The number of comma-separated values on each of these lines should match the number of attribute names listed on the first noncomment line, although the ldapsync tool is smart enough to ignore extra values and treat missing ones as absent (no update needed).

Here's an example of an input line that specifies a join value of [phone] and updates the cn attribute with Babs Jensen and the sn attribute with Jensen :

 [phone],Babs Jensen,Jensen 

And here's an example of an input line that specifies a join value of [phone] , no cn update, and an sn update of Jones :

 [phone],,Jones 

The synchronization process is driven entirely from the input data. For each line that includes values, the ldapsync tool issues an LDAP search to locate an entry. The LDAP search filter is constructed on the basis of the join attribute value. For example, if the join attribute is user ID ( uid ) and the join value provided is mcs , then a search filter like this is used: (&(objectClass=person)(uid=mcs)) . If one entry is found, the ldapsync tool checks each new value provided to see if that value is already present in the entry. If any values are missing, updates are required and the ldapsync tool uses an LDAP modify operation to replace the values that need to be updated. As it processes the input data, the ldapsync tool logs error and informational messages to standard output.

Usage Examples

Suppose that you want the telephone numbers for people in your directory service to come from a human resources database and that both the HR database and the directory store a user ID value (the uid attribute in the directory). Listing 23.1 shows a sample ldapsync input file whose purpose is to update many people's telephoneNumber attribute values in your directory. Telephone number changes were extracted from the HR database to create the input file. Some lines have been omitted and replaced with " ... " to reduce the length of the listing; in all there are 150 telephoneNumber changes.

Listing 23.1 An ldapsync Input File That Contains telephoneNumber Updates

 # join attribute name,attribute name uid,telephoneNumber # # uid value, telephoneNumber value scarter,[phone] tmorris,[phone] kvaughan,[phone] ... elott,[phone] cnewport,[phone] jvedder,[phone] 

If the data in Listing 23.1 were stored in a file called changes-from-hr , the ldapsync tool could be invoked like this:

 ldapsync.pl < changes-from-hr 

Listing 23.2 shows the output of this command when run against a directory server that contains the contents of the Example.ldif file that Netscape bundles with its Directory Server product.

Listing 23.2 Output from the First ldapsync Sample Run

 INFO.: Mon Jul 15 11:40:38 2002 - ldapsync started INFO.: Found one entry with uid scarter. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid scarter INFO.: Modify done. INFO.: Found one entry with uid tmorris. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid tmorris INFO.: Modify done. INFO.: Found one entry with uid kvaughan. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid kvaughan INFO.: Modify done. ... INFO.: Found one entry with uid elott. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid elott INFO.: Modify done. INFO.: Found one entry with uid cnewport. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid cnewport INFO.: Modify done. INFO.: Found one entry with uid jvedder. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid jvedder INFO.: Modify done. INFO.: -------------------------------------- INFO.: Summary: INFO.:   150 change records were processed. INFO.:   150 entries were updated. INFO.: Mon Jul 15 11:40:53 2002 - ldapsync finished. 

Again, some lines have been omitted and replaced with " ... " to reduce the length of the listing. Notice the summary at the end: "150 entries were updated." If the same ldapsync.pl command is executed a second time, no changes are made to the LDAP directory (because all of the values provided in the input file are now present in the directory server). Listing 23.3 shows the resulting output.

Listing 23.3 Output from the Second ldapsync Sample Run

 INFO.: Mon Jul 15 11:40:55 2002 - ldapsync started INFO.: Found one entry with uid scarter. Checking for updates... INFO.: Found one entry with uid tmorris. Checking for updates... INFO.: Found one entry with uid kvaughan. Checking for updates... ... INFO.: Found one entry with uid elott. Checking for updates... INFO.: Found one entry with uid cnewport. Checking for updates... INFO.: Found one entry with uid jvedder. Checking for updates... INFO.: -------------------------------------- INFO.: Summary: INFO.:   150 change records were processed. INFO.:   No entries were updated. INFO.: Mon Jul 15 11:40:58 2002 - ldapsync finished. 

Listing 23.4 shows another sample input file. This file also uses uid as the join attribute, but it specifies an assortment of changes to the cn , sn , and telephoneNumber directory attributes.

Listing 23.4 An ldapsync Input File That Contains Updates to Three Different Attribute Types

 # Sample ldapsync input file # # the first noncomment line lists the attribute names (key first): uid,cn,sn,telephoneNumber # # updates: bjensen,B Jensen,Jones,[phone] scarter,S Carter kwinters,,,[phone] 

For the bjensen entry, cn , sn , and telephoneNumber are all to be updated. For the scarter entry, only cn is to be updated. For the kwinters entry, only telephoneNumber is to be updated. Listing 23.5 shows the output produced by the ldapsync tool when it is driven from the data in Listing 23.4.

Listing 23.5 Output from the Third ldapsync Sample Run

 INFO.: Mon Jul 15 11:49:31 2002 - ldapsync started INFO.: Found one entry with uid bjensen. Checking for updates... INFO.:     - replace cn with B Jensen INFO.:     - replace sn with Jones INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid bjensen INFO.: Modify done. INFO.: Found one entry with uid scarter. Checking for updates... INFO.:     - replace cn with S Carter INFO.: Modifying entry with uid scarter INFO.: Modify done. INFO.: Found one entry with uid kwinters. Checking for updates... INFO.:     - replace telephoneNumber with [phone] INFO.: Modifying entry with uid kwinters INFO.: Modify done. INFO.: -------------------------------------- INFO.: Summary: INFO.:   3 change records were processed. INFO.:   3 entries were updated. INFO.: Mon Jul 15 11:49:32 2002 - ldapsync finished. 

As these examples show, ldapsync is a simple but versatile tool.

The Source Code

The ldapsync tool uses the PerLDAP object-oriented LDAP access module available from the Mozilla Web site at http://mozilla.org/directory/perldap. The Perl source code for ldapsync is all in one file, named, logically enough, ldapsync.pl . Listing 23.6 shows the first portion of the ldapsync code.

Listing 23.6 The ldapsync Code (Part 1 of 5)

 1. #!/usr/local/bin/perl  2. #  3. # ldapsync -- Perl 5 script that synchronizes a  4. #   comma-separated text file of attribute values.  5. #  6. # From the 2nd Edition of the book:  7. #   "Understanding and Deploying LDAP Directory Services"  8. #   by Timothy A. Howes, Mark C. Smith, and Gordon S. Good.  9. # 10. # usage: ldapsync.pl < file 11. # 12. # Where the contents of file are of the form: 13. #   joinAttrName,attrname1,attrname2... 14. #   joinAttrValue,attrvalue1,attrvalue2... 15. #   joinAttrValue,attrvalue1,attrvalue2... 16. #   ... 17. # 18. # For example: 19. #   uid,cn,telephoneNumber 20. #   bjensen,Barbara Jensen,[phone]. #   cms,Christina Smith 22. #   jjones,John Jones,[phone]. #   ajackson,,[phone]. # 25. # Requires: PerLDAP 26. # 27. 28. use Mozilla::LDAP::Conn; 29. 30. # LDAP server information: 31. $ldapSearchBase = "dc=example,dc=com"; 32. $ldapHost       = "ldap.example.com"; 33. $ldapPort       = "389"; 34. $ldapBindDN     = "cn=LDAP Sync Tool,ou=Special Users," 35.                             .  $ldapSearchBase; 36. $ldapBindPW     = "Tb58-#Wxza"; 37. 38. # Start of main: 39. 40. logInfo( (scalar localtime) . " - ldapsync started" ); 41. 42. # Open an authenticated connection to the LDAP server. 43. $ldap = new Mozilla::LDAP::Conn( $ldapHost, $ldapPort, 44.         $ldapBindDN, $ldapBindPW ); 45. if ( ! $ldap ) { 46.     logError( "Unable to connect to server at " 47.             . "ldap://$ldapHost:$ldapPort" ); 48.     exit 1; 49. } 50. 

The Perl interpreter identified on line 1 must be one that has the PerLDAP module installed (the copy of Perl that Netscape bundles with its Directory Server 6 product was used in testing this script). A set of variables that holds the LDAP server information is set by the code on lines 30 “36. A subroutine is called on line 40 to log a startup message (the code for logInfo() is shown later in this section, in Listing 23.10). An LDAP connection is opened by the code on lines 42 to 49, and the synchronization tool authenticates itself as a special directory entry that presumably has been given permission to update the desired attributes in the LDAP server.

Listing 23.7 shows the code that reads and parses the ldapsync input. The while loop that begins on line 54 encloses the main body of the ldapsync.pl program. The loop is executed as long as there is more input to be read.

Listing 23.7 The ldapsync Code (Part 2 of 5): Reading and Parsing Input

 51. # For each line of input, search for the directory entry 52. # corresponding to the first field, and see if its value 53. # for the second field needs to be updated 54. while ( <STDIN> ) { 55. # Read one line. 56.     $line = $_; 57. 58. # Skip lines that begin with '#' (comments) 59.     if ( $line =~ /^#/ ) { 60.         next; 61.     } 62. 63. # Discard newline and return characters; break at commas 64.     chop $line; 65.     if ( $line =~ /\r$/ ) { 66.         chop $line; 67.     } 68.     @args = split( /,/, $line ); 69. 70. # If this is the first line, read attribute names 71.     if ( ! $joinAttrName ) { 72.         $joinAttrName = @args[0]; 73.         @attrNameList = @args; 74.         next; 75.     } 76. 77.     ++$changeCount; 78. 79. # Parse join attribute and attribute values to update 80.     $joinAttrValue = @args[0]; 81.     @attrValueList = @args; 82. 

The code on lines 55 to 61 reads one input line and takes care of skipping comment lines. The code on lines 63 to 68 parses the input line into an array of strings (using commas as separators). The code on lines 70 to 75 processes the first noncomment input line; it creates a joinAttrName string variable and an attrNameList array variable to store the relevant attribute names. Line 77 increments a counter that is used to generate the summary, and the code on lines 79 “81 sets the joinAttrValue and attrValueList variables that hold the join value and any values to update that are present on the input line.

Next some LDAP- related work is done. Listing 23.8 shows the code that searches for an entry to update. It uses the PerLDAP search() method.

Listing 23.8 The ldapsync Code (Part 3 of 5): Searching for an Entry to Update

 83. # Search for entry with uid equal to the join attribute  84.     $filter = "(&(objectClass=person)"  85.             . "($joinAttrName=$joinAttrValue))";  86.     $entry = $ldap->search( $ldapSearchBase, "subtree",  87.                 $filter, 0, @attrNameList );  88.  89. # Found a match - update if necessary  90.     $matchLogString = "with $joinAttrName $joinAttrValue";  91.     $entryCount = 0;  92.     for ( $tmpEntry = $entry; $tmpEntry;  93.                 $tmpEntry = $ldap->nextEntry ) {  94.         ++$entryCount;  95.     }  96.     if ( $entryCount == 0 ) {  97.         logError( "Found no entry $matchLogString." );  98.         ++$notFoundCount;  99.     } elsif ( $entryCount > 1 ) { 100.         logError( "$entryCount entries found $matchLogString." ); 101.         ++$multipleMatchCount; 102.     } else { 103.         logInfo( "Found one entry $matchLogString." 104.                 . " Checking for updates..." ); 

The code on lines 83 to 87 constructs a search filter and issues an LDAP search operation against the directory server. The remainder of the code checks that exactly one entry was found and logs an error message if not (the code for the logError() subroutine is shown later in this section, in Listing 23.10).

Listing 23.9 shows the code that determines whether any values in the entry that was found need to be updated and issues an LDAP modify command if updates are needed.

Listing 23.9 The ldapsync Code (Part 4 of 5): Modifying an Entry

 105.         $entryNeedsUpdating = 0; 106.         for ( $i = 1; $i < @attrNameList.size; ++$i ) { 107.             if ( $i > @attrValueList.size ) { 108. # No more new values: break out of for loop. 109.                 done; 110.             } 111.             if ( @attrValueList[$i] eq "" ) { 112. # New value is empty: try the next one 113.                 next; 114.             } 115. 116. # Replace attribute's values if new value does not match old 117.             if ( ! $entry->hasValue( @attrNameList[$i], 118.                     @attrValueList[$i], 0 )) { 119.                 $entry->setValues( @attrNameList[$i], 120.                         @attrValueList[$i] ); 121.                 logInfo( "    - replace @attrNameList[$i]" 122.                         . " with @attrValueList[$i]" ); 123.                 $entryNeedsUpdating = 1; 124.             } 125.         } 126. 127. # Update entry over LDAP if needed 128.         if ( $entryNeedsUpdating ) { 129.             logInfo( "Modifying entry $matchLogString" ); 130.             $ldap->update( $entry ); 131.             if ( $ldap->getErrorCode()) { 132.                 logError( "Modify failed for entry" 133.                         . " $matchLogString: " 134.                         . $ldap->getErrorString() ); 135.                 ++$updateFailureCount; 136.             } else { 137.                 logInfo( "Modify done." ); 138.                 ++$updateSuccessCount; 139.             } 140.         } 141.     } 142. } 143. 

The PerLDAP hasValue() method is used by the code on lines 117 and 118 to determine whether a value read from the input file is already present in the entry; this check avoids unnecessary updates. If the value is not present, the code on lines 119 and 120 uses the PerLDAP setValues() method to replace the existing values in the entry with the new ones. An entryNeedsUpdating flag is set on line 123 so that the code on lines 127 to 141 can determine whether an LDAP modify operation needs to be issued. If so, the code on line 130 uses the PerLDAP update() method to perform the LDAP modify.

Listing 23.10 shows the remainder of the ldapsync source code. The code on lines 144 to 171 closes the LDAP connection and prints a series of "INFO.:" lines that summarize the ldapsync run. The code segments for the logError() and logInfo() subroutines start on lines 179 and 189, respectively. These subroutines simply print error and informative lines in a consistent way.

Listing 23.10 The ldapsync Code (Part 5 of 5)

 144. # Close LDAP connection and clean up 145. $ldap->close; 146. 147. # Print an activity summary and determine exitCode 148. logInfo( "--------------------------------------" ); 149. logInfo( "Summary:" ); 150. $exitCode = 0; 151. if ( $changeCount > 0 ) { 152.     logInfo( "  $changeCount change records were processed." ); 153.     if ( $updateSuccessCount > 0 ) { 154.         logInfo( "  $updateSuccessCount entries were updated." ); 155.     } else { 156.         logInfo( "  No entries were updated." ); 157.     } 158.     if ( $updateFailureCount > 0 ) { 159.         logInfo( "  $updateFailureCount failed update(s)." ); 160.         $exitCode = 1; 161.     } 162.     if ( $notFoundCount > 0 ) { 163.         logInfo( "  $notFoundCount entries were not found." ); 164.         $exitCode = 1; 165.     } 166.     if ( $multipleMatchCount > 0 ) { 167.         logInfo( "  $multipleMatchCount $joinAttrName values" 168.                 . " matched more than one entry." ); 169.         $exitCode = 1; 170.     } 171. } 172. 173. # That's all folks! 174. logInfo( (scalar localtime) . " - ldapsync finished." ); 175. exit $exitCode; 176. # End of main. 177. 178. 179. # Start of logError(): 180. sub 181. logError { 182.     local( $msg ) = @_; 183. 184.     print "ERROR: $msg\n"; 185. } 186. # End of logError(). 187. 188. 189. # Start of logInfo(): 190. sub 191. logInfo { 192.     local( $msg ) = @_; 193. 194.     print "INFO.: $msg\n"; 195. } 196. # End of logInfo(). 

Ideas for Improvement

Many things could be done to improve the ldapsync tool; if you decide to use it, some customization will likely be needed to meet your specific synchronization requirements. Here are a few general ideas for improvement:

• Add an option to create entries that are missing from the LDAP directory service.

• Add data translation features. For example, user IDs could be changed into e-mail addresses by the addition of "@domain." Telephone extensions could be turned into complete international standard phone numbers by the addition of "+1" and the other missing digits.

• Improve efficiency in the face of errors by adding code to create a "reject" file that contains all updates that could not be made (for example, some updates might fail because of an LDAP server outage or a permissions problem). Once the problem that caused the updates to fail has been corrected, the ldapsync tool could be run again with the reject file as input.

• Improve security by using SSL or TLS to protect the TCP connection to the LDAP server and by using certificate-based authentication instead of password-based authentication.