Maintaining a Replication System

Data, including passwords, are replicated between all of the servers in a replication system. The three services that take part in replication are the LDAP server, Password Server, and the Kerberos KDC.

A process called slurpd replicates the LDAP server; Apple has not changed the standard OpenLDAP replication process. The Open Directory master hosts a single master LDAP database. Since it's the only writable LDAP database in the replication system, all LDAP changes are pushed out by the master to the replicas.

Password Server is a multimaster system, so the version on the Open Directory master and on each replica can accept a password or a password change for a user. This change is then replicated from the originating server to the master server, and from the master server to the other replicas

Replication for the KDC is handled by the local Password Server. All changes are replicated through the normal Password Server replication process, which then updates the local KDC and reduces the potential for additional traffic on the LAN.

Because the replication architecture relies heavily on timestamps, it is good practice to keep the master and all replicas on the same Network Time Protocol (NTP) server. Mac OS X Server can act as an NTP server.

Replication Architecture

To implement a robust and complete Open Directory replication, Mac OS X Server relies on a diverse set of tools. Understanding which tools and files are used by the replication process enables the server administrator to track and troubleshoot replication problems. The following figure illustrates the processes, configuration files, and log files involved in the replication process.

Some of the more commonly referenced files that concern the replication architecture are:

• /var/run/openldap-slurp/replica/slurpd.replog: A listing of replicated LDAP database changes that is kept on the master server. All changes are timestamped.

• /var/run/openldap-slurp/replica/slurpd.status: A listing of all replica servers and the last time that they were updated for any changes to the LDAP database.

• /var/db/authserver/authserverreplicas: A file that keeps a listing of all the currently known replica Password Servers and when they were last updated.

LDAP Replication

When a change is made to the LDAP server, that change is written to a log file named /var/db/openldap/openldap-slurp/replication.log, which contains an LDIF entry and a timestamp for every change made to the LDAP database.

This log file is read by slurpd to acquire the changes pending replication. slurpd compares the timestamps of the log entries with the timestamp of the last successful replication with each replica. Replication status is kept in /var/run/openldap-slurp/replica/slurpd. status. slurpd then attempts to use standard LDAP commands to push changes out to the replicas. The master and the replicas share a user name and password, which ensures that only the master can propagate changes to the replicas.

If slurpd is unable to communicate with a replica, it will not increment the timestamp for that particular server in slurpd.status. During the next replication interval, slurpd will attempt to reconnect to the failed server and replay all changes since the last completed replication, which is useful if a server has been temporarily offline.

Only the master LDAP database can be changed. If the master server is offline, no changes to user records or other LDAP entries can be made until either the master has been brought online again, or one of the replicas has been promoted to a master.

If you attempt to connect using Workgroup Manager to a replica, you will be automatically redirected by an LDAP referral to the master server without any indication of being redirected. Similarly, if a user attempts to change the password hint or the picture for the user's account, the Mac OS X computer will be redirected by an LDAP referral to the master server. If the master's LDAP server is down, the Mac OS X computer will be unable to perform any changes.

Password Server Replication

Password Server engages in a multimaster replication scheme. Each server is capable of taking a change to a user's password or password policy. Using this system, password changes can be made even if the Open Directory master is extremely busy or offline.

The Password Server that took the change will contact the Password Server on the Open Directory master. The master Password Server then replicates the changes to the Password Servers on the replica servers.

The originating Password Server refers to a list of replicas in /var/db/authserver/ authserverreplicas, and also uses this file to track when it last synchronized with a replica and whether that synchronization was successful.

When Password Server replicates, it acts in a fashion similar to that of the LDAP server; Password Server checks the timestamp of the last synchronization of each replica and then looks in its database for all changes since that time. All conflicts are settled by using the most recent change based upon the timestamp. If the Password Servers' clocks are more than a few minutes out of sync, they will not be able to replicate. So if you are having trouble authenticating in a replication system, check the timestamp on each server.

The following figure shows how a password change is made between a replica and a master:

1. The password is passed from the replica back to the Open Directory master.

2. Password Server checks the password.

3. The Open Directory master pushes the change out to other replicas.

The security of the Password Server replication is entirely encrypted between each Password Server process. All share the same public/private key pair to both encrypt the actual database files on the local computer and also secure the communication between the servers.

Note

By default, each PasswordService process has a unique range of 500 slots or password IDs for storing passwords. The password IDs must be unique per PasswordService process in the replication system (between the master and all replicas). That way, Password Server can't use a password ID already in use by another PasswordService process in the replication system.

KDC Replication

The following figure shows how changing a Kerberos password is propogated to each Password Server:

1. Only the KDC on the Open Directory master runs kadmind, which is the process that accepts Kerberos password and password policy changes.

   All changes that are initiated from any of the Kerberos tools, such as kpasswd and the Kerberos application, will be redirected by the Kerberos realm to this server.

2. Once the changes are made on the master's KDC, the master's Password Server receives the changes and then attempts to replicate the changes to all other Password Servers in the replication system.

3. Once the changes are received, each updated Password Server will update its local KDC in turn.

Tip

Because password changes initiated with Kerberos must occur on the master server, no password changes initiated by Kerberos may take place when the master server is down.

Examples of Replication

This scenario shows the behavior of replication:

1. The Mac OS X computer starts up.

2. It attempts to use the server to which it was bound.

3. It stores the list of LDAP servers, Kerberos servers, and password servers from the replication system.

4. The LDAP server goes offline.

5. The Mac OS X computer consults its server list and attempts to connect to a new LDAP on that list.

Replication Updates

This scenario illustrates the update behavior of replication:

1. The user successfully logs in and makes a change to his or her password. The new password is stored in the Password Server database of the replica that the Mac OS X computer is currently using for authentication.

2. Password Server will now begin the replication process via the master, which pushes this change out to all other Password Servers. Because the Chicago replica is down, the password update will fail there.

3. The Chicago replica comes back online.

4. After a secure connection is established with the replicas San Francisco master, the Chicago replica checks its logs to find the last time it was updated and requests updates from the San Francisco master. The San Francisco master plays back all entries in the database that have a timestamp more recent than the Chicago replica's last successful replication, thus synchronizing the Chicago replica with the rest of the replication system.

5. The Chicago replica comes back online.

6. The Open Directory master pushes the password change out to the Chicago replica.

Promoting a Replica to an Open Directory Master

When your Open Directory master goes permanently offline, you can promote one of the replicas to be a master. Pick a replicaideally a server that will be fast and robust enough to handle the demands of being the master server. The promotion process does not alter the data in the LDAP, Password Server, or Kerberos databases, but it does change the services' configuration files and causes Password Server to be rekeyed. A new public/private key pair is generated and the password database is reencrypted using the new key pair.

Tip

Never promote a replica that you do not want to permanently hold the position of master.

All of the new configuration information, including the Kerberos record and the list of available LDAP servers and Password Servers in the replication system, are uploaded into the server's LDAP database. When Mac OS X computers connect to a server in the replication system, they will update their configurations and their lists of replication servers.

Once you have created the new master, you will need to go to each replica in the domain and manually associate them with the new master as fast as possible. Otherwise, you might run into a situation where Mac OS X computers are using computers configured with the old replication data.

Tip

When you promote the replica, make sure that you use the same information for both the LDAP domain and the Kerberos realm as the previous master. Failure to do so will render your user records inaccessible.

Upgrading Mac OS X Server v10.3 to v10.4

In a replication system, you cannot have a mix of Mac OS X Server v10.3 and v10.4 computers. A Mac OS X Server v10.3 replica will not connect to a Mac OS X Server v10.4 master. If you upgrade your master server to Mac OS X Server v10.4, you must also upgrade the replicas.

The following figure shows four steps for upgrading a master/replica system from Mac OS X Server v10.3 to Mac OS X Server v10.4.

When upgrading, you should start with the master server; the installer will handle migrating the old databases into the new formats. After that, none of your replicas will be connected any longer. Do a clean install of Mac OS X Server v10.4 on each replica, and go through the process to configure it as a replica of your server and download the directory data to the replica.

During the migration process, the Mac OS X computers shouldn't experience any downtime. When a server is down while it is being upgraded, the Mac OS X computer will bind to a different server.

No data is lost during the upgrade, because the master server contains all the data and copies it to the upgraded replicas. The only potential issue is if a user upgraded his or her password while the servers were being upgradedthen the Password Servers on the replicas would accept and store changes.

After the master server has been upgraded and the replicas are disconnected, password changes can't be sent to the master Password Server. When Mac OS X Server v10.4 is installed on the replicas, the changed password will be lost, and the user's password will revert to the old password that was on the master server.

Replica Troubleshooting

The logs are very valuable when troubleshooting the replication process. The LDAP and Password Server logs will show any attempted replications and perhaps why the attempts failed.

You can try to force replication from Server Admin; even if it doesn't work, it might trigger errors to be written to the logs. You can also initiate replication manually from the command line with slapconfig to provide more error feedback than you get by using Server Admin.

For the initial replication to occur, you need to connect via SSH as the root user between the replica and the master server. Issues with SSH keys or other problems can prevent this from happening, so look at the SSH logs for errors.

Another common issue is time skew between the replicas or the Mac OS X computers and the authentication server. Point all of your computers to the same NTP server to ensure clock synchronization.

More Info

The NTP service on Mac OS X Server should be synchronized to a Stratum 1, 2, or 3 NTP server. See http://ntp.isc.org/bin/view/Servers/WebHome for more information.

Directory service access control lists (ACLs) are also replicated as part of the Open Directory replication support. All nondefault ACLs will be copied to each replica each time they are updated.

Finally, as with all network services, make sure that the network is running correctly. Replication is a critical part of a robust and dependable Open Directory architecture. Apple has implemented its replication architecture using industry standard components and leveraging existing technologies. Understanding the way that replication works on Mac OS X Server is critical to planning, deploying, and supporting Open Directory in the workplace.