Directory Service Maintenance

Ongoing maintenance of HugeCo's large directory service requires a lot of attention from IS system administrators. This is especially true at the present time because the service is still evolving as new directory-enabled applications are being integrated. All basic maintenance is handled by automated procedures that are similar to those used for other systems that the IS organization manages . The following sections provide specific information on each aspect of directory maintenance within HugeCo's deployment.

Data Backups and Disaster Recovery

As discussed earlier in this chapter, there are two master servers for each portion of the HugeCo directory namespace. Once a month, one of the master servers is taken down, and the system is tested to ensure that everything can still function well with only one master.

The master servers are backed up to disk nightly via the directory server's "hot backup" feature and archived to tape via digital linear tape (DLT) drives . Twice a week, each region sends a set of backup tapes to another region for off-site storage. The backup procedures are largely automated and similar to those used for all the services that HugeCo's IS organization supports.

HugeCo outsources all its disaster recovery planning and services to IBM Business Continuity and Recovery Services, which maintains cold sites in each of HugeCo's four regions . So far HugeCo has not experienced a disaster that required use of the cold sites.

Maintaining Data

The IS organization spends a lot of time and money on data maintenance across all of HugeCo's systems. Corporate data is held in a variety of databases, and keeping the data up-to-date is largely a manual process. One goal of the directory service team was to increase the overall data maintenance burden as little as possible. The team managed to minimize maintenance demands by automating some processes and distributing data maintenance responsibilities.

To integrate with its PeopleSoft HR database, HugeCo contracted with America Online's Professional Services organization to create a directory synchronization tool. The synchronization tool runs once per hour to transfer changes made in the HR database to the directory service. Basic information about employees is synchronized, including name , contact information, ID number, and location. The synchronization tool takes care of creating new hugeCoPerson entries in the directory service when employees join HugeCo, and it disables user accounts by altering passwords after an employee leaves the company. The synchronization tool, written in Perl, operates on text extracts generated from the PeopleSoft database, and it uses the PerLDAP module to access the LDAP directory.

To distribute the data maintenance responsibilities, the HugeCo team defined the following categories of directory data managers:

• Directory administrators , who are granted complete access to all the data in the directory service

• Departmental administrators , who are granted nearly full access rights to the people and group entries for their department (but are not allowed to change any attributes managed by the HR database synchronization process)

• Help Desk staff , who are permitted to set passwords for all people entries

• End users , who are allowed to change home contact information, URLs, descriptions, and a few other fields within their own entries

For access control purposes, groups are maintained in the directory for each category of data administrators. The one exception is the end-user category: End users are identified by the absence of group membership. Access control rules were placed in the directory to give people in each category an appropriate level of access. Because departmental administrators and end users are allowed to manage some of their own information, the data management burden carried by the IS employees (directory administrators and Help Desk staff) is minimized.

20/20 Hindsight: Improving Data Quality

As an increasing number of HugeCo employees found out about the new directory service and began to examine their own data, the IS Help Desk started to receive reports of erroneous information. To understand the problem better and determine the cause, the central IS organization is developing an e-mail survey tool that will extract information from both the PeopleSoft HR database and the directory service. Surveys will be sent to a random sample of 5,000 employees in an effort to determine how widespread the data quality problems are. The results will be checked against directory audit logs to determine the source of the incorrect information, and the data gathered by the survey will be used to decide where to focus future data quality improvement efforts.

Monitoring

The overall HugeCo strategy for network monitoring revolves around HP OpenView, a commercial network management system (NMS). Each regional IS department runs an HP OpenView system that monitors the network and the applications located in that region. In addition, the central IS organization runs an HP OpenView system that monitors the global network and centrally managed applications such as the PeopleSoft system.

A combination of techniques was used to integrate the Netscape Directory Server software and important directory-enabled applications into the NMSs. First the Simple Network Management Protocol (SNMP) support built into the server software was used to provide basic service and performance monitoring. Then a set of Perl scripts was developed with the PerLDAP module to probe all the critical directory servers from several locations on HugeCo's network. Finally, indirect monitoring of the directory service was started through extensive observation of critical directory-enabled applications, including the e-mail servers, the PeopleSoft synchronization process, the phone book servers, the Netegrity SiteMinder servers, and the Web servers that support critical applications. As much as possible, probes mimic the operations that end users and applications frequently perform.

20/20 Hindsight: The Value of Indirect Probes

About a month after the HugeCo directory service was first rolled out worldwide, the IS e-mail team received a complaint from one of the executive vice presidents in the Latin America region. E-mail was being delayed for up to 30 minutes before reaching all the intended recipients. This was puzzling because most messages were routinely delivered by the network of Sun ONE message transfer agent (MTA) servers within 5 minutes.

After an afternoon of investigation, the e-mail administrators discovered that all the delayed messages had been sent to a dynamic group (one in which membership is determined by a search of the directory). They quickly brought some directory experts over to look at the problem. In the end, the root cause of the problem was traced to a missing index in the configuration of the directory replica servers used by the MTAs. Although easily corrected, the problem had gone unnoticed for almost a month (much to the chagrin of the IS staff).

This incident prompted the IS employees to design and implement a series of indirect directory probes that closely emulate the behavior of important applications such as the messaging servers. By proactively monitoring the performance of the system as experienced by end users, the HugeCo IS staff hopes to detect problems earlier in the future.

When a problem is detected by HugeCo's OpenView monitoring system, the following automated notification methods are used to bring the problem to the attention of the appropriate system administrator:

• Text pager messages are sent when an urgent system outage is detected.

• E-mail messages are used to send weekly directory activity summaries and to notify administrators immediately about problems such as reduced performance of a directory application.

• IS staff and end users can access a continuously updated Web page that lists information about all known outages.

Overall, the directory service and associated applications have proven to be reliable. So far, there has been no need to automate such actions as restarting failed directory server processes or machines.

Troubleshooting

HugeCo's IS organization maintains a well-documented set of escalation procedures stating that members of the IS staff with increasing seniority will be called in over time to address critical problems. Directory-specific procedures were developed during the directory pilot deployment and refined over time to ensure that problems are addressed quickly by the right people.