Obituaries

     

Obituaries

Some of the most common problems in NDS/eDirectory are caused by obituaries (commonly referred to as obits ) not being processed properly. Any number of reasons can cause obituaries to not be processed (resulting in what is known as stuck obits ), ranging from a down server or communication link to an invalid backlink list in an object. If you understand the background processes and how certain processes use obituaries, you may find it easier to determine the best course of action for correcting a problem.

NOTE

Many of the most common DS problems are caused by problems with obituaries purging, but they initially appear to be caused by something else.


Obituaries are operational attributes (that is, not attributes that can be controlled by the user ) that DS uses to ensure referential integrity between objects during certain operations ”such as object and partition move operations, object deletions, and object restorations. DS uses the obit attribute internally, with the syntax type SYN_OCTET_STRING . The attribute has the DS_READ_ONLY_ATTR constraint flag that restricts its access to only the DS servers.

There are three obituary type classes: primary, secondary, and tracking. A primary obituary indicates an action on an object. A secondary obituary indicates the servers that must be contacted and informed of the primary obituary's action. A tracking obituary is an informational obituary that is associated with certain primary obituaries. A tracking obit does not go through the same process as the primary and secondary obits.

Table 6.1 shows the different obituary types and when DS generates them.

Table 6.1. Obituary Types and Classes

OBITUARY TYPE

TYPE VALUE

CONSTANT NAME

OBITUARY CLASS

DESCRIPTION

Restored

0x0000

OBT_RESTORED

Primary

Created when an object is restored from a backup.

Dead

0x0001

OBT_DEAD

Primary

Created when an object is deleted.

Moved

0x0002

OBT_MOVED

Primary

Created when an object is moved from one container to another. This obituary is created for the object's original location.

Inhibit Move

0x0003

OBT_INHIBIT_MOVE

Tracking

Created when an object is moved from one container to another. This obituary is created for the object's destination location. This obituary prevents the object from being moved again until the previous move has completed.

OLD_RDN

0x0004

OBT_OLD_RDN

Tracking

Created when an object is renamed . This obituary is created for the old object name.

NEW_RDN

0x0005

OBT_NEW_RDN

Primary

Created when an object is renamed. This obituary is created for the new object name.

BackLink

0x0006

OBT_BACKLINK

Secondary

Represents the object on another server's database that must be notified when the object is modified (such as when it is deleted, renamed, or moved). This is a secondary obituary. BackLink obituaries may represent real copies of the object as found on a Read/Write replica or any server with an external reference to the object.

Tree_OLD_RDN

0x0007

OBT_TREE_OLD_RDN

Tracking

Created when a partition root object (and not an NDS tree name) is renamed. This is a special case of the OLD_RDN obituary type.

Tree_NEW_RDN

0x0008

OBT_TREE_NEW_RDN

Primary

Created when a partition root object (and not an NDS tree name) is renamed. This is a special case of the NEW_RDN obituary type.

Purge All

0x0009

OBT_PURGE_ALL

Primary

Created internally by NDS to identify an object whose attribute values need to be purged (so that only an object husk remains).

Move Subtree

0x000A

OBT_MOVE_ALL

Secondary

Created when a partition root object is moved from one container to another.

Moved From

0x000B

OBT_MOVED_FROM

Secondary

Created when an object is moved from one container to another.

Used By

0x000C

OBT_USED_BY

Secondary

Contains a list of partitions that have an interest in this object and need to be notified of changes to the modified object. The creation time of the primary obituary is stored on the Used By obituary value.


NOTE

Some Novell documentation and TIDs refers to the Used By obit as the Type C obit because of its value.


In addition to the obituary types and classes, obituaries move through four distinct states or stages. These states are always executed in the same order to ensure that the servers process obituaries properly and then purge them from the system. Obituary advancement through the four states occurs during the synchronization process. By observing the synchronization process, you can see the obituaries actually being purged. Listing 6.1 shows where obituaries appear in the synchronization process. Notice that the object User1.West.XYZCorp has two obituary entries: one of Type 2 ( Moved ) and one of Type 6 ( BackLink ). The obituary stage is shown in the flags= field.

Listing 6.1. Obituary State Advancement
 SYNC: Start sync of partition <[Root]> state:[0] type:[0]  SYNC: Start outbound sync with (#=2, state=0, type=1) [010000C3] <RIGEL.West.XYZCorp>  SYNC: Using version 5 on server <CN=RIGEL>    SENDING TO ------> CN=RIGEL   SYNC: sending updates to server <CN=RIGEL>    SYNC:[010000B8][(22:20:00),2,1] ORION.East.XYZCorp (NCP Server)    SYNC:[010002A4][(19:49:49),2,1] JimH.West.XYZCorp(User)    SYNC:[010000C3][(08:31:47),1,1] RIGEL.West.XYZCorp (NCP Server)  SYNC: [150002E4] obituary for User1.West.XYZCorp   valueTime=36905EB9,1,20 type=2, flags=0, oldCTS=36905E6F,1,1   valueTime=36905EB9,1,21 type=6, flags=0, oldCTS=36905E6F,1,1  SYNC:[150002E4][(00:04:05),1,1] User1.West.XYZCorp (User)    SYNC: [0E0002BC] obituary for User1.East.XYZCorp     valueTime=36905EB9,1,17 type=3, flags=0, oldCTS=36905EB9,1,1    SYNC:[0E0002BC][(23:24:57),1,1] User1.East.XYZCorp (User)   SYNC: Objects: 7, total changes: 74, sent to server <CN=RIGEL>  SYNC: update to server <CN=RIGEL> successfully completed  Merged transitive vector for [010000C3] <RIGEL.West.XYZCorp>   succeeded SYNC: SkulkPartition for <[Root]> succeeded SYNC: End sync of partition <[Root]> All processed = YES. 

The stages that an obituary passes through before it is deleted are shown in Table 6.2. The four stages are always followed in the order presented. When an obituary is marked Flags=0004 ( Purgeable ), it is then up to each server to purge it from its local database.

Table 6.2. Obituary Processing Stage Definitions

OBITUARY STAGE

FLAG VALUE

DESCRIPTION

Issued

This is the initial stage, in which an obit has been created and is ready for processing.

Notified

1

This stage indicates that the obituary is at the Notified stage, which means that the servers identified in the BackLink or Tree Move obituaries have been contacted and notified of the operation or action on an object.

Ok_to_Purge

2

At this stage, the obituary is being cleaned up on the local database of each server identified in the BackLink or Tree Move obituaries. This cleanup includes resolving all objects that reference the object with the obituary and informing them of the change (such as a deletion or a move).

Purgeable

4

At this stage, all servers are ready to purge the value or object from their local databases. The purge process essentially recovers the value to the free chain and enables it to be reused.


You might notice a couple weird things about the information in Table 6.2. The first oddity is the flag values ”the progression is , 1 , 2 , 4 . This field is referred to as a bit field . The software checks specific bits rather than looking for specific integer values.

The second thing that may appear strange is the Issued ( flags=0 ) and Ok_to_Purge ( flags=2 ) states; these states indicate the beginning or end of another stage rather than their own processing procedure. Stage 0 is initially set when an obituary is set; this change is then replicated to all servers. When the replication cycle is complete, DS knows that all servers are at Stage 0, and it can go ahead and start the notification process (Stage 1). A change in the obituary is made, and that information is replicated to the other servers that need to be notified. After all servers have been notified, the obituary is set to Stage 2, meaning that Stage 1 (notification) has completed. When all servers have received a flag indicating that it is okay to purge ( flags=2 ), the servers mark the obituaries as purgeable ( flags=4 ), and that change is replicated to all the servers. At this point, the individual servers process the actual purge process, but because all servers have to know that the obituary is now purgeable, no additional notification needs to be done after the obituaries have actually been purged.

NOTE

The four obituary processing stages actually describe a multiserver transaction processing system. You can think of the processing of obituaries as a synchronized transaction that takes place nearly simultaneously on multiple servers.


Before an obituary (regardless of its class) can move to the next state, the current state must have been synchronized to all replicas of the real object. In order to determine whether all replicas in the replica ring have seen a given obituary state, a time is computed from the Transitive Vector attribute of the partition that contains the obituary. If the modification timestamp (MTS) on the obituary is older than this time, the server responsible for that obituary can advance it to the next state.

NOTE

eDirectory 8.5 and previous versions of NDS use the Purge Vector attribute's time as the time to indicate when an obituary's state should be advanced. ( Purge Vector is a nonsynchronizing, single-valued attribute of the partition root object whose value, according to the Transitive Vector attribute (NetWare 5 and higher) or the Synchronized Up To attribute (NetWare 4) of this partition represents the oldest state in time that has been seen by each DSA in the replica ring. Purge Vector is updated only if the partition has a successful sync cycle. On the other hand, eDirectory 8.6 and higher use the Obituary Time Vector attribute ” a value stored in server memory that is recalculated each time the Janitor process runs (every two minutes, by default).


Primary obituaries can be advanced in their states only after all associated secondary obituaries have advanced through all their states. After the primary obituary reaches its last state and that state synchronizes to all servers in the replica ring, all that remains is the object "husk," which is an object without attributes ”an object that can subsequently be purged from the database by the Purge process. Tracking obituaries are removed after the primary obituary is ready to be removed or, in the case of OBT_INHIBIT_MOVE , the tracking obituary is removed after the primary obituary has moved to the flags=1 ( Notified ) state on the Master replica.

For a secondary obituary of type BackLink , the server that holds the Master replica of the object with the obituary is responsible for advancing the states. For a secondary obituary of type Used By , the server that created it is responsible for advancing the obituary states as long as that replica still exists. If it does not still exist, the server holding the Master of that partition takes over advancing the obituary states for the Used By obituary. For a Move Subtree obituary, the Master replica of the parent partition is responsible for advancing the states.

NOTE

The Obituary process is scheduled on a per-partition basis, after the partition finishes a successful inbound sync. If only one replica (the Master) of a partition exists, the Heartbeat interval still schedules an Outbound Replication process, which in turns kicks off the Obituary process.

The type of the obituary determines the replica responsible for processing the obits (the sender). With the exception of OBT_USED_BY , the Master replica is responsible for starting the background process. The processing of a Used By obit is started by the replica that actually modified the object. If this replica no longer exists, the Master replica then kicks off the background process.


The steps involved in obituary processing are complicated. However, the basic concept can be illustrated by using a somewhat simplified example. eDirectory performs the following operations when an object is deleted:

  1. eDirectory adds a primary obituary of type Dead to the "deleted" object and sets the flag to Issued . This takes place on the Master replica.

  2. eDirectory creates a secondary obit of the type BackLink and sets the stage flag to Issued for every server that has an external reference to this object; the server DNs are listed in the BackLink attribute of the object. Store the creation time of the Dead obit as part of this secondary obit.

  3. eDirectory creates a secondary obit of type BackLink and sets the stage flag to Issued for every server that holds a real replica of the object ”not an external reference. Store the creation time of the Dead obit as part of this secondary obit.

  4. eDirectory creates a secondary obit of type Used By and sets the stage flag to Issued for every DN listed in the Used By attribute of the deleted object. The Used By attribute contains a list of partitions (not servers) that have an interest in this object and need to be notified of changes to this entry. Store the creation time of the Dead obit as part of this secondary obit.

  5. eDirectory removes all attributes except the obituaries, which results in an object husk. The flag on the entry's Object Class attribute is set to Non Present , making the object "invisible" in most of the standard management tools, such as ConsoleOne (but not in DSBrowse or NDS iMonitor).

  6. The Outbound Replication process synchronizes the deletion of attributes to all other servers in the replica ring.

  7. After the next successful inbound synchronization of this partition, the Obituary process is started.

The Obituary process does the following:

  • Computes a time vector that is equivalent to the minimum Transitive Vector attribute, referred to as the Purge Vector attribute. eDirectory 8.6 and higher compute a second minimum vector, called the Obituary Time Vector attribute, which does not include timestamp values from Subordinate Reference replicas.

  • Categorizes each obituary in this partition and takes appropriate action:

    • If the obituary is a Used By obit and this server is the server where the deletion occurred (determined by comparing the replica number in the obituary's MTS to the replica number), this server is responsible for processing this obituary. Therefore, the main server notifies the other servers about this obit and sets the stage flag to Notified . The next time the Obituary process runs, this state flag is advanced to the next stage, until it reaches Purgeable (that is, after all partitions in the Used By attribute have been notified), and then it is purged.

    • If the obituary is a BackLink obit and this server has the Master replica, this server is responsible for processing this obituary. Therefore, the xxx notifies the other servers about this obit and sets the stage flag to Notified . The next time the Obituary process runs, this state flag is advanced to the next stage, until it reaches Purgeable (that is, after all servers in the BackLink attribute have been notified), and then it is purged.

    • If the obituary is a primary obituary (such as a Dead obit, in this example), there are no secondary obituaries outstanding for this primary obit, and the attribute's MTS on the obituary is older than the Purge Vector / Obit Time Vector attribute, the obit's flag value can be set to Purgeable because all servers have seen the change.

    • When the obit value flag on the primary obit is set to Purgeable , the Purger process, also known as the Flat Cleaner process, removes the object's record (which is no longer flagged as Present ) from the database and completes the deletion action.

Because stuck obits ”that is, servers not clearing out obits from their local databases, thus preventing certain operations from being performed ”are the source of many NDS/eDirectory problems, your having a good grasp of obituaries' dependency of other DS processes is important to understanding DS background processes, which are discussed next.



Novell's Guide to Troubleshooting eDirectory
Novells Guide to Troubleshooting eDirectory
ISBN: 0789731460
EAN: 2147483647
Year: 2003
Pages: 173

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net