Overview of the Database Reorganization Process | An Introduction to IMS: Your Complete Guide to IBMs Information Management System

The database reorganization process can vary from being very simple to very complex, depending on the databases involved. If the databases involved do not have IMS logical relationships or secondary indexes, then the process is very simple. When logical relationships and secondary indexes are involved, the process becomes more involved.

IMS supplies utility programs that help you with the reorganization process. They are mentioned in the following sections that discuss the database reorganization process. An overview of when you would use which utility is in "Offline Reorganization Utilities" on page 130.

Most database types must be reorganized while they are offline (unavailable for updatessee "Offline Reorganization" on page 129). DEDBs and IMS Version 9 HALDBs can be reorganized while they are online (available for updatessee "Online Reorganization" on page 144).

Reorganizing HALDBs

Both PHDAM and PHIDAM HALDBs can be reorganized online or offline. A PSINDEX HALDB can only be reorganized offline. Whether you are reorganizing your HALDB online or offline, the reorganization process is different from the reorganization processes used for other full-function databases.

You can reorganize HALDBs online using the HALDB Online Reorganization function (OLR) that is available with IMS Version 9 and later. HALDB OLR maintains the availability of your data during the reorganization process, even in the partition it is actively reorganizing.

Reorganizations of HALDBs with logical relationships and secondary indexes do not require the execution of utilities to update these pointers. Instead, HALDB uses a self-healing pointer process to correct these pointers when they are used.

Related Reading: For more information about reorganizing HALDBs offline, see "Reorganizing HALDBs Offline" on page 142. For more information about reorganizing HALDBs online, see "Reorganizing HALDBs Online" on page 145.

Offline Reorganization

The offline reorganization process, in its simplest form, is to unload the database, delete and redefine the physical data set, and then reload it. If the database is not involved in any logical relationships and does not have any secondary indexes, then that is the complete process. To reorganize HD databases that have both logical relationships and secondary indexes, complete the following steps:

1.	Back up the databases (both the data and, if you are changing them, the appropriate control blocks; for example, DBDs) so that you have a fallback point if there are any problems during the reorganization. See Chapter 11, "The Database Recovery Process," on page 151 for more information.
2.	Unload the existing database data sets to sequential files using the IMS utilities. The process is discussed in "Database Unload Process" on page 132.
3.	Delete the database data sets. If you are making any changes to the definitions of the database data sets, make them now, remembering to save the old definitions as a fallback.
4.	Redefine the database data sets.
5.	This step is necessary only if you are making any changes to the database structure by altering the DBD. Make the changes to the DBD and reassemble it by running the Database Description Generation (DBDGEN) utility. Then run the Application Control Blocks Maintenance utility (ACBGEN) utility with the DBD= parameter to ensure all appropriate control blocks are regenerated. You must make sure that all programs and utilities use the new versions of the control blocks if you change the DBD; otherwise, database corruption will result.
6.	Run the IMS utilities to reload the database. If you altered the DBD, the reload utility, and any subsequent programs or utilities, should use the new DBD.
7.	If the database has secondary indexes or participates in logical relationships, then you must run additional utilities to rebuild these connections. These connections (unless using symbolic pointers) rely on the database segments' relative position in the database, which has been altered by the reorganization. The utilities will determine the new positions and amend the direct pointers in the indexes and logically related databases.
8.	If your databases are registered with DBRC (as they should be), then you must take an image copy of the reorganized databases. Taking an image copy provides you with a backup copy of the database and establishes a point of recovery with DBRC (in case of system failure). IMS database forward recovery, using changes recorded in IMS logs, relies on the position of the segments relative to the start of the data set, which is altered by the reorganization. You must take the image copies to establish a new base from which the databases can be rolled forward.

Offline Reorganization Utilities

The IMS-supplied utilities that perform offline database reorganization are completely described in IMS Version 9: Administration Guide: Database Manager and IMS Version 9: Utilities Reference: Database and Transaction Manager. The following sections briefly describe the database reorganization utilities.

The database reorganization utilities fall into three groups, based on the type of reorganization you plan to perform:

"Partial Offline Database Reorganization"
"Offline Database Reorganization Using the Utility Control Facility" on page 131
"Offline Database Reorganization without Using the Utility Control Facility" on page 131

Partial Offline Database Reorganization

If you are reorganizing an HD database, you can reorganize parts of it instead of the entire database. There are reasons why you might want to reorganize parts of the database:

Only parts of it need to be reorganized.
You can minimize the amount of time it takes to do a total reorganization by breaking the total time into smaller pieces.

The IMS utilities that perform a partial reorganization are:

The Database Surveyor utility (DFSPRSUR), which helps you determine which parts of your database to reorganize.
The Partial Database Reorganization utility (DFSPRCT1 and DFSPRCT2), which does the actual physical reorganization.

Note:

The Partial Database Reorganization utility does not apply to HALDBs. Also, the Partial Database Reorganization utility needs a lot of information from the database to perform its functions and is not usually considered to be worthwhile compared with reorganizing the whole database.

Offline Database Reorganization Using the Utility Control Facility

You can reorganize a database using a single program, called the Utility Control Facility (DFSUCF00), or by using various combinations of the utilities. When you use UCF, it acts as a controller, determining which of the various reorganization utilities needs to be run, and then running them. The Utility Control Facility:

Reduces the number of JCL statements you must create.
Eliminates the need to sequence the running of the various utilities.
Allows you to stop, and then later restart, a job.
Reduces the number of decisions operations people must make.

Note:

The only reorganization utilities that cannot be run under the control of the Utility Control Facility are the Database Surveyor utility and the Partial Database Reorganization utility. Also, the Utility Control Facility does not support HALDBs.

Offline Database Reorganization without Using the Utility Control Facility

When you do not use UCF, reorganization of the database is done using a combination of utilities. Which utilities you need to use, and how many, depends on the type of database and whether it uses logical relationships or secondary indexes.

If your database does not use logical relationships or secondary indexes, you simply run the appropriate unload and reload utilities:

For HISAM databases, the HISAM Reorganization Unload utility and the HISAM Reorganization Reload utility
For HIDAM index databases (if reorganized separately from the HIDAM database), the HISAM Reorganization Unload utility and the HISAM Reorganization Reload utility or the VSAM REPRO utility
For SHISAM, HDAM, and HIDAM databases, the HD Reorganization Unload utility and the HD Reorganization Reload utility

If your database uses logical relationships or secondary indexes, you need to run the HD Reorganization Unload and Reload utilities (even if it is a HISAM database). In addition, you must run a variety of other utilities to collect, sort, and restore pointer information from a segment's prefix. Remember, when a database is reorganized, the location of segments changes. If logical relationships or secondary indexes are used, update prefixes to reflect new segment locations. The various utilities involved in updating segment prefixes are:

Database Prereorganization utility
Database Scan utility (DFSURGS0)
Database Prefix Resolution utility
Database Prefix Update utility

Note:

The Database Prefix Resolution and Database Prefix Update utilities do not apply to HALDBs.

Recommendation:

Use the DBRC GENJCL command to generate the JCL necessary to run the reorganization utilities. For complete details about the DBRC commands, see IMS Version 9: Database Recovery Control (DBRC) Guide and Reference.

Database Unload Process

The unload processing for HD databases is very simple. The HD Reorganization Unload utility (DFSURGU0) unloads the main database and the primary index data set if the database is HIDAM. The output of this utility is a sequential data set, which is input to the HD Reorganization Reload utility (DFSURGL0). If the database is a HIDAM database, then the primary index database must also be present. The HD Reorganization Unload utility can unload only a single database at a time. If there are logically related databases that are to be reorganized at the same time, then the step should be executed once for each database. Figure 10-1 shows a diagram of the inputs and output for the HD Reorganization Unload utility.

Figure 10-1. Database Unload Processing

Keep the following information in mind when you plan for the database unload process:

IBM highly recommends that you make an image copy of the database before you attempt to reorganize it.
The database is allocated by the ddnames in the DBD. Dynamic allocation cannot be used; the database DD statements must be present.
If a HIDAM database is being unloaded, the primary index database DD statement must also be present.
The HD Reorganization Unload utility checks with DBRC for database registration. If the database is registered, then the HD Reorganization Unload utility requests RD access authorization. HD Reorganization Unload utility is allowed to authorize the database even if the PROHIBIT AUTH flag is set on.
If the database is being reorganized to change the structure of the database, then the old DBD definition should be used.
Regardless of how many database data set groups the database is divided into, there is only one output data set.
The HD Reorganization Unload utility can unload only one database per job step. To unload multiple databases, you must use multiple job steps.

Defining Databases

After unloading the database and before reloading it, you must define the new database.

If the access method used for the database is VSAM, then an IDCAMS job step is required to delete and redefine the VSAM cluster. The IMS reload utilities will fail if the data sets are not empty. If OSAM is used, specify DISP=OLD in the DD statement to overwrite the data set. If, however, the database is on more than a single DASD volume, IBM highly recommends that you delete the data set and redefine it (using the z/OS program IEFBR14 to preallocate the data sets) to ensure that the correct end-of-file marker is placed.

Database Reload Process

The reload process for HD databases can be more complex than the unload process. If the database does not have any secondary indexes and is not involved in a logical relationship, then the database can simply be reloaded.

If there are no secondary indexes or logical relationships involved, the reloading of the database is the same as the unloading: simply reload the database.

If, however, there are secondary indexes or logical relationships involved, you must run additional utility programs before and after the database is reorganized to rebuild the logical relationships and secondary indexes so that they reflect the new physical positions of the segments in the reorganized database. Until all this processing is complete, the logical relationships and secondary indexes are not usable. If you attempt to use them before completing this process, the applications will fail with IMS abend codes indicating that there are invalid IMS pointers.

The following sections discuss each combination of reload processing required:

"Reload Only"
"Reload with Secondary Indexes" on page 135
"Reload with Logical Relationships" on page 137
"Reload with Logical Relationships and Secondary Indexes" on page 138

Reload Only

The reload processing for an HD database without any logical relationships or secondary indexes is shown in Figure 10-2.

Figure 10-2. Simple Database Reload Processing

Keep the following information in mind when you plan for the database reload process:

The database is allocated by the ddnames in the DBD. Dynamic allocation cannot be used; the database DD statements must be present.
If a HIDAM database is being reloaded, the primary index database DD statement must also be present.
The HD Reorganization Reload utility checks with DBRC for database registration. If the database is registered, the HD Reorganization Reload utility requests EX access authorization. The HD Reorganization Reload utility is allowed to authorize the database even if the PROHIBIT AUTH flag is set on.
If the database is being reorganized to change the structure of the database, the new DBD definition should be used.
Regardless of how many database data set groups the database is divided into, there is only one input data set.
The HD Reorganization Reload utility can reload only one database per job step. To reload multiple databases, you must use multiple job steps.
The DFSURWF1 DD statement can be specified as DUMMY, but it must be present.

Reload with Secondary Indexes

Figure 10-3 on page 136 illustrates the reload process when secondary indexes are involved. The process is as follows:

1.	Identify the databases involved in secondary index relationships using the Database Prereorganization utility (DFSURPR0). This utility creates a control file (DFSURCDS) that contains this information. DFSURCDS is then used subsequently by other utilities.
2.	The databases are reloaded using the HD Reorganization Reload utility. This utility updates the RECON data set and also generates a work data set (DFSURWF1) that is used as input to the Database Prefix Resolution utility.
3.	The Database Prefix Resolution utility is used to extract and sort the RBAs. This utility generates a work data set (DFSURIDX) that contains the unload secondary index segments and is used as input to the HISAM Reorganization Unload utility (DFSURUL0).
4.	The HISAM Reorganization Unload utility reads the DFSURIDX data set and creates load files for each secondary index database. The secondary index databases can be empty.
5.	The HISAM Reorganization Reload utility (DFSURRL0) reloads the secondary index databases and also updates the RECON data set. The HISAM Reorganization Reload utility can reload all the secondary index databases in one job step.

Figure 10-3. Database Reload Processing with Secondary Indexes

Considerations to keep in mind:

The database is allocated by the ddnames in the DBD. Dynamic allocation cannot be used; the database DD statements must be present.
If a HIDAM database is being reloaded, the primary index database DD statement must also be present.
The HISAM Reorganization Reload utility checks with DBRC for database registration. If the database is registered, then the utility requests EX access authorization. The HISAM Reorganization Reload utility is allowed to authorize the database even if the PROHIBIT AUTH flag is set on.
If the database is being reorganized to change the structure of the database, then the new DBD definition should be used.
Regardless of how many database data set groups the database is divided into, there is only one input data set.
The HISAM Reorganization Reload utility can reload only one database per job step. To reload multiple databases, you must use multiple job steps.
The DFSURWF1 DD statement can be specified as DUMMY, but it must be present.

Reload with Logical Relationships

Figure 10-4 on page 138 illustrates the reload process when logical relationships are involved. The process is as follows:

1.	Identify the databases involved in logical relationships using the Database Prereorganization utility. This utility creates a control file (DFSURCDS) that contains this information. DFSURCDS is then used subsequently by other utilities.
2.	The databases are reloaded using the HD Reorganization Reload utility. This utility updates the RECON data set and also generates a work data set (DFSURWF1) that is used as input to the Database Prefix Resolution utility. If all the databases that are logically related to each other are being reloaded (and they normally are), use the DBIL option on the control statement. The DBIL option resets all the pointers and logical parent counters. If not, then use the DBR option.
3.	The Database Prefix Resolution utility is used to accumulate and sort the information that is in the DFSURWF1 working data set. This utility generates a work data set (DFSURWF3) that contains the sorted prefix information and is used as input to the Database Prefix Update utility (DFSURGP0).
4.	The Database Prefix Update utility updates the database segment prefixes. The prefix fields updated by this utility include the logical parent, logical twin, and logical child pointer fields, and the counter fields associated with logical parents. Log output data sets are optionally created if log DD statements are included in the JCL. This log output can be used if a later database recovery is required.

Figure 10-4. Database Reload Processing with Logical Relationships

Considerations to keep in mind:

The database is allocated by the ddnames in the DBD. Dynamic allocation cannot be used; the database DD statements must be present.
If a HIDAM database is being reloaded, the primary index database DD statement must also be present.
The HD Reorganization Reload utility checks with DBRC for database registration. If the database is registered then the HD Reorganization Reload utility requests EX access authorization. The HD Reorganization Reload utility is allowed to authorize the database even if the PROHIBIT AUTH flag is set on.
If the database is being reorganized to change the structure of the database, then the new DBD definition should be used.
The HD Reorganization Reload utility can reload only one database per job step. To reload multiple databases, use multiple job steps.
The DFSURWF1 DD statement must be present.
The Database Prefix Update utility acquires EX access to the databases being updated.
The IMAGE COPY NEEDED flag is set on (in the RECON data set) by the HD Reorganization Reload utility.

Reload with Logical Relationships and Secondary Indexes

The reload processing for both secondary indexes and logical relationships is a combination of both the individual reload processes described in "Reload Only" on page 134, "Reload with Secondary Indexes" on page 135, and "Reload with Logical Relationships" on page 137.

Figure 10-5 on page 140 illustrates the database reload process when both secondary indexes and logical relationships are involved. The process is as follows:

Figure 10-5. Database Reload Processing with Secondary Indexes and Logical Relationships

1.	Identify the databases involved in secondary indexes and logical relationships using the Database Prereorganization utility. This utility creates a control file (DFSURCDS) that contains this information. DFSURCDS is then used subsequently by other utilities.
2.	The databases are reloaded using the HD Reorganization Reload utility. This utility updates the RECON data set and also generates a work data set (DFSURWF1) that is used as input to the Database Prefix Resolution utility.
3.	The Database Prefix Resolution utility is used to accumulate and sort the information that is in the DFSURWF1 working data set. This utility generates two work data sets: DFSURWF3, which contains the sorted prefix information and is used as input to the Database Prefix Update utility. DFSURIDX, which contains the unload secondary index segments and is used as input to the HISAM Reorganization Unload utility.
4.	The Database Prefix Update utility updates the database segment prefixes. The prefix fields updated by this utility include the logical parent, logical twin, and logical child pointer fields, and the counter fields associated with logical parents. Log output data sets are optionally created if log DD statements are included in the JCL. This log output can be used if a later database recovery is required.
5.	The HISAM Reorganization Unload utility reads the DFSURIDX data set and creates load files for each secondary index database. The secondary index databases can be empty.
6.	The HISAM Reorganization Reload utility reloads the secondary index databases and also updates the RECON data set. The HISAM Reorganization Reload utility can reload all the secondary index databases in one job step.

Considerations to keep in mind:

The database is allocated by the ddnames in the DBD. Dynamic allocation cannot be used and the database DD statements must be present.
If a HIDAM database is being reloaded, the primary index database DD statement must also be present.
The HD Reorganization Reload utility checks with DBRC for database registration. If the database is registered then the HD Reorganization Reload utility requests EX access authorization. The HD Reorganization Reload utility is allowed to authorize the database even if the PROHIBIT AUTH flag is set on.
If the database is being reorganized to change the structure of the database, use the new DBD definition.
The HD Reorganization Reload utility can only reload one database per job step. To reload multiple databases use multiple job steps.
The DFSURWF1 DD statement must be present.
The Database Prefix Update utility acquires EX access to the databases that are being updated.
The IMAGE COPY NEEDED flag is set on (in the RECON data set) by the HD Reorganization Reload utility.

Reorganizing Fast Path DEDBs Offline

The process for reorganizing a Fast Path DEDB can be appreciably different from reorganizing full-function databases.

If you are reorganizing a DEDB only to reclaim fragmented free space or get the best placement of the segments for performance (that is, the DBD or data set definitions are not being changed), then you can run the High-Speed DEDB Direct Reorganization utility (DBFUHDR0). This utility can be run while the DEDB is either offline or online. In this instance, offline means that the DEDB is still up and available, but you have prevented applications from accessing it. The DEDB is still technically online.

Related Reading: For more information about the High-Speed DEDB Direct Reorganization utility (DBFUHDR0), see IMS Version 9: Utilities Reference: Database and Transaction Manager.

If you are reorganizing a DEDB to alter the structure, then you need to have your own user-written programs to unload and reload the database data set at the appropriate points, or use the DEDB Unload and Reload utilities from the separately priced IBM IMS High Performance Fast Path Utilities, V2.1 (program number 5655K94). You must also run the DEDB Initialization utility (DBFUMIN0), provided with the IMS base product, immediately before reloading the database. You do not have to run additional utilities after the database is reloaded because DEDBs do not support secondary indexes or logical relationships.

Related Reading: For more information about:

The separately priced tools available from IBM, see Chapter 26, "IBM IMS Tools," on page 443.
The database reorganization process and the steps you must take to alter specific attributes of the structure of a DEDB, see the chapters on monitoring and tuning databases in IMS Version 9: Administration Guide: Database Manager.

Reorganizing HALDBs Offline

The offline reorganization processes for a HALDB database and other full-function IMS databases are similar: they both consist of an unload and reload of the database; however, the HALDB reorganization process has advantages over the reorganization process of other full-function databases, such as:

You can reorganize one HALDB partition at a time, or reorganize multiple partitions in parallel.
The self-healing pointer process of HALDBs eliminates the need to manually update logical relationships and secondary indexes after reorganizing a HALDB.
You do not need to include DD statements for HALDB data sets when you reorganize a HALDB. HALDB data sets are dynamically allocated.

An offline reorganization of a HALDB database can be done with one or more parallel processes. These processes unload one or more partitions and reload them. If the database has secondary indexes or logical relationships, additional steps are not required. The amount of time required for a reorganization depends on the sizes of the partitions. Smaller partitions reduce the time. You can reduce your reorganization time by creating more partitions and reorganizing them in parallel.

The basic steps involved in reorganizing a HALDB offline are:

1.	Run the HD Reorganization Unload utility to unload the entire database, a range of partitions, or a single partition.
2.	Optionally, initialize the partitions by running either of the following utilities: HALDB Partition Data Set Initialization utility (DFSUPNT0) Database Prereorganization utility
3.	Run the HD Reorganization Reload utility to reload the database or partitions.
4.	Make image copies of all reloaded partition data sets.

Figure 10-6 on page 143 shows the offline processes used to reorganize a HALDB database with logical relationships and secondary indexes. In this case, the partitions are reorganized by parallel processes. Each partition can be unloaded and reloaded in less time than unloading and reloading the entire database. This is much faster than the process for a non-HALDB full-function database. Additionally, no time is required for updating pointers in the logically related database or rebuilding secondary indexes. This further shortens the process.

Figure 10-6. Offline Reorganization of a HALDB database

Reorganizing HALDB Partitioned Secondary Index Databases

You might need to reorganize your partitioned secondary index (PSINDEX) database. Because the reorganization of HALDBs does not require the recreation of their secondary indexes, a PSINDEX database can become disorganized as entries are added to it over time.

The HD Reorganization Unload utility and the HD Reorganization Reload utility can be used to reorganize PSINDEX databases. The restrictions and recommendations for reorganizing other HALDB databases also apply to PSINDEX databases with one exception: HALDB secondary indexes have no ILDSs. The HD Reorganization Reload utility control statements should not be used with secondary indexes.

The steps for reorganizing a PSINDEX database are the same as those for reorganizing other types of HALDBs offline.

Options for Offline Reorganization of HALDBs

You have several options when reorganizing a HALDB database:

You can reorganize any number of partitions. If you need to reorganize only one partition, you can unload and reload it without processing other partitions.
You can reorganize partitions in parallel or you can reorganize the database with one process. The degree of parallelism is determined by the number of reorganization jobs you run. Each job can process one or multiple partitions. To increase the parallelism you can increase the number of reorganization jobs and decrease the number of partitions each job processes.
You can reuse existing database data sets or you can delete them after they are unloaded and allocate new data sets for the reload.
You can add partitions, delete partitions, or change partition boundaries.

Online Reorganization

With offline reorganization, the database is unavailable during the reorganization process. With online reorganization, which is available only for HALDB and DEDB databases, the database remains available for updates during the reorganization process.

Related Reading: For complete information about HALDB Online Reorganization, see IMS Version 9: Administration Guide: Database Manager.

Reorganizing Fast Path DEDBs Online

As mentioned in "Reorganizing Fast Path DEDBs Offline" on page 141, if you are only reorganizing to reclaim fragmented free space or to get the best placement of the segments for performance (that is, the DBD or data set definitions are not being changed), or both, you can run the High-Speed DEDB Direct Reorganization utility while the DEDB is online.

Related Reading: For more information about the High-Speed DEDB Direct Reorganization utility, see IMS Version 9: Utilities Reference: Database and Transaction Manager.

Reorganizing HALDBs Online

Prior to IMS Version 9, you had to ensure that HALDB partitions were offline before you could perform database reorganization for them. IMS Version 9 introduced an integrated HALDB Online Reorganization function that allows HALDB partitions to remain online and available for IMS application programs during a database reorganization.

An online reorganization of PHDAM or PHIDAM HALDB partitions runs non-disruptively, allowing concurrent IMS updates, including updates by data sharing IMS systems. The online reorganization is non-disruptive because IMS copies small amounts of data from the partition's original data sets (the input data sets) to separate output data sets. IMS tracks which data has been copied so that IMS applications can automatically retrieve or update data from the correct set of data sets.

When you reorganize either PHDAM or PHIDAM HALDB partitions online, the online reorganization process requires additional data sets to hold the reorganized data. The additional data sets are equal in number to the data sets being reorganized, excluding the ILDS.

In a PHDAM database, online reorganization increases the maximum number of data sets associated with a partition to twenty-one. In a PHIDAM database, which includes a primary index, online reorganization increases the maximum number of data sets associated with a partition to twenty-three. In either case, online reorganization needs only as many new data sets as the data sets that exist in the partition at the time the reorganization process begins.

Each additional data set requires an additional ddname. To distinguish between the ddnames for the data sets that are being reorganized and the ddnames for the data sets into which online reorganization moves the reorganized data, HALDB online reorganization extends the HALDB naming convention to include the suffixes M through V for the ddnames of the primary data sets and the suffix Y for the ddname for the additional primary index. The suffixes M through V and Y correspond in order to the standard HALDB ddname suffixes A through J and X.

The initial load or offline reorganization reload of a HALDB partition always uses the A-through-J (and X) data sets. Until the first time that you reorganize a HALDB partition online, only the A-through-J (and X) data sets are used.

There are three phases of online reorganization for a HALDB partition:

The initialization phase, during which IMS prepares the output data sets and updates the RECON data sets.
The copying phase, during which IMS performs the actual reorganization by copying the data from the input data sets to the output data sets.
The termination phase, during which IMS closes the input data sets and updates the RECON data sets.

The Initialization Phase for HALDB Online Reorganization

You start the online reorganization of a HALDB partition using the INITIATE OLREORG command. See the IMS Version 9: Command Reference for more information about this command.

During the initialization phase, IMS updates the RECON data sets to establish the ownership of the online reorganization by the IMS subsystem that is performing the online reorganization. This ownership means that no other IMS subsystem can perform a reorganization of the HALDB partition until the current online reorganization is complete or until ownership is transferred to another IMS subsystem. IMS adds the M-through-V (and Y) DBDSs to the RECON data sets if those DBDS records do not already exist. IMS also adds the M-through-V (and Y) DBDSs to any existing change accumulation groups and DBDS groups that include the corresponding A-through-J (and X) DBDSs.

Definition:

Change accumulation is the process of creating a compacted version of one or more IMS log data sets by eliminating records not related to recovery, and by merging multiple changes to a single segment into a single change. For more information about change accumulation, see "Database Change Accumulation Utility" on page 158.

Before online reorganization begins for a HALDB partition, there is a single set of active data sets for the HALDB partition. These active data sets are the input data sets for the copying phase. There might also be a set of inactive data sets from a prior online reorganization that is not used by IMS application programs.

During the initialization phase, IMS evaluates each of the inactive data sets to ensure that it meets the requirements for output data sets. If any of the output data sets does not exist, IMS creates it automatically during this phase.

At the end of the initialization phase, IMS considers the original active set of data sets as the input set and the inactive data sets as the output set. This use of the input and output sets of data sets is represented by the cursor-active status for the partition, which is recorded in online reorganization records in the RECON data sets. A listing of the partition's database record in the RECON data sets shows OLREORG CURSOR ACTIVE=YES. A listing of the partition also shows that both sets of DBDSs are active; the first set of DBDSs listed is for the input data set, and the second set of DBDSs is for the output data set, for example, DBDS ACTIVE=A-J and M-V. While the partition is in the cursor-active status, both sets of data sets must be available for the partition to be processed by any application.

The Copying Phase for HALDB Online Reorganization

During the copying phase, the HALDB partition is comprised of the A-through-J (and X) data sets and the M-through-V (and Y) data sets. During this phase, both sets of data sets must be available in order for IMS applications to access the partition.

While IMS reorganizes a HALDB partition online, IMS applications can make database updates to the partition. Some of the database updates are made to the input data sets, while others are made to the output data sets, depending on which data is updated by the application. Which data sets are updated is transparent to the application program. Figure 10-7 illustrates the relationship between the input and output data sets at a point during the online reorganization.

Figure 10-7. The Relationship between Input Data Sets and Output Data Sets during the Online Reorganization of a HALDB Partition

Figure 10-7 shows two sets of database data sets for a HALDB partition, the input data sets that have not been reorganized and the output data sets that have been (at least partially) reorganized. The figure shows the reorganization as progressing from left to right, from the input data sets above to the output data sets below. The data sets in the figure are divided into four areas:

1.	An area within the input data sets that represents the data that has been copied to the output data sets. This area reflects the old data organization (prior to the reorganization) and is not used again by IMS applications until the data sets are reused as the output data sets for a later online reorganization.
2.	An area within the output data sets that represents the data that has been copied from the input data sets. This data in this area has been reorganized and can be used by IMS applications during the reorganization.
3.	An area within both the input and output data sets that represents data that is locked and in the process of being copied and reorganized from the input data sets to the output data sets. This area of locked records is called a unit of reorganization. From a recovery point of view, this unit of reorganization is equivalent to a unit of recovery. While IMS processes the current unit of reorganization, IMS applications that access any of the locked data records must wait until IMS completes the reorganization for those records. After the copying and reorganization completes for the unit of reorganization, IMS commits the changes and unlocks the records, thus making them available again for IMS applications.
4.	An area within the input data sets that represents the data that has not yet been copied to the output data sets. This data has also not yet been reorganized and can be used by IMS applications during the reorganization.

As the online reorganization progresses, IMS uses a type of pointer called a cursor to mark the end point of those database records that have already been copied from the input data sets to the output data sets. As the reorganization and copying proceeds, this cursor moves forward through the partition, from left to right in the figure.

When an IMS application program accesses data from a HALDB partition that is being reorganized online, IMS retrieves the data record:

From the output data sets if the database record is located at or before the cursor.
From the input data sets if the database record is located after the cursor.

If the data record happens to fall within the unit of reorganization, IMS retries the data access after the records are unlocked. An application program does not receive an error status code for data within a unit of reorganization.

To allow recovery of either an input data set or an output data set, all database changes are logged during the online reorganization, including the database records that are copied from the input data set to the output data sets.

The Termination Phase for HALDB Online Reorganization

The online reorganization of a HALDB partition terminates after the end of the copying phase, or when IMS encounters an error condition during the reorganization. You can also stop the online reorganization of a HALDB partition using the TERMINATE OLREORG command. See IMS Version 9: Command Reference for more information about this command.

After the copying phase is complete for a HALDB partition, the output data sets become the active data sets, and the input data sets become the inactive data sets. The active data sets are used for all data access by IMS application programs. The inactive data sets are not used by application programs, but can be reused for a subsequent online reorganization. Unless you perform an initial load or a batch reorganization reload for the partition, successive online reorganizations for the partition alternate between these two sets of data sets.

IMS updates the partition's database record in the RECON data sets to reset the cursor-active status for the partition to reflect that there is now just one set of data sets. A listing of this record from the RECON data sets shows OLREORG CURSOR ACTIVE=NO and the ACTIVE DBDS field shows the active (newly reorganized) data sets. IMS also updates the online reorganization records in the RECON data sets with the timestamp when the reorganization completed.

If you specified the DEL keyword for the INITIATE OLREORG command (or the UPDATE OLREORG command), IMS deletes the inactive data sets after resetting the cursor-active status for the partition. Before deleting the inactive data sets, IMS notifies all sharing IMS subsystems (including batch jobs) that the online reorganization is complete and is recorded in the RECON data sets. The IMS subsystem that is performing the online reorganization waits until it receives an acknowledgement from each of these sharing subsystems that they have closed and deallocated the now-inactive data sets, and then it deletes these data sets. However, if the acknowledgements are not received within 4.5 minutes, the owning subsystem will attempt to delete the inactive data sets anyway. Without the acknowledgements, the deletion attempt is likely to fail.

Finally, at the end of the termination phase, IMS updates the RECON data sets to reset the ownership of the online reorganization so that no IMS subsystem has ownership. This resetting of ownership means that any IMS subsystem can perform a subsequent reorganization of the HALDB.

If online reorganization of a HALDB partition terminates prior to completion, either because of an error or because you issued the TERMINATE OLREORG command, you need to restart the online reorganization or you need to perform an offline reorganization for the partition.

Figure 10-8 on page 150 shows the normal processing steps of a successful online reorganization of a HALDB partition. The columns represent the flow of control through the phases of the online reorganization, from the user to IMS, and the status of the data sets as the processing events occur.

Figure 10-8. The Normal Processing Steps of HALDB Online Reorganization

Related Reading: For more information about reorganizing HALDBs online, see IMS Version 9: Administration Guide: Database Manager.