The database reorganization process can vary from being very simple to very complex, depending on the databases involved. If the databases involved do not have IMS logical relationships or secondary indexes, then the process is very simple. When logical relationships and secondary indexes are involved, the process becomes more involved.
IMS supplies utility programs that help you with the reorganization process. They are mentioned in the following sections that discuss the database reorganization process. An overview of when you would use which utility is in "Offline Reorganization Utilities" on page 130.
Most database types must be reorganized while they are offline (unavailable for updatessee "Offline Reorganization" on page 129). DEDBs and IMS Version 9 HALDBs can be reorganized while they are online (available for updatessee "Online Reorganization" on page 144).
Both PHDAM and PHIDAM HALDBs can be reorganized online or offline. A PSINDEX HALDB can only be reorganized offline. Whether you are reorganizing your HALDB online or offline, the reorganization process is different from the reorganization processes used for other full-function databases.
You can reorganize HALDBs online using the HALDB Online Reorganization function (OLR) that is available with IMS Version 9 and later. HALDB OLR maintains the availability of your data during the reorganization process, even in the partition it is actively reorganizing.
Reorganizations of HALDBs with logical relationships and secondary indexes do not require the execution of utilities to update these pointers. Instead, HALDB uses a self-healing pointer process to correct these pointers when they are used.
Related Reading: For more information about reorganizing HALDBs offline, see "Reorganizing HALDBs Offline" on page 142. For more information about reorganizing HALDBs online, see "Reorganizing HALDBs Online" on page 145.
The offline reorganization process, in its simplest form, is to unload the database, delete and redefine the physical data set, and then reload it. If the database is not involved in any logical relationships and does not have any secondary indexes, then that is the complete process. To reorganize HD databases that have both logical relationships and secondary indexes, complete the following steps:
Offline Reorganization Utilities
The IMS-supplied utilities that perform offline database reorganization are completely described in IMS Version 9: Administration Guide: Database Manager and IMS Version 9: Utilities Reference: Database and Transaction Manager. The following sections briefly describe the database reorganization utilities.
The database reorganization utilities fall into three groups, based on the type of reorganization you plan to perform:
Partial Offline Database Reorganization
If you are reorganizing an HD database, you can reorganize parts of it instead of the entire database. There are reasons why you might want to reorganize parts of the database:
The IMS utilities that perform a partial reorganization are:
The Partial Database Reorganization utility does not apply to HALDBs. Also, the Partial Database Reorganization utility needs a lot of information from the database to perform its functions and is not usually considered to be worthwhile compared with reorganizing the whole database.
Offline Database Reorganization Using the Utility Control Facility
You can reorganize a database using a single program, called the Utility Control Facility (DFSUCF00), or by using various combinations of the utilities. When you use UCF, it acts as a controller, determining which of the various reorganization utilities needs to be run, and then running them. The Utility Control Facility:
The only reorganization utilities that cannot be run under the control of the Utility Control Facility are the Database Surveyor utility and the Partial Database Reorganization utility. Also, the Utility Control Facility does not support HALDBs.
Offline Database Reorganization without Using the Utility Control Facility
When you do not use UCF, reorganization of the database is done using a combination of utilities. Which utilities you need to use, and how many, depends on the type of database and whether it uses logical relationships or secondary indexes.
If your database does not use logical relationships or secondary indexes, you simply run the appropriate unload and reload utilities:
If your database uses logical relationships or secondary indexes, you need to run the HD Reorganization Unload and Reload utilities (even if it is a HISAM database). In addition, you must run a variety of other utilities to collect, sort, and restore pointer information from a segment's prefix. Remember, when a database is reorganized, the location of segments changes. If logical relationships or secondary indexes are used, update prefixes to reflect new segment locations. The various utilities involved in updating segment prefixes are:
The Database Prefix Resolution and Database Prefix Update utilities do not apply to HALDBs.
Use the DBRC GENJCL command to generate the JCL necessary to run the reorganization utilities. For complete details about the DBRC commands, see IMS Version 9: Database Recovery Control (DBRC) Guide and Reference.
Database Unload Process
The unload processing for HD databases is very simple. The HD Reorganization Unload utility (DFSURGU0) unloads the main database and the primary index data set if the database is HIDAM. The output of this utility is a sequential data set, which is input to the HD Reorganization Reload utility (DFSURGL0). If the database is a HIDAM database, then the primary index database must also be present. The HD Reorganization Unload utility can unload only a single database at a time. If there are logically related databases that are to be reorganized at the same time, then the step should be executed once for each database. Figure 10-1 shows a diagram of the inputs and output for the HD Reorganization Unload utility.
Figure 10-1. Database Unload Processing
Keep the following information in mind when you plan for the database unload process:
After unloading the database and before reloading it, you must define the new database.
If the access method used for the database is VSAM, then an IDCAMS job step is required to delete and redefine the VSAM cluster. The IMS reload utilities will fail if the data sets are not empty. If OSAM is used, specify DISP=OLD in the DD statement to overwrite the data set. If, however, the database is on more than a single DASD volume, IBM highly recommends that you delete the data set and redefine it (using the z/OS program IEFBR14 to preallocate the data sets) to ensure that the correct end-of-file marker is placed.
Database Reload Process
The reload process for HD databases can be more complex than the unload process. If the database does not have any secondary indexes and is not involved in a logical relationship, then the database can simply be reloaded.
If there are no secondary indexes or logical relationships involved, the reloading of the database is the same as the unloading: simply reload the database.
If, however, there are secondary indexes or logical relationships involved, you must run additional utility programs before and after the database is reorganized to rebuild the logical relationships and secondary indexes so that they reflect the new physical positions of the segments in the reorganized database. Until all this processing is complete, the logical relationships and secondary indexes are not usable. If you attempt to use them before completing this process, the applications will fail with IMS abend codes indicating that there are invalid IMS pointers.
The following sections discuss each combination of reload processing required:
The reload processing for an HD database without any logical relationships or secondary indexes is shown in Figure 10-2.
Figure 10-2. Simple Database Reload Processing
Keep the following information in mind when you plan for the database reload process:
Reload with Secondary Indexes
Figure 10-3 on page 136 illustrates the reload process when secondary indexes are involved. The process is as follows:
Figure 10-3. Database Reload Processing with Secondary Indexes
Considerations to keep in mind:
Reload with Logical Relationships
Figure 10-4 on page 138 illustrates the reload process when logical relationships are involved. The process is as follows:
Figure 10-4. Database Reload Processing with Logical Relationships
Considerations to keep in mind:
Reload with Logical Relationships and Secondary Indexes
The reload processing for both secondary indexes and logical relationships is a combination of both the individual reload processes described in "Reload Only" on page 134, "Reload with Secondary Indexes" on page 135, and "Reload with Logical Relationships" on page 137.
Figure 10-5 on page 140 illustrates the database reload process when both secondary indexes and logical relationships are involved. The process is as follows:
Figure 10-5. Database Reload Processing with Secondary Indexes and Logical Relationships
Considerations to keep in mind:
Reorganizing Fast Path DEDBs Offline
The process for reorganizing a Fast Path DEDB can be appreciably different from reorganizing full-function databases.
If you are reorganizing a DEDB only to reclaim fragmented free space or get the best placement of the segments for performance (that is, the DBD or data set definitions are not being changed), then you can run the High-Speed DEDB Direct Reorganization utility (DBFUHDR0). This utility can be run while the DEDB is either offline or online. In this instance, offline means that the DEDB is still up and available, but you have prevented applications from accessing it. The DEDB is still technically online.
Related Reading: For more information about the High-Speed DEDB Direct Reorganization utility (DBFUHDR0), see IMS Version 9: Utilities Reference: Database and Transaction Manager.
If you are reorganizing a DEDB to alter the structure, then you need to have your own user-written programs to unload and reload the database data set at the appropriate points, or use the DEDB Unload and Reload utilities from the separately priced IBM IMS High Performance Fast Path Utilities, V2.1 (program number 5655K94). You must also run the DEDB Initialization utility (DBFUMIN0), provided with the IMS base product, immediately before reloading the database. You do not have to run additional utilities after the database is reloaded because DEDBs do not support secondary indexes or logical relationships.
Related Reading: For more information about:
Reorganizing HALDBs Offline
The offline reorganization processes for a HALDB database and other full-function IMS databases are similar: they both consist of an unload and reload of the database; however, the HALDB reorganization process has advantages over the reorganization process of other full-function databases, such as:
An offline reorganization of a HALDB database can be done with one or more parallel processes. These processes unload one or more partitions and reload them. If the database has secondary indexes or logical relationships, additional steps are not required. The amount of time required for a reorganization depends on the sizes of the partitions. Smaller partitions reduce the time. You can reduce your reorganization time by creating more partitions and reorganizing them in parallel.
The basic steps involved in reorganizing a HALDB offline are:
Figure 10-6 on page 143 shows the offline processes used to reorganize a HALDB database with logical relationships and secondary indexes. In this case, the partitions are reorganized by parallel processes. Each partition can be unloaded and reloaded in less time than unloading and reloading the entire database. This is much faster than the process for a non-HALDB full-function database. Additionally, no time is required for updating pointers in the logically related database or rebuilding secondary indexes. This further shortens the process.
Figure 10-6. Offline Reorganization of a HALDB database
Reorganizing HALDB Partitioned Secondary Index Databases
You might need to reorganize your partitioned secondary index (PSINDEX) database. Because the reorganization of HALDBs does not require the recreation of their secondary indexes, a PSINDEX database can become disorganized as entries are added to it over time.
The HD Reorganization Unload utility and the HD Reorganization Reload utility can be used to reorganize PSINDEX databases. The restrictions and recommendations for reorganizing other HALDB databases also apply to PSINDEX databases with one exception: HALDB secondary indexes have no ILDSs. The HD Reorganization Reload utility control statements should not be used with secondary indexes.
The steps for reorganizing a PSINDEX database are the same as those for reorganizing other types of HALDBs offline.
Options for Offline Reorganization of HALDBs
You have several options when reorganizing a HALDB database:
With offline reorganization, the database is unavailable during the reorganization process. With online reorganization, which is available only for HALDB and DEDB databases, the database remains available for updates during the reorganization process.
Related Reading: For complete information about HALDB Online Reorganization, see IMS Version 9: Administration Guide: Database Manager.
Reorganizing Fast Path DEDBs Online
As mentioned in "Reorganizing Fast Path DEDBs Offline" on page 141, if you are only reorganizing to reclaim fragmented free space or to get the best placement of the segments for performance (that is, the DBD or data set definitions are not being changed), or both, you can run the High-Speed DEDB Direct Reorganization utility while the DEDB is online.
Related Reading: For more information about the High-Speed DEDB Direct Reorganization utility, see IMS Version 9: Utilities Reference: Database and Transaction Manager.
Reorganizing HALDBs Online
Prior to IMS Version 9, you had to ensure that HALDB partitions were offline before you could perform database reorganization for them. IMS Version 9 introduced an integrated HALDB Online Reorganization function that allows HALDB partitions to remain online and available for IMS application programs during a database reorganization.
An online reorganization of PHDAM or PHIDAM HALDB partitions runs non-disruptively, allowing concurrent IMS updates, including updates by data sharing IMS systems. The online reorganization is non-disruptive because IMS copies small amounts of data from the partition's original data sets (the input data sets) to separate output data sets. IMS tracks which data has been copied so that IMS applications can automatically retrieve or update data from the correct set of data sets.
When you reorganize either PHDAM or PHIDAM HALDB partitions online, the online reorganization process requires additional data sets to hold the reorganized data. The additional data sets are equal in number to the data sets being reorganized, excluding the ILDS.
In a PHDAM database, online reorganization increases the maximum number of data sets associated with a partition to twenty-one. In a PHIDAM database, which includes a primary index, online reorganization increases the maximum number of data sets associated with a partition to twenty-three. In either case, online reorganization needs only as many new data sets as the data sets that exist in the partition at the time the reorganization process begins.
Each additional data set requires an additional ddname. To distinguish between the ddnames for the data sets that are being reorganized and the ddnames for the data sets into which online reorganization moves the reorganized data, HALDB online reorganization extends the HALDB naming convention to include the suffixes M through V for the ddnames of the primary data sets and the suffix Y for the ddname for the additional primary index. The suffixes M through V and Y correspond in order to the standard HALDB ddname suffixes A through J and X.
The initial load or offline reorganization reload of a HALDB partition always uses the A-through-J (and X) data sets. Until the first time that you reorganize a HALDB partition online, only the A-through-J (and X) data sets are used.
There are three phases of online reorganization for a HALDB partition:
The Initialization Phase for HALDB Online Reorganization
You start the online reorganization of a HALDB partition using the INITIATE OLREORG command. See the IMS Version 9: Command Reference for more information about this command.
During the initialization phase, IMS updates the RECON data sets to establish the ownership of the online reorganization by the IMS subsystem that is performing the online reorganization. This ownership means that no other IMS subsystem can perform a reorganization of the HALDB partition until the current online reorganization is complete or until ownership is transferred to another IMS subsystem. IMS adds the M-through-V (and Y) DBDSs to the RECON data sets if those DBDS records do not already exist. IMS also adds the M-through-V (and Y) DBDSs to any existing change accumulation groups and DBDS groups that include the corresponding A-through-J (and X) DBDSs.
Change accumulation is the process of creating a compacted version of one or more IMS log data sets by eliminating records not related to recovery, and by merging multiple changes to a single segment into a single change. For more information about change accumulation, see "Database Change Accumulation Utility" on page 158.
Before online reorganization begins for a HALDB partition, there is a single set of active data sets for the HALDB partition. These active data sets are the input data sets for the copying phase. There might also be a set of inactive data sets from a prior online reorganization that is not used by IMS application programs.
During the initialization phase, IMS evaluates each of the inactive data sets to ensure that it meets the requirements for output data sets. If any of the output data sets does not exist, IMS creates it automatically during this phase.
At the end of the initialization phase, IMS considers the original active set of data sets as the input set and the inactive data sets as the output set. This use of the input and output sets of data sets is represented by the cursor-active status for the partition, which is recorded in online reorganization records in the RECON data sets. A listing of the partition's database record in the RECON data sets shows OLREORG CURSOR ACTIVE=YES. A listing of the partition also shows that both sets of DBDSs are active; the first set of DBDSs listed is for the input data set, and the second set of DBDSs is for the output data set, for example, DBDS ACTIVE=A-J and M-V. While the partition is in the cursor-active status, both sets of data sets must be available for the partition to be processed by any application.
The Copying Phase for HALDB Online Reorganization
During the copying phase, the HALDB partition is comprised of the A-through-J (and X) data sets and the M-through-V (and Y) data sets. During this phase, both sets of data sets must be available in order for IMS applications to access the partition.
While IMS reorganizes a HALDB partition online, IMS applications can make database updates to the partition. Some of the database updates are made to the input data sets, while others are made to the output data sets, depending on which data is updated by the application. Which data sets are updated is transparent to the application program. Figure 10-7 illustrates the relationship between the input and output data sets at a point during the online reorganization.
Figure 10-7. The Relationship between Input Data Sets and Output Data Sets during the Online Reorganization of a HALDB Partition
Figure 10-7 shows two sets of database data sets for a HALDB partition, the input data sets that have not been reorganized and the output data sets that have been (at least partially) reorganized. The figure shows the reorganization as progressing from left to right, from the input data sets above to the output data sets below. The data sets in the figure are divided into four areas:
As the online reorganization progresses, IMS uses a type of pointer called a cursor to mark the end point of those database records that have already been copied from the input data sets to the output data sets. As the reorganization and copying proceeds, this cursor moves forward through the partition, from left to right in the figure.
When an IMS application program accesses data from a HALDB partition that is being reorganized online, IMS retrieves the data record:
If the data record happens to fall within the unit of reorganization, IMS retries the data access after the records are unlocked. An application program does not receive an error status code for data within a unit of reorganization.
To allow recovery of either an input data set or an output data set, all database changes are logged during the online reorganization, including the database records that are copied from the input data set to the output data sets.
The Termination Phase for HALDB Online Reorganization
The online reorganization of a HALDB partition terminates after the end of the copying phase, or when IMS encounters an error condition during the reorganization. You can also stop the online reorganization of a HALDB partition using the TERMINATE OLREORG command. See IMS Version 9: Command Reference for more information about this command.
After the copying phase is complete for a HALDB partition, the output data sets become the active data sets, and the input data sets become the inactive data sets. The active data sets are used for all data access by IMS application programs. The inactive data sets are not used by application programs, but can be reused for a subsequent online reorganization. Unless you perform an initial load or a batch reorganization reload for the partition, successive online reorganizations for the partition alternate between these two sets of data sets.
IMS updates the partition's database record in the RECON data sets to reset the cursor-active status for the partition to reflect that there is now just one set of data sets. A listing of this record from the RECON data sets shows OLREORG CURSOR ACTIVE=NO and the ACTIVE DBDS field shows the active (newly reorganized) data sets. IMS also updates the online reorganization records in the RECON data sets with the timestamp when the reorganization completed.
If you specified the DEL keyword for the INITIATE OLREORG command (or the UPDATE OLREORG command), IMS deletes the inactive data sets after resetting the cursor-active status for the partition. Before deleting the inactive data sets, IMS notifies all sharing IMS subsystems (including batch jobs) that the online reorganization is complete and is recorded in the RECON data sets. The IMS subsystem that is performing the online reorganization waits until it receives an acknowledgement from each of these sharing subsystems that they have closed and deallocated the now-inactive data sets, and then it deletes these data sets. However, if the acknowledgements are not received within 4.5 minutes, the owning subsystem will attempt to delete the inactive data sets anyway. Without the acknowledgements, the deletion attempt is likely to fail.
Finally, at the end of the termination phase, IMS updates the RECON data sets to reset the ownership of the online reorganization so that no IMS subsystem has ownership. This resetting of ownership means that any IMS subsystem can perform a subsequent reorganization of the HALDB.
If online reorganization of a HALDB partition terminates prior to completion, either because of an error or because you issued the TERMINATE OLREORG command, you need to restart the online reorganization or you need to perform an offline reorganization for the partition.
Figure 10-8 on page 150 shows the normal processing steps of a successful online reorganization of a HALDB partition. The columns represent the flow of control through the phases of the online reorganization, from the user to IMS, and the status of the data sets as the processing events occur.
Figure 10-8. The Normal Processing Steps of HALDB Online Reorganization
Related Reading: For more information about reorganizing HALDBs online, see IMS Version 9: Administration Guide: Database Manager.