4.11 Recovery

---

Oracle performs recovery operations through a two-phased approach. Under this method of recovery Oracle passes through the required file twice to complete the recovery operation. This feature speeds up the recovery process while making the system available to the users more quickly.

After detection, and during the remastering of GCS of the failed instance and cache recovery, most work in the surviving instance is paused and, while transaction recovery takes place, work occurs at a slower pace. This point is considered full database availability, as now all data is accessible, including that which resided on the failed instance. The application is now responsible for reconnecting the users and repeating any uncommitted work they have done.

As part of the failover process when an instance crashes and the processes fails over to the other surviving nodes, the GCS resources that were previously mastered at the failed instance are redistributed across the surviving instances through the process of resource remastering. Once this is completed, the resources are reconstructed at their new master instances. While resources from the failed instance are distributed among the surviving nodes, all other resources previously mastered at surviving instances are not affected.

For any lock request, there is a 1/n chance that the request will be satisfied locally and an (n - 1)/n chance that the lock request involves a remote operation. In the case of one surviving instance, all lock operations are satisfied locally.^[1]

On completion of the remastering of resources from the failed instance to the surviving instances, Oracle performs a clean-up operation to remove in-progress transactions of the failed instance. This is called instance recovery (IR).

The active instance that first identifies a member in the cluster that is not responding and deduces its failure is responsible for the recovery operation. The active instance that deduced the failure through its LMON process controls the recovery operation by taking over the redo logs files (in a shared disk subsystem the redo logs are visible by all instances participating in the cluster) of the failed instance.

Based on the new method of recovery in two passes, the recovery operation is divided into cache recovery and transaction recovery. Apart from these two modes of recovery there is another method that is unique to a RAC implementation called online block recovery.

4.11.1 Cache recovery

Cache recovery is the first pass of reading through the redo logs by SMON on the active instance. The redo logs files are read and applied to the active instance performing the recovery operation through a parallel execution.

During this process, SMON will merge the redo thread ordered by the SCN to ensure that changes are applied in an orderly manner. It will also find the BWR in the redo stream and remove entries that are no longer needed for recovery because they were PIs of blocks already written to disk. SMON recovers blocks found during this first pass and acquires the required locks needed for this operation. The final product of the first pass log read is a recovery set that only contains blocks modified by the failed instance, with no subsequent BWR to indicate that the blocks were later written. The recovering SMON process will then inform each lock element's master node for each block in the recovery list that it will be taking ownership of the block and lock for recovery. Other instances will not be able to acquire these locks until the recovery operation is completed.

This two-phased approach to recovery provides full database access once Oracle has completed replaying of redo log files. After the completion of the first phase of the recovery operation, cache recovery, the second phase, transaction recovery, begins.

4.11.2 Transaction recovery

Compared to the cache recovery scenario where the recovery is of a forward nature, i.e., rolling forward of the transactions from the redo logs, the transaction recovery scenario handles uncommitted transactions, hence rolling them back. Also during this pass, the redo threads for the failed instances are merged by SCN and the redo is applied to the data files.

During this process of rolling back uncommitted transactions, Oracle uses a technology called fast-start recovery where it performs the transaction recovery as a deferred process, as a background activity. Under this feature Oracle uses a multiversion and consistency method to provide on-demand rollback of only those rows blocked by expired transactions. This feature helps new transactions by not requiring them to wait for the rollback activity to complete. Fast-start recovery can be of two kinds, fast-start on demand and fast-start parallel rollback.

Fast-start on demand

Under this option, users are allowed to perform regular business not being interfered with by the uncommitted or expired transactions from the other instance.

Fast-start parallel rollback

Fast-start parallel rollback is performed by SMON, which acts as a coordinator and rolls back transactions using parallel processing across multiple server processes. The parallel execution option is useful where transactions run for a longer duration before committing. Under these circumstances the SMON process automatically decides to perform a parallel rollback operation. When using this feature each node spawns a recovery coordinator and recovery process to assist with parallel rollback operations.

Fast-start parallel rollback features is enabled by setting the parameter FAST_START_PARALLEL_ROLLBACK to either low or high. This setting indicates the number of processes to be involved for performing parallel rollback. It should be either twice or four times the CPU_COUNT parameter, for the low and high values, respectively.

Examining some of the data dictionary views could help in the monitoring of the fast-start rollback feature:

• V$FAST_START_SERVERS

• V$FAST_START_TRANSACTIONS

Note 

More details about the parallel recovery process will be covered in Chapter 6 (Parallel Processing) and more details regarding these and other data dictionary views will be covered in Chapter 9 (Parameters and Views).

4.11.3 Online block recovery

Online block recovery is another kind of recovery that is unique to the RAC implementation. Online block recovery occurs when a data buffer becomes corrupt in an instance's cache. Block recovery will occur if either a foreground process dies while applying changes or if an error is generated during redo application. If the block recovery is to be performed as a result of the foreground process dying, then PMON initiates online block recovery. However, if this is not the case, then the foreground process attempts to make an online recovery of the block.

Under normal circumstances this involves finding the block's predecessor and applying redo records to this predecessor from the online logs of the local instance. However, under the cache fusion architecture, copies of blocks are available in the cache of other instances and, therefore, the predecessor is the most recent PI for that buffer that exists in the cache of another instance. If, under certain circumstances, there is no PI for the corrupted buffer, the block image from the disk data is used as the predecessor image before changes from the online redo logs are used.

^[1]Oracle 9i Real Application Cluster Concepts Documentation.

---