Oracle and Split-Mirror Technologies

---

We decided to include this section simply because more and more database administrators are walking away from meetings knowing that split-mirror technologies are about to seriously impact their day-to-day life. And split mirrors are being leveraged, more and more, as sites for backups to occur, so it's worth mentioning split-mirror technology in this RMAN chapter.

The Split-Mirror Configuration

If you haven't been subjected to this yet, let's sum up: split-mirror technologies refers to vendor solutions that use a specific hardware configuration that mirrors all writes to a disk volume. At certain intervals, the mirrors are broken, such that the writes continue to one of the volumes, but the second volume is frozen in time at the moment of the split.

Meanwhile, a third volume is 'resilvered' with the primary volume, so that a mirror exists again. The split mirror sits idle, waiting for a possible failure to occur, in which case the system administrator can failover to the broken mirror, making it the primary volume. After a predetermined amount of time ticks by, the primary and the third volume are split, and the second mirror is 'resilvered' back to the primary. See Figure 8-9 for a look at this solution.

Figure 8-9: Sync and split fail-safes

The benefits of this technology are, essentially, the following:

• The failover to the split-mirror volume, in case of a problem at the primary, is immediate.

• The need for backups is eliminated, as a fallback always exists that is immediately accessible.

• If backups are needed, they can be taken from the split mirror, so there is no hit at the primary volume site.

To use split-mirror technology successfully with an Oracle database, it is required that, prior to the mirror split, every tablespace in the database be placed into hot backup mode first. This is required because, despite what the vendor tells you, the mirror split is not a singular event. It is more like peeling a banana: it starts in one place, and progresses to the end. While fast, it is not occurring at the same time across the entire volume. Therefore, it is possible that a write is occurring to the primary volume database, but the write only occurs at the bottom of the unpeeled secondary volume.

To cope with this problem, the hot backup gives Oracle a safety valve, in the form of 'before image' data blocks in the redo stream. That way, if the split-mirror copy of a datafile is copied back to the primary to perform recovery, any corruption due to the mirror peel will be overwritten during the recovery of the file at the primary.

RMAN Backups from the Split Mirror

With increasing frequency, it is becoming apparent to DBAs with split-mirror investments that an additional layer of protection is required, in the form of RMAN backups of the database. With an idle copy of the database simmering on the back burner of the split mirror, a light bulb appears above the DBAs head: he or she should just mount the split-mirror drive onto a different server, and take the RMAN backup from the split mirror directly to tape (or a different disk volume that can be mounted on the primary).

Great idea! Sounds simple enough, right? Well... there's a few tricky points that need to get worked out first; otherwise, you will have the case of the mysteriously disappearing backups.

Here's the problem: RMAN accesses the controlfile to determine what to back up, and after the backup is complete, it updates the controlfile with the details of the backup. If you are connected to a split-mirror copy of the controlfile, it is this copy that gets updated with the details about the backup. So then, of course, when you go to resilver the split volume with the primary, the controlfile is overwritten with the data in the primary controlfile, and the backup data is lost forever.

The solution, you figure, is to use a recovery catalog when you back up at the split mirror. A sound, logical decision: after the backup is complete, the split volume controlfile is updated with the backup records, which are then synchronized to the catalog. Then it's simply a matter of syncing the catalog with the primary volume so that the backups can be used. Too cool!

So let's say you do this: you back up from the secondary volume, you sync the backup records to your recovery catalog, and then you connect RMAN to the primary volume database and to the catalog. You perform a resync. And this is where things get really, really weird. Sometimes, when you try to perform an operation, you get the error

RMAN-20035: invalid high recid

Other times, things work just fine it seems, but the backups you took at the split-mirror database have disappeared from the recovery catalog.

The problem, now, has become the internal mechanism of how RMAN handles record building in the controlfile and the recovery catalog. Every record that is generated gets a record ID (RECID), which is generated at the controlfile. When the backup occurs at the split-mirror DB, the controlfile gets its high RECID value updated, and this info gets passed to the catalog. But the recid at the primary database controlfile has not been updated, necessarily. So when you connect to the catalog and the primary DB, if the catalog's high recid is higher than the one in the controlfile, you get the 'invalid high recid' error. If the recid in the catalog is lower than the recid of the primary database controlfile, RMAN initiates an update of the catalog that effectively eliminates all the records since the last sync operation with the primary cfile. Poof! Backup records from the split volume are gone.

The solution to this problem is to set the controlfile at the split mirror to become a backup controlfile. If RMAN detects that it is backing up from a noncurrent controlfile (backup or standby), it does not increment the recid in the catalog, so that the records are available after a resync with the current controlfile at the primary database.

HA Workshop: Configure RMAN to Back Up from the Split Mirror

Workshop Notes

This workshop assumes that you put all the tablespaces into hot backup mode (a requirement) during the period of the split. After the split, you connect the split volume to a new server that has Oracle Database 10g installed, and you now want to take an RMAN backup. Because RMAN will give an error if files are in backup mode, you will need to manually end backup for every file, as shown. It's best to write a script for this. This workshop also assumes that you split the archivelog destination and bring it across to the clone at the same time for archivelog backup.

Step 1.  Mount the database on the clone server, and prepare the controlfile for RMAN backup.

startup mount; alter database datafile 1 end backup; --- you will need to do this for every file recover database using backup controlfile until cancel; cancel exit

Step 2.  Connect RMAN to the clone instance (as the target), and to the recovery catalog, and run the datafile backup.

rman target / rman> connect catalog rman/password@rman_cat_db rman> backup database plus archivelog not backed up two times;

Step 3.  Connect RMAN to the production database (as the target) and the catalog, and perform a sync operation and archivelog cleanup, and then back up the controlfile.

rman target / rman> connect catalog rman/password@rman_cat_db rman> resync catalog; rman> delete archivelog completed before sysdate -7; rman> backup controlfile;

Use DG Instead

The bottom line, from our perspective, is that if you can interject yourself into the conversation about hardware configuration before the decision is made, you should encourage your sys admins to look into a Data Guard configuration instead of sync and split hardware modeling. If the entire purpose of the sync and split is for the hardiness of the Oracle database, you probably would save money and add other functionality by switching to a DG model instead. A combined Data Guard, RMAN, and Flashback Database configuration will provide the same functionality as split-mirror technology, without the proprietary hardware costs. You still get disaster recovery, off-loaded backup I/O, and a fallback database position that is quickly implemented. But you can lay it all down on commodity-priced servers.

---