Section 12.3. Implementation Example | Backup & Recovery: Inexpensive Backup Solutions for Open Systems

12.3. Implementation Example

The system administrator noticed that system logs are showing indications of hardware errors and filesystem corruption in the root volume group. Further investigation included the use of system hardware diagnostics to isolate the failure to the boot disk hardware. Now it's time for the bare-metal recovery plan to be enforced to get the system back into fully operational use. The following example illustrates the use of make_net_recovery. (The make_tape_recovery steps would be similar.)

Before the point of failure, the system administrator created a recovery archive by taking these steps:

The administrator installed the Ignite-UX software onto a network server from the HP web site at http://software.hp.com or from the HP-UX Applications media.
Following installation, the administrator ran the /opt/ignite/bin/ignite command to launch the GUI/TUI interface and configure the Ignite-UX server.
Still in the Ignite-UX server interface, the system administrator entered the name of the client machine being archived. Once Ignite-UX connected to the client, it collected needed information and displayed an icon for that client.
The administrator right-clicked on the client icon and selected Create Network Recovery Archive. Ignite-UX displayed a sequence of screens with various instructions and tips.
Ignite-UX displayed the recovery archive defaults, and the administrator changed a few options, such as the maximum number of retained archives and an archive description. The system administrator verified that the destination directory had sufficient filesystem space to store the archive. The default destination directory name on the server was var/opt/ignite/recovery/archives/<client_name>.

By default, Ignite-UX archives only the root volume group. Optionally, other disks and volume groups may be included in an archive, but it is a good practice to archive only the root volume group and to ensure that it contains only what is necessary to boot and run the OS.

After selecting the option to Finish, the archive creation began.
The icon color reflected success, errors, or warnings in the archive process. (Green indicates success while red and yellow indicate errors and warnings, respectively. To monitor progress of the archive creation in real time, right-click the client icon and select Client Status.)
The system administrator invoked subsequent make_net_recovery archive operations either from the Ignite-UX interface or from the command line of the client. (Steps 1 through 7 are required only for the first archive creation. )

After the disaster occurred, and the point of failure was identified, the replacement disk hardware was ordered and received. The system administrator started recovery operations using the following steps:

The administrator shut down and powered off the client system, and replaced the failed boot disk.
Since the new disk had no operating system, the administrator powered on the client and brought it to the firmware prompt, initiating remote boot operations to pull an install kernel from the Ignite-UX server.
Using the Ignite-UX TUI on the client console, the administrator selected an archive for recovery. (Archives are labeled based on the description provided when the archive was created. The default naming convention indicates the creation date and time, such as Recovery Archive 2006-02-04,07:43).
The system administrator poured a cup of coffee and monitored the recovery until it completed, which was about 45 minutes later.

12.3.1. Command-Line Examples

The following make_net_recovery examples assume that the defaults file was automatically generated through the "Add New Client for Recovery" option, with a client name of hal. The hostname of the Ignite-UX server is ibanez.

ibanez: /var/opt/ignite/clients/hal/recovery # cat defaults # # This file is used to specify defaults for make_net_recovery. # It is overwritten every time the Recovery UI is run, # even if an archive is not created.  It should not be used # as a reliable source of information about any given archive. # RECOVERY_TYPE=net RECOVERY_LOCATION=ibanez:/var/opt/ignite/recovery/archives/hal TAPE_DESTINATION=none RECOVERY_DESCRIPTION=Recovery Archive SAVE_NUM_ARCHIVES=2 ARCHIVE_TYPE=tar

Since the Ignite-UX server already has an archive configuration, the following command may be run on the client's command line (as opposed to being initiated through the Ignite-UX server interface) to create a network recovery archive that uses the existing client configuration on the ibanez server.

# make_net_recovery -s ibanez

A cron job can be defined to launch this command automatically at desired intervals.

The following command creates a local tape recovery archive of the system on which it's run, using settings from the last run of this command on this machine:

# make_tape_recovery -s ibanez

Suppose that the /var filesystem on the Ignite-UX server is close to capacity and that there is insufficient disk space to store a new archive. The following command overrides the existing recovery_location parameter, assuming that the new archive location has been manually created and properly exported via NFS.

# make_net_recovery -s ibanez -a ibanez:/depot/recovery/archives/hal

These three examples each create a minimal recovery archive, containing only the directories and files that Ignite-UX considers essential for a functional system as defined by the mnr_essentials file.

Typically, symbolic links are not followed during the creation of an archive; only the link is backed up. However, if a link is listed in mnr_essentials, the link is followed to the file it links to, and that file is backed up.

The following command examples use the -x inc_entire= option to create an archive that contains the entire root volume group (vg00 in this example), including as part of the root volume group any directories and files listed in mnr_essentials, no matter what volume group they reside in:

# make_net_recovery -x inc_entire=vg00 -s ibanez # make_tape_recovery -x inc_entire=vg00 # make_tape_recovery -x inc_entire=vg00 -s ibanez

The A option of the make_tape_recovery and make_net_recovery commands is quite different from the A option to the obsolete make_recovery command. A now means that if any additionally included file or directory resides outside of the root volume group, disk group, or disk, that entire volume group, disk group, or disk is included in the archive.

12.3.2. Verifying Archive Contents

You should test system recovery archives before relying on them to restore a production system. If a system recovery archive cannot be restored, it is better to discover this before the failure of a production system. The only foolproof method of verifying a recovery image is by performing a full recovery of that image back onto the system where it originated. Here are some ways to test that Ignite-UX is doing what it's supposed to:

Use preview mode to determine whether an archive of the currently running system will succeed.

You can specify the p argument to make_net_recovery or make_tape_recovery to preview the processing that takes place without actually creating the archive. The following example was run from a client system, specifying archive preview mode on ibanez.

# make_net_recovery -s ibanez -p

Inspect the flist file on the Ignite-UX server to view the list of files that would be part of an actual archive. If make_net_recovery preview-mode sessions are run in succession before actually creating an archive, Ignite-UX creates new archive recovery directories, each using a date and timestamp, with the most recent directory having a link created to it that is called latest.

# view /var/opt/ignite/clients/hal/recovery/2006-06-22,12:01/flist

After running make_net_recovery in preview mode, subsequent execution using the -r option continues with the creation of an archive, using the configuration information in the directory that is pointed to by the latest link.

If you run the make_tape_recovery command in preview mode, review the flist file on the client at the following location. Successive make_tape_recovery preview sessions followed by one with the -r option operate in a manner similar to that of make_net_recovery.

# view /var/opt/ignite/recovery/latest/flist

Review the list of directories and files in the recovery archive.

The flist file is generated in preview mode when the archive is actually created. Read flist in a text editor.

Determine whether you can read the archives.

The following commands read the tar index information from the specified recovery archive stored on the disk of an Ignite-UX server and write that information to the specified output file. tar is the default archive format (but you can use cpio instead). The pax command can read either format. The following examples use tar and pax to access one of two make_net_recovery archives stored under unique directory names that are based on the creation date and time of each archive:

# cd /var/opt/ignite/recovery/archives/hal # file 2006* 2006-01-24,19:51:       gzip compressed 2006-01-31,17:06:       gzip compressed # gunzip -c 2006-01-31,17:06 | tar -tvf - > /tmp/file.list # gunzip -c 2006-01-31,17:06 | pax -vf - > /tmp/file.list

You can use a similar approach for tapes produced using make_tape_recovery on HP9000 systems. This example reads from a tape loaded into the specified drive (note the use of a no-rewind device file), seeking past the boot components at the beginning of the tape to access the archive. pax reads the tar index information from the recovery archive and writes it to the specified output file.

# mt -r /dev/rmt/0m rew # mt -t /dev/rmt/0mn fsf 1 # pax -vf /dev/rmt/0m > /tmp/file.list

Compare the contents of an existing recovery archive against the system on which it was created using either the check_net_recovery or check_tape_recovery command.

If the archive is stored on a remote server, run the following command on the client system to be verified, specifying the hostname of the Ignite-UX server that contains the network recovery archive. The output of this command reports whether any files in the archive have been added, deleted, or changed since the archive was created.

# check_net_recovery -s ibanez

Verify that the client system will boot from the Ignite-UX server containing its make_net_recovery archive (or from the recovery tape that was created with make_tape_recovery).

For the latest information on recovery tape operations for HP Integrity systems, refer to documentation found online at http://docs.hp.com. Use a current version of Ignite-UX to make full use of commands such as make_ipf_tape that enable new tape functionality on HP Integrity systems. Support for native UEFI 2.0 bootable tapes starts with Ignite-UX version C.6.8. (This requires firmware updates on most HP Integrity servers to enable native tape boot.)

For HP Integrity systems, verify that the attached tape drive and host bus adapter (HBA) supports direct boot capabilities for tapes created in bootable UEFI 2.0 format. For tapes that are formatted for booting on an HP9000 system, use a dual-media boot process on HP Integrity systems. You first boot from a DVD (such as the OS install media) and then select the Logical Interchange Format (LIF) formatted tape for booting. All recovery tapes created before Ignite-UX version C.6.3 are HP9000 format tapes. Recovery tapes created using Ignite-UX versions C.6.3 to C.6.7 are not UEFI 2.0-compliant; with these you must boot from DVD first, then tape.

After verifying that a client system may be booted for archive recovery, use the Ignite-UX itool interface on the client console to review the configuration that would be installed were the recovery operation to actually be initiated:

                                    itool (  )                                                                                                                        /-------\/----------\/--------\/-------------\/----------\                       | Basic || Software || System || File System || Advanced |                       |       \-------------------------------------------------------------------\    |                                                                           |    | Configurations:  [ HP-UX B.11.00 Default    ->] [ Description...  ]       |    |                                                                           |    | Environments:    [                            ->] (HP-UX B.11.00)         |    |                                                                           |    | [ Root Disk...  ] SEAGATE_ST32151N, 8/16/5.5.0, 2048 MB                   |    |                                                                           |    | File System:     [ Logical Volume Manager (LVM) with VxFS  ->]            |    |                                                                           |    | [ Root Swap (MB)... ] 1024      Physical Memory (RAM) =  2560 MB          |    |                                                                           |    | [  Languages...   ]                   [ Keyboards...  ] [ Additional...  ]|    |                                                                           |    \---------------------------------------------------------------------------/         [ Show Summary...  ]                          [ Reset Configuration ]      ------------------------------------------------------------------------------    [  Go!   ]                       [ Cancel ]                       [  Help  ]

12.3.3. Troubleshooting Recovery Operations

Generally speaking, initial troubleshooting includes the following questions.

Has the Ignite-UX server in question been used successfully in other archive creation and recovery operations? If so, what is different about the current operation? If not, has the Ignite-UX server been properly configured?
Are the Ignite-UX client and server systems on the same network subnet? If not, has an Ignite-UX boot helper server been configured?
Was this particular recovery archive created on the same physical system now being recovered? Has this archive ever been successfully used in a recovery operation?
What specific error messages have been observed? What details are found in related logfiles?

When a recovery operation fails, look for a message on the client console that explains the nature of the problem, along with a prompt to work from a debug shell. For example:

ERROR:   The ifconfig(1M) command (ifconfig lan2 inet 15.43.233.125 netm ask           255.255.248.0 up) failed to enable the lan2 interface (return value:           1).  Check that the system's networking hardware is functioning and           properly connected. ERROR:   Could not start networking.          The configuration process has incurred an error, would you like          to push a shell for debugging purposes? (y/[n]): y        * Type "exit" to reboot, "exit 2" to ignore error To continue the recovery after working from the debug shell, enter exit 2.

In another example, after an archive recovery operation was initiated, the network-based Ignite-UX server monitored the client's install.log at defined intervals, eventually reporting the following warning.

WARNING: Client hal appears to be hung.  Since the server started          checking, the client's logfile has not been updated in 20 minutes.          The timeout in effect is 20 minutes.

After seeing the warning, review the client's install.log to determine the point in the recovery process where a failure may have occurred. In this example, the client name is hal, and the client's install log would be at /var/opt/ignite/clients/hal/install.log on the Ignite-UX server. On the client itself, from the debug shell, you can view the install log at /var/opt/ignite/local/install.log:

* Loading_software:  Begin        * Installing boot area on disk.        * Enabling swap areas.        * Backing up LVM configuration for "vg00".        * Processing the archive source (Recovery Archive).        * Fri Sep 20 15:08:07 EDT 2002: Starting archive load of the source          (Recovery Archive). ERROR:   Cannot load OS archive (Recovery Archive)          The configuration process has incurred an error, would you like          to push a shell for debugging purposes? (y/[n]):        *

This error could have a number of possible causes, including a physical network connection that is failing or otherwise performing poorly. In the case of poor performance, determine whether a network switch is being used between the Ignite-UX client and server. While booted to an Ignite-UX install kernel, if the client's network interface fails to auto-negotiate the network speed and duplex settings with the switch, the default is 100/half-duplexregardless of the settings on the switch. This results in an extremely slow recovery archive installation.

If auto-negotiation is not used on the switch being used for the recovery operation, and a mismatch in network speed and duplex settings occurs, the default settings used by Ignite-UX may be modified in the installation filesystem (INSTALLFS) located on either the Ignite-UX server or boot helper server being accessed by the client system for remote booting. For this change, the _hp_lanadmin_args parameter is modified within the Ignite-UX INSTALLFS through the use of the instl_adm command as described here. For reference, the Ignite-UX configuration file syntax is discussed in the instl_adm(4) manpage.

First, read the current Ignite-UX configuration into a temporary file using the following command:

# instl_adm -d >/tmp/installfs.cfg

Next, as detailed in the instl_adm(4) manpage, set _hp_lanadmin_args equal to -X 100FD in the temporary file /tmp/installfs.cfg.

Finally, write the new Ignite-UX configuration to INSTALLFS.

# instl_adm -f /tmp/installfs.cfg