9.7 Backuprecovery

---

9.7 Backup/recovery

When running Lotus Domino, like any application on zSeries, it is necessary to have a well-defined backup strategy in place to ensure data recovery. This section lists some options for Domino. We do not document detailed implementation instructions for these options, but we present a high level point of view, listing advantages and disadvantages, as appropriate.

9.7.1 Understanding your backup and recovery needs

There are two basic categories of data in a Domino environment:

• Files, including system configuration data and the Domino executables, notes.ini, and ID files

• Domino databases and templates

We make this distinction because several methods and tools can be used to back up the files and Domino databases, including Linux archive commands such as tar and the TSM Backup-archive client. But these methods cannot be safely used to back up the Domino databases while the Domino server is active.

In order to take a good clean backup of Domino databases while the server is up, you need a utility which uses the Domino backup and recovery APIs. Although there are several products available for other platforms, at the time of writing, the only tool available for Linux on zSeries was the IBM Tivoli Storage Manager for Mail: Data Protection for Lotus Domino. We worked with a beta version of the Linux on zSeries client.

In regards to Domino data, there are three distinct recovery situations.

1. Full recovery: Common disaster recovery situations, like failure of a disk subsystem. All databases contained on the subsystem must be restored.

2. Database level recovery: A single database may have been corrupted. Recover by restoring the last backup of the database.

3. Document level recovery: One or more documents in a database have been lost, either deleted or corrupted. Normal recovery in this situation involves restoring the backup of the database to a temporary location and then copying the required documents to the active version of the database.

   Or you may be required to do a point-in-time restore (forward recovery from a full backup with application of transaction logs) of the database using the backup and recovery APIs in Domino.

The data on the Domino server can be backed up with different backup techniques. We review the following:

• The Backup-archive client of Tivoli Storage Manager (TSM)

• IBM Tivoli Storage Manager for Mail: Data Protection for Lotus Domino for Linux (formerly called TDP)

9.7.2 Tivoli Storage Manager (TSM)

Tivoli Storage Manager (TSM) is a client/server solution which provides backup, archive, and space management services to clients. Its functionality includes:

• Administrative clients that allow the administrator to control server activities, define storage management policies for workstation files, and set up schedules to provide automated backup and archive services at regular intervals.

• Support for backup archive clients that allow users to restore or retrieve files from a TSM server.

• Support for backing up and archiving files on a variety of devices and broad cross-platform and storage device support.

• Extensive and flexible centralized management that helps improve administrator productivity by enabling an administrator to proactively manage, operate on, and view many TSM servers from a single point.

These capabilities allow TSM to scale up to manage, administer, and automate data backup and restore. The base TSM provides two important functions:

1. Backup/Restore: IBM Tivoli Storage Manager includes multiple techniques to reduce data transfer sizes to their minimums to make backups and restores as fast as possible.

2. Archive/Retrieve: Tivoli Storage Manager can move data to offline storage to archive it and to free online disk space for more important active data. When you need that data again, IBM Tivoli Storage Manager can retrieve it for you.

Both the TSM Backup-archive client and the Tivoli Storage Manager Server Version 5.2 are available for Linux for zSeries systems. The server part of the solution (Tivoli Storage Manager server) can also run on other platforms such as AIX, z/OS, Windows, and other. You can get the latest information about the supported client and server platforms from:

• http://www-3.ibm.com/software/tivoli/products/storage-mgr/platforms.html

At the time of writing, the latest release of TSM Backup-archive client available for Linux for zSeries was Version 5.2. In general, the TSM client version you use can be earlier or later than the TSM server version that you use. But you need to review the README that comes with the TSM client to make sure you will have a compatible and interoperable client with your version of the TSM server.

TSM Backup-archive client

The Backup-archive client has the ability to back up any online files, including Domino executables, databases and templates, notes.ini file, ID files, and Linux symbolic links.

There are two types of backup, incremental and selective:

• Incremental backs up files, directories, or subdirectories that are new or have changed since the last incremental backup.

• Selective backs up specific files or entire directories unconditionally.

To avoid fuzzy backups, specific backup options are set for the Backup-archive client in TSM's policies:

• Static: if being modified, TSM will not back up the file or directory.

• Shared static: if being modified, TSM will not back up the file or directory, but will retry the backup a predetermined number of times.

• Shared dynamic: if being modified, TSM will not initially back up the file or directory. It will retry the operation a predetermined number of times, and it will back up the file on its last attempt, even if it is being modified.

• Dynamic: TSM will always back up the file or directory, whether or not it is being modified.

For more information about TSM and the Backup-archive client, refer to IBM Tivoli Storage Management Concepts, SG24-4877.

Once the TSM client is installed, work with your TSM server administrator to create a Node and policy for your TSM client as well as to create the configuration files dsm.sys and dsm.opt. These contain details about what will be backed up and where to located the TSM server. They need to be placed in the /opt/tivoli/tsm/client/ba/bin directory. Example 9-18 shows a sample dsm.sys file.

Example 9-18: Sample dsm.sys file for TSM backup/archive client

    TSM Client Configuration dsm.sys1    SErvername TSM_SERVER    COMMmethod TCPip    ERRORLOGName /var/log/tsmerror.log    ERRORLOGRetention 14D    MANAGEDServices webclient schedule    * NODename lnx19    PASSWORDAccess generate    SCHEDLOGName /var/log/tsmsched.log    SCHEDLOGRetention 14D    SCHEDMODe POlling    TCPPort 1500    TCPServeraddress tsm.itso.company.com    VIRTUALMountpoint /    exclude /var/log/tsm*.log    Example xx TSM Client Configuration dsm.opt    DATEformat 12    SUbdir Yes 

Review your TSM Client documentation (or launch the TSM client interactively and type help) to get an explanation of the options.

Starting the TSM client

TSM can back up your system interactively by executing the command: dsmc i.

The session will be displayed to your screen as the backup is processed, with a summary of the session upon completion. Alternatively, by starting the scheduler daemon (the most common way to use TSM), you can let the TSM server determine when the system is backed up. The simplest way of using the scheduler is by putting the following in your /etc/rc.d/rc.local script:

    /opt/tivoli/tsm/client/ba/bin/dsmcad 

The rc.local script is called when the system is booted. This line will ensure that the TSM scheduler client is started at that time. (This daemon automatically releases from the terminal, so there is no need to run this process in the background.) If the daemon is not running, you can start it by entering dsmc i while logged in as root.

Tip 

Try to keep your dsm.opt and dsm.sys files as generic as possible. This will enable you to distribute the same files to all your virtual Linux systems. When you have shared read-only filesystems, place the option files on that shared filesystem.

9.7.3 Data Protection for Lotus Domino

Data Protection for Lotus Domino is an application client that enables online backups and restores, as well as activation of Domino server database and transaction logs. IBM TSM for Mail: Data Protection for Lotus Domino is designed to use the backup and recovery API in Lotus Domino to provide online backup and restore capabilities using a Tivoli Storage Manager server. Unlike the TSM Backup-archive client, Data Protection always guarantees a consistent backup, since access to the data is through the Domino backup and recovery API.

Data Protection for Lotus Domino communicates with a Tivoli Storage Manager Server using the Tivoli Storage Manager API and it communicates with a Domino server using the Lotus Domino API.

Data Protection for Lotus Domino manages the Domino Server data and makes it easy to perform the following actions:

• Backup online Domino databases.

• Maintain multiple versions of Domino databases.

• Archive Domino transaction log files when archival logging is in effect.

• Restore backup versions of a Domino database and apply changes made since the backup from the transaction log.

• Restore Domino databases to a specific point in time.

• Expire database backups automatically based on version limit and retention period.

• Expire archived transaction log files when no longer needed.

• Automate scheduled backups.

• Data Protection for Lotus Domino provides two types of database backup (incremental and selective) and a log archive function.

• Incremental backup provides a full online backup of Domino databases when necessary. The specific conditions that determine when a new backup is necessary vary, depending on whether the database is logged or not.

• Selective backup unconditionally backs up the specified databases, unless they are excluded from backup through exclude statements.

• When archival logging is in effect, changes to logged databases can be captured in between full backups, by archiving the transaction log.

Even if you choose not to enable transaction logging, Data Protection for Lotus Domino can still be used to selectively and incrementally back up Domino databases and templates. It provides the only way to guarantee a consistent backup on an online Domino database, because it can access the pre-image data buffers through the Domino backup and recovery API. However, point-in-time recovery is not available without archive transaction logging enabled.

Note 

Even when Data Protection for Lotus Domino is implemented, the TSM Backup-archive client is still needed for backing up the Domino executables, notes.ini file, and ID files.

Data Protection for Lotus Domino backs up only Domino databases and templates, including databases which are symbolically linked to the Domino data path by directory or database links.

Restoring databases

There are two methods used to restore databases when transaction logs need to be applied. The database can be restored back to the same server it was on, usually into an alternate directory or with a different name. Or it can be restored to an alternate server specifically configured for the purpose of restoring databases.

Restoring files to the original server

The option of restoring a file to the original server is the easiest method to implement, since no additional work is needed to set up this option. However, it is the method that has the most effect on performance, both for the server and for the restore process.

The process of restoring a database to the original server simply requires that the Data Protection for Lotus Domino restore command be run, followed by the activatedbs command. If needed, the archived transaction log extent files are restored to the active transaction log directory and replayed against the database.

There are two problems with this method. First, the transaction logging process performs best when the head is moving sequentially across the disk surface and not moving unnecessarily. Since the archived extent files must be written to the active directory, the head needs to be repositioned to an empty portion of the disk and the file written there. The log file must then be replayed to find all transactions that need to be applied to the database. In a restore, it may be necessary to restore and replay a large number of extent files.

Second, the transaction logging process will give absolute preference to writing new transactions to disk. The process of replaying the restored logs needs to wait for periods of lower activity to do its work. This results in long restore times.

Domino 6 introduced a new notes.ini parameter, TRANSLOG_RECOVER_PATH. It allows the path to which the archived transaction log extent files are restored to be specified. Since the restored extents no longer need to be restored to the active transaction log directory, this reduces disk I/O contention on the drive that contains the active transaction log. It allows the database to be restored more quickly if transaction logs need to be applied.

However, it can have an effect on overall server performance. Because the restore operation completes more quickly, it may also use more CPU in a concentrated period of time.

To use the alternate path for restoring transaction logging extent files, add the following line to the notes.ini of the Domino server:

    TRANSLOG_RECOVER_PATH=<path to restore extents to> 

Restoring files to an alternate server

The alternate method employed to restore databases is to set up a server, or rather a server shell structure, for restores. This server is never started as a Domino server; instead, it has directory tree structures mirroring those of the production server. It must be at the same Domino release level as the source server and must have a copy of the server.id and notes.ini files from the source server or servers.

Additionally, when a restore is requested, the most current transaction log extent files should be archived and restored to the restore server's transaction log directory. There is a twelve-step process, described in Appendix B of the Tivoli Data Protection for Lotus Domino manual, that needs to be followed to set up the restore server for restoring a database.

Most administrators use a series of scripts to set up the alternate server restore environment and process the restore. While the task of setting up this environment is a bit difficult, it offers the benefit of faster restores with no effect on the production Domino servers.

Since the archived extent files are restored to the transaction log directory of the restore server for replay, there is no effect on the production environment. This is the recommended method for restoring a database. Instructions for restoring to an alternate server or partition are contained in IBM Tivoli Storage Manager for Mail: Data Protection for Lotus Domino for UNIX and OS/400® Installation and User's Guide, SC32-9056.

9.7.4 Transaction logging

In 8.4, "Transaction logging" on page 198, we discuss how to enable transaction logging and the benefits of using it. There are additional benefits, with regard to backup and restore of Domino databases, offered by archive transaction logging:

• Reduced backup times: Backing up terabytes of data is a very time-consuming process, typically taking many hours to complete. When archived-style transaction logging is used, the amount of time needed to do daily backups of databases can be reduced by as much as 80 percent. This savings is realized because only the transactions which have been applied to databases are backed up on a daily basis, rather than the entire contents of the databases.

  The exact opposite is true for restores. The longer the interval between full backups, the longer the time to restore a database, since more logs must be processed. You should consider not only the backup times but also the restore times to determine the right balance of log versus full backups.

  Typically, a base full backup of all databases is taken once a week. The used transaction log extent files are backed up on at least a daily basis. On a heavily used Domino server, the used extents should be backed up several times a day. (Even on a heavily used server, the amount of data in the transactions in the log is only a small percentage of the total size of the data in the database.) All transactions against all databases in Domino 6 are recorded in the transaction log.

  To restore a database to a point in time, you need the full backup and the transaction log extents containing transactions executed against the database. Backups are simplified since only a weekly full backup of all databases, and frequent backup of the relatively small used transaction log extent files, are needed to back up the databases, rather than full database backups on a daily basis.

• More efficient handling of disk I/O: Transaction logging may improve performance and save I/O processing time because it allows Domino to defer database updates to disk during periods of high server activity. Transactions are committed sequentially to the logfiles, typically within about 1/100 of a second of the completion of a transaction, rather than being committed to the database on disk.

  This is much quicker than database updates to random, nonsequential parts of a disk. Since writes to disk are deferred, they can be consolidated, and thus more work can be done with fewer operations. Since the transactions are already recorded, Domino can safely defer database updates until a period of low server activity.

Transaction logging and the Domino 6 lock manager

Domino 6 replaces the semaphore method of locking a database used in Release 5 with a new lock manager. It provides for greater granularity by allowing locking at the element level, which allows for greater concurrent access and updating of the database by multiple users.

One of the most obvious effects of this is seen when databases are transactionally logged and the number of transactions increases substantially. Under R5, as a server became increasingly busy and the number of transactions increased, the response time would fall off in a direct proportion to the increase in transactions. This was due to an increase in the frequency of the flushing of changes from the universal buffer manager (UBM) to the on-disk copy of the database. During this time, the database was locked, which directly affected response time to that database.

With the implementation of object-level locking and improvements in the flushing algorithm, users are able to access the database even during the flushing operation. The net result is improved performance.

To take full advantage of the object-level locking capabilities of the new lock manager, the databases must be in Domino 6 (ODS 43) format and transaction logging must be enabled for the database. Domino 6 databases that are not transactionally logged, as well as release 5 format databases, will still use the new lock manager. However, they will not benefit from its full capabilities and will perform at a level similar to Domino release 5.

9.7.5 Data Protection for Lotus Domino setup

We tested a beta version of the new IBM Tivoli Storage Manager for Mail: Data Protection for Lotus Domino for Linux Version 5 Release 2 Level 1. This is the first version of Data Protection for Lotus Domino to support Linux.

In this section, we describe some of the configuration files and processes we used during our testing and present them as examples; however, this section does not contain a complete description of everything needed for installation and use of Data Protection for Domino. For that, refer to the production documentation:

• http://publib.boulder.ibm.com/tividd/td/StorageManagerforMail5.2.html

Usually your Storage Administrator will install Data Protection for Lotus Domino and run the dominstall program. The dominstall program automatically configures Data Protection for Lotus Domino to work with your Domino server. This program replaces the domsetup script provided in earlier releases of Data Protection for Domino.

The dominstall program also supports multiple Domino server partitions. For more information on installing and configuring Data Protection for Lotus Domino, see IBM Tivoli Storage Manager for Mail 5.2: Data Protection for Lotus Domino for UNIX and OS/400 Installation and User's Guide, SC32-9056.

These are the Domino environment variables:

DOMI_DIR	Directory where Data Protection for Lotus Domino was installed. The default installation directory is /opt/tivoli/tsm/client/domino/bin.
DOMI_LOG	Directory where the Data Protection for Lotus Domino log file (domdsm.log) will be stored. The default is the directory where Data Protection for Domino is installed. Specify this variable to change the default setting.
DOMI_CONFIG	File name of the Data Protection for Lotus Domino preferences file. The default is domdsm.cfg in the directory where Data Protection for Domino is installed.

Tivoli Storage Manager environment variables:

DSMI_DIR	Directory where the Tivoli Storage Manager API is installed. This environment variable is required and there is no default.
DSMI_LOG	Directory where the Tivoli Storage Manager API error log file (dsierror.log) will be stored. The default directory is the directory where Data Protection for Domino is installed. Use this variable to change the default setting.
DSMI_CONFIG	The Tivoli Storage Manager API options file name. The default is the dsm.opt file.

Refer to the README file that comes with the product for the requirements and specific details for installing Data Protection for Lotus Domino on Linux.

We set up our environment for testing Data Protection for Lotus Domino for Linux on zSeries using these settings for the Domino environment variables:

    export DOMI_DIR=/opt/tivoli/tsm/client/domino/bin    export DOMI_LOG=/domserva/notesdata    export DOMI_CONFIG=/domserva/notesdata/domdsm.cfg 

These were our settings for the TSM environment variables:

    export DSMI_DIR=/opt/tivoli/tsm/client/api/bin    export DSMI_LOG=/domserva/notesdata    export DSMI_CONFIG=/domserva/notesdata/dsm.opt 

Important: 

The most common reason for problems with starting Data Protection for Lotus Domino is that the environment variables have not been set correctly. You need to ensure that all Domino and TSM variables have been defined.

Data Protection for Lotus Domino uses three files to store configuration information:

domdsm.cfg	The Data Protection for Domino preferences file. It contains parameters specific to Data Protection for Domino.
dsm.opt	The Tivoli Storage Manager client options file. It identifies the TSM server to contact and the node name by which Data Protection for Domino is known to the TSM server.
dsm.sys	The Tivoli Storage Manager system options file. It contains stanzas to identify the TSM server to access. It also specifies backup and restore processing options and communication parameters.

Note 

If you are using the TSM backup-archive client together with Data Protection for Domino, it is recommended that you define separate stanzas in the dsm.sys system options file for the two clients. It is also recommended that you use different node names for the two clients.

The dsm.sys file that we used is shown in Example 9-19:

Example 9-19: Sample dsm.sys file

 domserva@linuxa:/opt/tivoli/tsm/client/api/bin > cat /opt/tivoli/tsm/client/domino/bin/dsm.sys    ************************************************************************    * Tivoli Storage Manager *    *                                                                      *    * Sample Client System Options file for UNIX (dsm.sys.smp)             *    ************************************************************************       SErvername  domserva       COMMmethod         TCPIP       TCPPort            1500       TCPServeraddress   wtscmxa.itso.company.com       PASSWORDACCESS PROMPT       NODEname            linuxa    *   Password            linuxa       SCHEDLOGRetention 30       ERRORLOGRetention 30    DOMAIN   /    /domserva/notesdata    EXCLUDE  *    INCLUDE  /etc/passwd    INCLUDE  /etc/group    INCLUDE  /domserva/notesdata/mail1/*.nsf 

The dsm.opt file that we used is shown in Example 9-20:

Example 9-20: Sample dsm.opt file

 domserva@linuxa:/domserva/notesdata > cat dsm.opt    *======================================================================*    * Tivoli Data Protection for Lotus Domino                               *    ************************************************************************    SErvername domserva    Domain     /domserva/notesdata     * Quiet      Subdir YES      Tapeprompt NO 

9.7.6 Data Protection for Lotus Domino operations

We review some of the main command line commands for Data Protection for Lotus Domino to query, back up. and restore your Domino databases with sample results from our server.

Query commands

To display the current values for Data Protection for Domino, issue domdsmc query preferences, as shown in Example 9-21 on page 254.

Example 9-21: domdsmc query preferences

 domserva@linuxa:/domserva/notesdata > domdsmc q pref Data Protection for Domino Preferences -------------------------------- BUFFers ........................ 3 BUFFERSIze ..................... 1024 DATEformat ..................... 0 LOGFile ........................ domdsm.log LOGPRUne ....................... 60 MOUNTWait ...................... Yes NOTESInipath ................... /domserva/notesdata NUMberformat ................... 0 REPlace ........................ Yes STATistics ..................... No SUBDir ......................... No TIMEformat ..................... 0 

Note 

To change a configuration parameter, you can use the set command. For example, to change the date format, issue: domdsmc set DATEformat=1.

To display general information about the Domino Server, we issued domdsmc q domino. A portion of the output is shown in Example 9-22.

Example 9-22: domdsmc q domino command

To display general information about the Tivoli Storage Manager Server, we issued domdsmc query adsm, as shown in Example 9-23.

Example 9-23: domdsmc q adsm command

To display a list of Domino database backups stored on the TSM server, issue domdsmc q dbbackup, as shown in Figure 9-9 on page 256.

Figure 9-9: domdsmc q dbbackup

Backup commands

To back up a specific database, enter domdsmc selective, as shown in Figure 9-10 on page 257.

Figure 9-10: domdsmc selective

Note 

If you issue domdsmc selective *, it will back up all databases.

To back up the transaction log files when using archival logging, use the domdsmc archivelog command, as shown in Figure 9-11.

Figure 9-11: domdsmc archivelog

You can also back up the transaction log files using a threshold. In Example 9-24, we used a 70% threshold.

Example 9-24: domdsmc archivelog with threshold

 domserva@linuxa:/domserva/notesdata > domdsmc archivelog -adsmpwd=mypassword -threshold=70,0 Starting Domino transaction log archive... Initializing Domino connection... Logging on to the Tivoli Storage Manager server, please wait... Total Domino transaction log files ready for archive: 3 Total Domino transaction log files archived:          0 Throughput rate:                                      0.00 Kb/Sec Total bytes transferred:                              0 Elapsed processing time:                              0.00 Secs 

Restore commands

You can use the domdsmc restore command to restore database backups and optionally activate the databases, as shown in Figure 9-12 and Figure 9-13 on page 259.

Figure 9-12: domdsmc restore

Figure 9-13: domdsmc restore (continued)

We selected the file to restore by pick> 1 as shown in Figure 9-12, and then typed o for ok to start the restore, as shown in Figure 9-13 on page 259.

To bring the restored database online and apply the logs, use the domdsmc activatedbs command as shown in Figure 9-14 and Figure 9-15 on page 261.

Figure 9-14: domdsmc activatedbs

Figure 9-15: domdsmc activatedbs (continued)

We recommend that you make at least one full backup (domdsmc selective) each week. A full backup may be required more often on a server with a large number of transactions in order to reduce the number of transaction log extents which much be processed during a restore. You must find a balance between frequent full backups and long restore time.

An archivelog with no threshold (domdsmc archivelog) should be done once a day, preferably off-shift, as well as a daily incremental backup (domdsmc incremental), which would perform full backups of nsf files for which DBIIDs have changed (due to fixup or OS operation).

You should also inactivate backed up files that do not have any transactional logs associated with them (domdsmc inactive). Example 9-25 shows a sample schedule for Data Protection for Lotus Domino using crontab.

Example 9-25: Sample schedule for Data Protection for Lotus Domino using crontab

 crontab -l ###TDP Daily Archive 00 04 * * * domdsmc archivelog ###TDP Inactive Files 00 11 1 * * domdsmc inactive ###TDP Archive 01 * * * * domdsmc archivelog -threshold 70,0 ###TDP Incremental 00 01 * * * domdsmc incremental ###TDP Full Backup 00 1 * * 0,2,4 domdsmc selective "*" 

9.7.7 Anti-virus

An anti-virus solution was not available for testing at the time of writing. However, we could see a solution available from Trend Micro in the future. See the Trend Micro Web site for more information.

• http://www.trendmicro.com/en/products/email/smln/evaluate/overview.htm

9.7.8 Anti-spam

An IBM Redbook devoted to the topic of avoiding spam in a Domino 6 environment is available; refer to Lotus Domino 6 spam Survival Guide for IBM, SG24-6930.

---