5.2 In depth: Exchange 2003 backup operation

One of the keys to a solid disaster-recovery strategy for Exchange is thorough understanding of how Exchange’s database engine performs backup operations. While there is a tremendous amount of good information on the basics of how Exchange backup operations work, much of this information is inaccurate or not detailed enough for careful planning of your Exchange Server recovery. In this section, we will look closely at how Exchange backup works at the database engine and API level. A similar discussion on restore operations is included in the next section. Also, note that while this discussion is centered around Exchange 2000/2003, previous versions work similarly, even though several minor variances exist between different version and service pack levels of Exchange Server.

5.2.1 Starting with the basics

The Exchange database engine is known as the Extensible Storage Engine or ESE. ESE is a transacted storage engine that leverages write-ahead logging technology to perform transacted operations against a database or databases that are structured in a Microsoft-implemented B+Tree format. With a write-ahead logging database engine, write operations to the database are first written to the transaction logs and then to database pages cached in memory (called IS buffers). These write operations are later written to disk asynchronously in an optimal manner designed to batch these operations such that disk I/O performance is maximized. As transactions are committed to the database on disk, a database checkpoint (maintained in the EDB.CHK file) is advanced to reference the current point in the current transaction log file of the last transaction that was committed to the database. ESE uses discrete log files that are 5 MB in size and generationally sequenced. As transactions are written to the log files, once the log file reaches 5 MB in size, it is closed and a new log file is open (the current log file is always E0 n .LOG—where n signifies the storage group instance to which the log file belongs). This implementation provides high performance and maximum reliability expected from a transacted database engine.

ESE was also designed to allow the databases to be backed up while the server was on-line servicing users. These means that Microsoft had to devise a method where the databases could be stored to a backup set while they are mounted and transactions are being committed. For this purpose, the ESE backup API was designed. In Exchange 2000/2003 the backup API is implemented in two DLLs—ESEBACK2.DLL and ESEBCLI2.DLL— that allow backup applications to interface with the database engine to perform on-line backup and restore operations for Exchange. In the following discussions, the ESE backup API calls will be referenced as we step through the backup operation for Exchange Server. As an interesting side note, Windows AD is also based on ESE, and backup operations and API calls used for backup and restore are very similar to those discussed next.

Figure 5.1: Exchange 2003 backup operation.

5.2.2 Initializing the backup operation

At the start of Exchange’s backup operation (see Figure 5.1), several important operations occur. Two important API calls are initiated by the backup application (this could be Windows/NT Backup or a third-party backup product such as Legato, Veritas, or CA) at this point to begin a backup operation. HrESEBackupPrepare is called to establish an RPC connection to the information store process (STORE.EXE). Also, the HrESEBackupSetup API call is made and will specify which storage groups (ESE instances) will be involved in the backup operation. ESE responds to these API calls by performing several operations. First, when a full backup operation is initialized, ESE begins by flushing all dirty pages in the ESE cache (IS buffers) to disk and halting the checkpoint. The checkpoint will not advance until the backup operation is complete. If the backup is a not a full backup (i.e., differential, incremental, or copy), the checkpoint is allowed to advance, since the backup operation will not be touching the databases. Next, ESE will create patch files for each of the databases backed up. Patch files are used in special circumstances (transactions that require page splits and/or merges) during backup operations to ensure database integrity. If you are running Exchange 2000 SP2, patch files are not used, since Microsoft has figured out how to avoid this requirement in the SP2 release.

5.2.3 Copying the databases

The next step in the process involves backing up the database files. ESE gets the list of storage groups and databases to be backed up from the backup application via the backup APIs ( HrESEBackupSetup ). The backup application makes three API calls to backup the databases. HrESEBackupOpenFile opens the database file for reading, HrESEBackupReadFile reads the database, and HrESEBackupCloseFile closes the database when all the pages have been read. These databases are not simply copied to the backup set because they are open files (remember Exchange backups allow the databases to be back up on-line). Instead, ESE begins to send the backup application 64-KB chunks (16 4-KB pages at a time) of database pages in sequential order. This is also the crucial step where each page is checksummed and an error results in the backup operation terminating with a –1018 (the HrESEBackupReadFile is the API call that initiates checksumming operations for database, log, and patch files during the backup operation). ESE will continue this process until all pages for each database are sent to the backup application. Also remember that Exchange 2000/2003 supports two database files—the Property Store (EDB) and the Streaming Store (STM). Both database files are read page-by-page by ESE and sent to the backup application to store as part of the backup set. ESE will perform these operations for each database being backed up.

5.2.4 Backing up the transaction logs

During a full backup operation, the transaction logs must be stored to the backup set next. Remember from above that the checkpoint is halted at the beginning of backup (for a full backup). Even though the checkpoint is halted, transactions continue to be written to the log files, and dirty pages from the database cache in memory are flushed to disk during the backup operation. Note that with Exchange 2000 SP2 and later, dirty pages that cause patching operations are not flushed; this is how Microsoft was able to get rid of the patch files in SP2. In order to back up the log files, the backup application requests a list of log files (and patch files if applicable) from ESE (via the HrESEBackupGetLogAndPatchFiles API call). When ESE receives this call, it closes the current log file as the next log generation and opens a new E0 n .log. In the case of a full backup, ESE then returns a list of log files to the backup application that starts with the current log generation where the checkpoint is halted and ends with the generation of the log file that was just closed (E0 n .LOG – 1). In the case of an incremental or differential backup, ESE returns a list beginning with the oldest generation on disk up to the most recent log file closed (E0n.LOG – 1). During incremental, differential, and copy backup operations, only log files are backed up and the checkpoint is not halted. Using this list of files, the backup application can open file handles to the logs and copy them to the backup set (using the HrESEBackupOpenFile, HrESEBackupReadFile , and HrESEBackupClose-File ). During these operations, ESE also ensures that no log generation is missing from the sequence passed to the backup application. This operation will continue until all relevant log and patch files have been written to the backup set by the backup application.

5.2.5 Truncating the logs

Since the log files have been stored to a backup set, they are not needed on disk. ESE will then truncate the log files during full and incremental backup operations. When ESE receives the call from the backup application to truncate the logs ( HrESEBackupTruncateLog API call), it will truncate the log files on disk. Which log files ESE truncates is determined by the lowest generation of (1) the checkpoint log file generation, or (2) the log generation listed in the database header (you can view this using the ESEUTIL program with the /MH parameter) for the current full backup.

5.2.6 Cleaning up

Once the log files are truncated, the backup operation is complete and the backup set is closed. At this point, ESE can return to normal operations and allow the checkpoint to advance forward. To complete the backup operation, the backup application calls HrESEBackupInstanceEnd to end operations for that instance of backup. At this point ESE is able to allow the checkpoint for the storage group instance being backed up to advance and normal database operations to recommence. Finally, the backup application calls HrESEBackupEnd to disconnect from the information store process allowing it to return to normal operations.

5.2.7 Implications for backup planning and procedures

Traditional best practices for backup call for a combination of full and incremental or differential backups to ensure data recoverability. However, within the Exchange space, the de facto backup best practice is daily full backups with no incremental or differential backups (see Table 5.1 for the different backup types supported by Exchange and their impact on database and log files). For the most part, this strategy has served Exchange administrators well. However, in the event that a particular backup tape media goes bad, there is a potential for data loss.

Let’s look at this problem closer. Suppose you are performing daily full backups for your Exchange server. On Day 3 of the rotation, your Exchange database becomes corrupt and you must restore from backup. However, when you go to use the backup tape from Day 2, you discover that the media is damaged, rendering the backup unusable. In this scenario, you would be forced to go to the tape and backup set for Day 1. However, if you use this backup set, it contains a valid database and log files up to the point in time that the backup was performed on Day 1. On the subsequent day (Day 2), the full backup operations have truncated the log files residing on disk that occurred after Day 1. The consequences of this backup strategy provide for recovery of your Exchange data up to the point of backup on Day 1 (in the event that the Day 2 backup set is bad). However, since subsequent full backups have been performed on Days 2 and 3 and log files have been truncated, there are gaps in the generational sequence required to recover to Day 3 using the Day 1 backup set (Day 2’s backup is bad). At this point, the Exchange administrator will find himself or herself in a position where he or she can recover to Day 1 and has some log files on disk that were created since the Day 2 backup. There are gaps in the sequence. For example, if the database and log files 001 and 002 were recovered from the Day 1 backup set, Day 2 stored the database and logs 003 and 004, and logs 005 and 006 are on disk on Day 3, the loss of the Day 2 backup set results in a gap in the generational sequence of two log files. (Logs 001, 002, 005, and 006 are available. Logs 003 and 004 are not available.) This means that recovery is only possible to the point of log file 002 since ESE will not allow recovery to proceed beyond the gap in the sequence.

The above scenario exposes a potential flaw in the daily full backup strategy that has become the de facto standard for Exchange Server disaster recovery. To work around this potential problem, we must look at two things. First, we must address the issue of bad media. If frequent media failures are plaguing your disaster-recovery efforts, it is important to take steps to address this issue outside of Exchange management and look at the causes and factors that are contributing to this problem. You may need to look to your hardware vendor or to your procedures and processes to find the cause of frequent media failures. However, even with the most proactive approach, top-notch hardware, and bulletproof best practices, media failures may still occur. If your SLAs dictate the up-to-the-minute recovery of Exchange data, you may need to consider enhancing your backup strategy to protect from media failures and other issues that would result in missing sequences of log file generations.

One approach to the problem of missing log file sequences as a result of missing or bad backup sets is to augment daily full backups with differential backups at the mid-point during the day. For example, if you perform a full backup at 12:00 A.M., you would perform a differential backup at 12:00 P.M. This would increase the number of log files available on your backup media and potentially lessen the possibility of missing log files in the event of a media failure. However, there would be no guarantees. Another approach would be to perform a copy backup (a copy backup copies the databases and the log files, but does not truncate the log files) at 12:00 A.M. followed by an incremental backup at 12:00 P.M. This approach would preserve all log files during the copy backup and truncate the logs during the 12:00 P.M. incremental backup. Alternately, the 12:00 P.M. incremental backup could be a differential backup, which would not truncate the log files. With this alternative (copy + differential), however, the logs would never be truncated. It is up to the Exchange administrator to understand the implications of the type of backup and schedule chosen along with the potential hazard of bad or lost media. It is important that disaster recovery plans provide the right backup types and schedule to meet the established SLAs.

5.2.8 The poor, misunderstood patch file

Although no longer pertinent to Exchange Server 2003 (the need and use of patch files by ESE was eliminated with Exchange 2000 SP2), but important to versions prior, it is important not to avoid skipping this topic for those who are still running versions of Exchange Server prior to Exchange 2000 SP2. Patch files (*.PAT) are special-purpose files used by the Exchange Server database engine on limited occasions. During on-line backup operations, the database file is written out 64 KB at a time in a sequential fashion. In other words, the 4-KB database pages are written sequentially in groups of 16 pages at a time (16 4 KB = 64 KB). Since Exchange Server allows backup operations to be performed while the server is running and users are connected, it is possible that changes to pages in the already written (or backed up) section of the database will not be on tape. This case, however, is handled by Exchange since these changes will be stored in the transaction logs, which will be copied to the backup as well. Another case (illustrated in Figure 5.2) exists, however, in which a single 4-KB page that resides in the portion of the property store (EDB file) already copied in the backup becomes full (i.e., all 4 KB are used) as a result of transactions that have occurred since the backup was begun. In this case, the page must be split and a new page allocated in the B-Tree structure of the database (remember that the streaming store is not a BTree database structure and therefore does not require patch file measures). A similar situation occurs if pages must be merged. If the pages are involved in a split or merge operation occurring across the backup committed/uncommitted boundary, they cannot be handled by the transaction logs alone. Therefore, these operations must be written to a separate location in order for the property store database to remain consistent when it is restored from backup. Updates to pages in the portion of the database that has already been copied in the backup are stored in the patch file. There is a patch file created during on-line backup for each database. During recovery, the database engine will update the database with any pages that are stored in the patch file. Starting with Exchange 2000, Microsoft implemented the same level of integrity checking of each page in the patch file as is available for the property store. Each page in the patch file is verified using the checksum stored in the page header. Patch files are an important concept to understand concerning disaster recovery for Exchange server running versions prior to Exchange 2000 SP2. This entire discussion of patch files and their use is purely academic since the important point is that ESE takes care of everything for you.

Figure 5.2: ESE’s patching operation during backup (Exchange 2000 SP1 and earlier).

So how did Microsoft eliminate the need for patch files in Exchange 2000 SP2? Surprisingly, it was rather simple. Microsoft developers determined that the reasoning behind the original design requirement was a bit flawed. The patch file was needed because of the possibility that database page modifications that resulted in pages being merged or split could not be handled well during backup operations because, if the location (either before or after the current backup location) of these merged or split, pages would be unknown. They decided to invent a mechanism to handle this—but was it the best way to solve the problem? After many years, the JET/ESE developers came to the conclusion that they could handle the page merge/split scenario and effectively accomplish the same thing as the patch file by simply not advancing the ESE checkpoint and not flushing dirty database pages to disk during backup operations. In this manner, ESE is able to handle page splits and merges during a backup operation. This little thing is a big win since the elimination of the patch file just means there is one less thing to go wrong during hard recovery operations. I wish I could have seen the light bulb go on and heard the resulting “Duh!” when ESE developers figured this one out.

5.2.9 ESE’s Page Zeroing feature

During an on-line normal backup, another important feature is available. This is ESE’ Page Zeroing feature. Page zeroing is the ability to “zero” each deleted page in the database. This is typically implemented as a security measure in which pages of the database that have been logically deleted (i.e., a user deletes a mail message, and it has been aged out of the deleted items cache) are overwritten to ensure that the data is truly deleted and cannot be recovered by would-be spies, hackers, or U.S Department of Justice staff members with too much time on their hands. ESE Page Zeroing became available in Exchange 5.5 Service Pack 2 (released in December 1998) and is an important feature for Exchange deployments desiring the highest levels of data security. ESE Page Zeroing for Exchange 2000/2003 is enabled via an Exchange System Manager option (shown in Figure 5.3) and is available on a per-storage-group level.

Figure 5.3: Enabling ESE Page Zeroing in Exchange system manager.

Microsoft chose to implement page zeroing as part of the backup process. More specifically, since this operation must touch the database and has nothing to do with the log files, page zeroing is done as part of an on-line normal backup operation. During a normal backup (when ESE Page Zeroing is enabled), as the database engine checks the integrity and copies pages to backup media, it will also zero delete pages in the database by writing a specific byte pattern to the page. Technically, the pages are not zeroed, but contain a byte pattern known to the database as an empty page with no data. Regardless of the technicality, each deleted page no longer contains the original data and is safe from potential security threats. As you might guess, ESE Page Zeroing can be a resource-intensive process for your Exchange server. Additional overhead beyond what the backup operation already consumes is required to perform the zeroing operation. When ESE Page Zeroing is enabled, the first normal backup operation will be the most resource intensive because all deleted pages in each database are zeroed. Once the initial operation has been completed by the database engine, only newly deleted pages will need to be zeroed on subsequent normal backups. If you select a strategy of only one normal backup per cycle, all page zeroing operations are only performed at that time. If you are concerned about the additional overhead of ESE Page Zeroing, I recommend that you perform the initial backup (doing a backup to disk is extremely fast) of your databases at a time of low user activity (which may be the most typical case anyway). Subsequent normal backups should not be particularly resource intensive. Also, if you have a large occurrence of deletion, such as when a large number of mailboxes or public folders are moved or deleted, I recommend the same procedure as in the case of the initial backup previously discussed. Overall, page zeroing should not be a significant performance problem on your Exchange server. However, following these recommendations may make your life a bit easier in the long run.

The backup operation for Exchange (ESE) is very intricate, and it can be confusing. However, it is crucial that Exchange administrators understand how this important operation works. As you can see, how you structure your backups and the combination of full and incremental or differential backups you employ will determine the recoverability of your valuable Exchange data. As an exercise, you should always plan on testing backup operations in a lab environment in order to understand this process better.