7.9 Database utilities

---

Table 7.8 lists the three utilities you can use to maintain or verify Exchange databases.

Table 7.8: Exchange Database Utilities

Utility	Purpose	Location
ESEFILE	Verify checksums of ESE pages	Server CD (see the SUPPORT\UTILS\i386 folder)
ISINTEG	Check folders and other application-level structures	Included in Store
ESEUTIL	Verify physical page structure	EXCHSRVR\BIN

A great deal of mystique has grown up around the ESEUTIL and ISINTEG utilities over the years, with some administrators believing that running either or both is sufficient to cure all database ailments. The truth is somewhat different. ESEUTIL can certainly help to get a database running again, but it cannot fix fundamental damage or corruption caused by hardware failure. In these circumstances, if you encounter a series of -1018, -1019, or -1022 errors in the event log, the best procedure is to restore the database from the last good backup and roll forward all subsequent transactions by replaying the transaction logs.

ISINTEG verifies the application-level structures (tables and pointers), but it depends on a good database. Thus, if ESEUTIL cannot fix the problems in a database, ISINTEG will not be able to work any magic either.

7.9.1 Running ESEUTIL

ESEUTIL has no knowledge of database contents and operates on a page level in different modes. When recovering from a database problem, the two most important modes are /D (defragmentation) and /P (repair). Always take a file-level backup of a database before you begin working with ESEUTIL. Only use the version of ESEUTIL compatible with the version of Exchange running on the server.

You can only run ESEUTIL against a dismounted database. Defragmentation means that a utility rebuilds the database, page by page, and this is usually done to recover disk space to the file system; you must have at least 120 percent of the database size available in free space to accommodate the temporary files used during processing. You can also redirect the location of the temporary files to a disk that has sufficient free space. If necessary, you can even run ESEUTIL on a computer that does not have Exchange installed by transferring the necessary program files and databases to that computer.

The Store runs an online defragmentation process nightly for every database on a server to "shuffle" pages internally so that linked pages are kept together and the "white space" (unused pages) is placed at the end of the database. However, online defragmentation never returns space to the file system and the overall size of the database is not reduced. Early versions of the online defragmentation process were not very efficient, which led to recommendations that administrators should rebuild their databases on a regular basis to prevent the databases from becoming too large. From Exchange 5.5 onward, the online defragmentation process is much more efficient and effective and you should not need to perform an offline defragmentation (rebuild) unless you have good reason to suspect that a large amount of disk space can be recovered.

After online defragmentation, ESE reports the amount of unused pages in the application event log as event 1221 (see Figure 7.26). In the left-hand screen (Exchange 2000 on Windows 2000), ESE reports that 4,633 MB of white space is available in a 9.32-GB Mailbox Store, which means that you could recover the space occupied by almost half of the database if you use ESEUTIL to perform an offline defragmentation. However, this is an exceptional occurrence due to many mailbox moves to another server. The right-hand screen (Exchange 2003 on Windows 2003) shows more typical results, with only 50 MB recovered in a 9-GB Mailbox Store.

Figure 7.26: Result of online defragmentation.

You can check events 700 (start of run) and 701 (end of run) to determine how long online defragmentation takes. In this case, processing lasted three hours and eight minutes, or roughly 3 GB/hour. Throughput varies with processor speed; I/O capacity; and system load, including other management operations that may be proceeding at the same time, such as the emptying of the deleted items cache, defragmentation of other databases, and the generation of the Offline Address Book.

Using ESEUTIL to rebuild a database (/D)

Typically, you should not consider running ESEUTIL to perform an offline rebuild unless more than 30 percent of the pages in the database is free space, a situation that normally only exists after a number of mailboxes are moved from one database to another, to another server, or the database has been left in use for many months.

The major advantage that you gain by rebuilding the database is the elimination of all white space in the file, leading to the recovery of disk space to the file system. Backups will therefore complete faster, since they have less data to process and it is more likely that the pages that constitute a mailbox and its contents are contiguous within the database, so they can stream out faster to the backup media. Some commentators have also observed that large databases often cause NTFS to extend the size of the file several times. The more extents there are on a disk, the longer Windows takes to run CHKDSK to verify the disk after an unexpected reboot. Hopefully, servers will not have many unexpected reboots, so removing Exchange from service to perform an offline rebuild to speed up CHKDSK does not really deliver a lot of benefit weighed against the disruption to users.

While a fully defragmented database occupies less space on disk, you must understand that the database will begin to expand almost as soon as it comes back into production use. It may, therefore, grow to reoccupy all of the recovered disk space in a short period.

To rebuild a database with ESEUTIL, you proceed as follows:

• Take a full online backup.

• Dismount the target database and so remove service from users whose mailboxes are in the Store (or remove service to public folders).

• Run ESEUTIL /D.

• Remount the database after ESEUTIL completes successfully,

• Take a full online backup, since you can no longer roll forward transactions in previous transaction logs to the newly rebuilt database.

Note that ESEUTIL defragments both the EDB and STM databases in the same operation.

When it runs, ESEUTIL reads a page from the source database, verifies its content, and writes the page to the new copy of the database. ESEUTIL ignores empty pages, so if the problem exists in an empty page, ESEUTIL can fix it because it does not write the problem page into the new database. However, if the page contains data, ESEUTIL will not be able to complete the defragmentation and you will have to resort to repair mode. Figure 7.27 shows how ESEUTIL reports the successful rebuild of a 42-GB database. In this instance, the source and target volumes were a 36-disk set and the process took 164 minutes, or approximately 15.38 GB/hour.

Figure 7.27: ESEUTIL succeeds in a database defragmentation.

The order in which you state the qualifiers in the command line is important. Always put the major option first (rebuild, repair, integrity check, etc.) followed by any options that affect how ESEUTIL runs.

ESEUTIL repair mode (/P)

When we refer to a database repair, we mean that you run ESEUTIL in a mode where it discards any problem page if it fails a checksum test. Sometimes, we refer to running ESEUTIL in this mode as performing a "hard" repair, because there is no middle ground in the decision to keep or reject pages, so data is easily lost. The only pages that end up in the new copy of the database are those whose checksums can be verified by ESEUTIL. The impact of discarded pages depends on the data they contain. Discarded leaf pages mean that important system or user data is no longer available, so complete folders may disappear from mailboxes. However, if ESEUTIL discards the pages containing system catalog data, the database may be unusable.

Repairing a database is a last-chance option that you should only proceed with if a good backup is unavailable. ESEUTIL is not a fast utility and typically processes data at up to 8-10 GB/hour depending on the system configuration, so running ESEUTIL against a moderately sized file will take hours to complete and you have no guarantee that the repair will be successful. Before starting a repair, ensure that you check out all possible hardware faults that may have contributed to the corruption in the first place. There is no point in spending hours waiting for ESEUTIL to repair a large database only to find that either the repair job fails or the newly repaired database runs into immediate problems because the storage subsystem is flaky.

If you successfully repair a database with ESEUTIL, the resulting database will function properly and Exchange will run. However, it is possible that some lingering problems persist in the database's internal structures that make it prone to future problems. For example, a corrupt structure may cause the Store process to consume excessive CPU or memory. For this reason, some administrators consider that the repair only partially completes the job and plan to swap in brand new databases to allow them to resume the messaging service for users, albeit with blank mailboxes. Meantime, behind the scenes, the administrators attempt to repair the failed database files and then recover the mailbox data and reload it back into the online databases. You need a separate recovery server for this exercise with Exchange 2000, but you can use the new Recovery Storage Group feature (see Chapter 11 for details) in Exchange 2003 to do everything on the same server, providing you have enough spare disk space to run the recovery utilities. While swapping in new databases and fixing failed databases elsewhere invariably results in a "clean" database, it cannot fix an underlying disk or storage subsystem problem that may have caused the corruption in the first place. In addition, the rebuild causes considerable disruption to users, including the loss of some data that ExMerge is unable to export and import. For this reason, creating new databases is very much a step that you shuold take only as the absolute last resort when you know that you cannot get back to a supportable situation in any other way.

Note that from Exchange 2000 SP1 onward, you can specify the /createstm switch for a repair operation to force ESEUTIL to create a brand new STM file if the current STM file is unrecoverable due to corruption. Do not take this step unless you are sure that the STM file is irreversibly damaged, since you immediately lose data when ESEUTIL creates the STM. Remember to take a full backup of the repaired database after ESEUTIL completes processing.

ESEUTIL and low disk space

Sometimes you will encounter a situation where you want to run ESEUTIL to rebuild but you do not have enough free disk space on your server to allow for the temporary storage used by ESEUTIL when it processes the database. In order of ease of use, there are three possible solutions:

• Map a network drive on a temporary basis, move the databases to the newly mapped drive, run ESEUTIL, and then move the files back to their original location.

• Copy the databases to another server that has enough available disk space and run ESEUTIL there.

• Redirect the location of the temporary files to some network storage.

You can specify a location for ESEUTIL to create the temporary database created during an offline defragmentation (/D) or repair (/P) operation with the /T qualifier. When the operation is complete, ESEUTIL moves the temporary database back to take the place of the original database, unless you use the /P qualifier to instruct ESEUTIL not to replace the original.

Using another server to run ESEUTIL is useful when you want to perform other work (such as applying an operating system upgrade) on the server where Exchange normally runs. You do not even have to use a server that has Exchange installed, since it is possible to move ESEUTIL.EXE and a small set of associated DLLs to a target server and run ESEUTIL there. Microsoft Knowledge Base article Q244525 describes the necessary steps. In all cases, remember to take a backup before and after you run ESEUTIL.

7.9.2 ISINTEG

ESEUTIL repairs a database on a page level but does not address any logical faults that exist within the tables that form the content of the database. For example, in some versions of Exchange 2000, it is possible for the Store to report inaccurate mailbox quotas if clients convert MAPI format messages to MIME, a reasonably common situation when you run a mixture of Outlook and Internet clients. ISINTEG can address problems such as this by examining the contents of the Store and then updating attribute values to reflect the true situation. You can check a database for logical faults by running ESEUTIL with the /MH switch to check the database header. Look for the "repair count," and, if this is greater than zero, run the ISINTEG utility to fix the logical errors, as follows:

ISINTEG -S [server_name] -FIX -TEST ALLTESTS

Unlike versions of Exchange prior to 2000, the ISINTEG utility is merely a stub image that invokes the code to perform the tests. This code is part of the Store.

Despite rumors to the contrary, ISINTEG cannot fix problems caused by hardware corruption. If you have a damaged database caused by hardware corruption and you cannot repair the problem with ESEUTIL (even at the expense of discarding damaged pages and losing data), it is not worthwhile to run ISINTEG. The only solution is to restore the database from the last good backup and replay transaction logs to recover any outstanding transactions. Note that if you do repair a database with ESEUTIL, it is possible that some of the relationships between tables in the database will become inconsistent because of the repair. In this situation, running ISINTEG after ESEUTIL successfully processes a database completes the fix-up by restoring lost links.

7.9.3 ESEFILE

ESEFILE serves two major purposes. First, it can be used (/C qualifier) to copy large files in a very efficient manner. While the utility helps to move large EDB databases from disk to disk, perhaps to run ESEUTIL on another computer, it can actually copy any Windows file. Do not use ESE-FILE to move databases to a new location for any purpose other than running ESEUTIL. Always use ESM if you want to move a database's location, since this ensures that you update Exchange's configuration data in the AD.

The second purpose is to perform a checksum test against a complete database (/S qualifier) or to verify a selected page (/D qualifier). As already noted, Exchange uses a simple algorithm to generate and verify checksums, so ESEFILE can process very large databases quickly. Backup products that mimic snapshots by stopping the Store, breaking a mirror copy, and then restarting the Store sometimes use ESEFILE to validate that the databases on the mirror are valid and intact. This is important, because the database copy on the mirror is a "crash consistent copy," similar to a database after an unexpected exit and may contain errors such as torn pages. For this reason, it is much better to use the combination of Windows 2003 Volume ShadowCopy Services (VSS), Exchange 2003, and suitable hardware and backup software to take fully supported hot snapshot backups. See Chapter 8 for more information on VSS.

---