Recovering from Database Corruption

If an Exchange database is corrupt, it is not extremely effective to restore the corrupt database to a production server. The server might continue to operate , but database corruption never goes away on its own, and you eventually will have to repair the database. In fact, when minor database corruption is not repaired, the corruption can get to the point that entire sections of the Exchange database become inaccessible.

A couple methods can be used to repair a corrupt Exchange database, or at least restore the database and extract good information from the database. Key to the successful recovery of as much information as possible is using the right tool. In many cases, administrators jump right into using the ESEUTIL /p repair command; instead of repairing the Exchange database to 100% condition, the utility finds a corrupt section of the database and deletes all information from that portion of the database on. So, although the Exchange system becomes 100% clean, the utility deleted 2030% of the data that was in the database to get the database to a clean state. The ESEUTIL /P command is the task of last resort: Other tools work around corrupt database areas and allow the administrator to recover as much of the data as possible.

Going all the way back to the start of the chapter in the "What to Do Before Performing Any Server-Recovery Process" section, this is where having a complete backup of the databases in Exchange is really important. If a process to repair or recover information causes more harm to the database than good, there is still a backup copy to restore and start again.

Flat File Copying the Exchange Databases

One of the best techniques Exchange experts use when working to recover corruption in a database is to make a flat file copy of the Exchange databases. A flat file copy is merely an exact copy of the Exchange databases copied to another portion of the server hard drive or to another server. To do a manual copy of the databases, do the following:

1. Unmount the Exchange database stores by going into the Exchange System Manager. Traverse the tree past Administrative Groups, Servers, Storage Group . Right-click on the mailbox store and select Dismount Store.

2. Dismount the store for all mailbox stores you will be working on.

3. Copy (using Windows Explorer, or XCOPY) the *.edb and *.stm files to a safe location (usually the filenames are priv1.edb and priv1.stm and are stored in \Program Files\Exchsrvr\mdbdata\ for the defaulthowever, as additional databases are created, the directory names and filenames can differ ).

NOTE

If the databases need to be manually restored, a simple XCOPY (or Windows Explorer copy) of the databases back to the original subdirectories will bring the data back to the condition the databases were in at the time the databases were copied off the system. Inside the directories ( \Program Files\Exchsrvr\mdbdata or the like) are other files, such as tmp.edb , *.log, files. These files are support files to the Exchange databases. These files could be copied off the system and then copied back if a restore is desired. The support files beyond the basic priv1.edb , priv1.stm , pub1.edb , and pub1.stm files typically need to be copied as a group. After maintenance is run on the main *.edb and *.stm files, the support files are typically unuseable because they are no longer associated with the master database files.

Moving Mailboxes to Another Server in the Site

One way of extracting mail from a corrupt database is to move the mailbox or mailboxes to a different server in the site. Instead of trying to run utilities to fix the corruption in the database, which can take several hours (or even days, depending on the size of the database and the amount of corruption that needs to be fixed), an administrator can set up another server in the Exchange site and move the mailboxes to a new server.

Moving mailboxes grabs all of the mail, Calendar, contacts, and other mailbox information from one server and moves the information to a new server. As the information is written to the new server, the information is automatically defragmented and corruption is not migrated . Additionally, mailboxes can be moved from one server to another without ever having to bring down the production server. A mailbox user must be logged out of Outlook and must not be accessing Exchange before the mailbox can be moved. However, if mailboxes are moved when individuals are out of the office or at lunch , or on weekends, the mailboxes can be moved without users ever knowing that their information was moved from one system to another.

The two caveats to moving mailboxes are these : Corrupt mailboxes will not move, and user Outlook profiles will be changed. For Outlook profiles, because a user's Outlook profiles point to a specific server, when a mailbox is moved from one server to another, the user's profile also needs to point to the new server. Fortunately, with Exchange and Outlook, when a user's mailbox is moved, Outlook tries to access the mailbox on the original server, and the server notifies Outlook that the mailbox has been moved to a new Exchange server. The user's Outlook profile automatically changes to associate the profile to the new server where the user's mailbox resides. So, as long as the old server remains operational and the user attempts to access email from the old server, the profiles will be automatically changed the next time the user tries to access email. Typically within 12 weeks after moving mailboxes from one server to another, the user profiles are all automatically changed.

As for corrupt mailboxes, unfortunately , Exchange typically does not move a corrupt mailbox. So, if a user's mailbox has been corrupted, the mailbox will remain on the old server. Moving the data from the corrupt mailbox will need to be handled in a manner specified in the following section, "Extracting Mail from a Corrupt Mailbox." However, if 8090% of the user mailboxes can be moved to a new server, the administrators are trying to recover only a handful of mailboxes instead of all mailboxes on a server. This could mean far less downtime for all users who had mail on the server and could limit the exposure of data loss to a limited number of users.

To move mailboxes between servers in a site, do the following:

1. Open the Exchange System Manager.

2. Select the Administrative Group where the mailboxes to be moved reside.

3. Highlight the mailboxes to be moved.

4. From the Action menu, select the Exchange Tasks option and click the Next button on the welcome screen.

5. At the Available Tasks page, select Move Mailbox and click the Next button to continue.

6. Click the Mailbox Store option and choose the destination store where the mailboxes will be moved. Select Next when complete.

7. Configure the options for addressing corruption, and set the desired limits for this move. Select the Next button to begin the mailbox move.

8. Review the results of the mailbox move, and click Finish to complete the move.

Extracting Mail from a Corrupt Mailbox

When mailboxes cannot be moved between servers, either due to mailbox corruption or because the organization does not have a spare server to move mailboxes between, the exmerge.exe utility can be used to export mailboxes to a file. The ExMerge utility is freely downloadable from the Microsoft downloads page at www.microsoft.com/downloads (search for "ExMerge" or "Mailbox Merge Wizard").

ExMerge allows an administrator to select specific mailboxes and export the data from the mailbox into a PST file. The PST file can then be imported (using the ExMerge utility) into another server. Unlike the Move Mailbox tool, which is an all-or-nothing migration tool, ExMerge extracts information on an item-by-item basis. When corruption is found in a user's mailbox, the corrupted item is skipped and the balance of the user's information is extracted.

The ExMerge utility can be used to extract all mailboxes from a server and import the information back into a new server. However, because ExMerge extracts and imports information item by item, it could take more than an hour to extract 1GB of mail and another hour to import that 1GB of mail back into a new server. So, if the Move Mailbox tool can migrate the information directly from one server to another over the wire, the migration process of good mailboxes goes much faster. However, for corrupt mailboxes, the ExMerge method does skip corrupt portions of mailboxes and minimizes the loss of information.

To use the ExMerge utility, do the following:

1. Download the ExMerge utility from Microsoft.

2. Extract the exmerge.exe file (this explodes out four to five ExMerge utility files).

3. Copy the files into the \Program Files\Exchsrvr\Bin directory on the Exchange server.

4. Launch the exmerge.exe program and click Next through the welcome screen.

5. Choose Export or Import (Two-Step Procedure) and click Next.

   NOTE

   The ExMerge utility has an Export and Import (One-Step Procedure) option that exports the information from one server and imports the information into another server. Exchange experts do not commonly use this because they want to isolate the export and import functions to have better control over the results. If during the export or import processes an error occurs when performing both the export and import processes, it's harder to determine whether the problem was an export problem or an import problem. It's also harder to know where the problem faulted and where to pick up to complete the data-migration process. Using the two-step procedure ensures that a clean export can be successfully completed before an import is begun. Any problem in the export or import process also can more easily be identified.

6. Select Step 1 to extract data from an Exchange server, and click on Next.

7. Enter the name of the server where the information is being extracted; then click Next.

8. Select the storage group from which the mailboxes will be extracted; then select Next.

9. Choose the mailboxes that are to be extracted (hold down the Ctrl key to select individual mailboxes, or click on the Select All button to select all mailboxes to be extracted). Click on Next to continue.

   NOTE

   The ExMerge utility extracts information from the source server, but it leaves a copy of the information in the server. This allows information to be extracted without impacting the pre-existing state of information in the source server.

10. Select the default locale or language set used for the mailbox (for example, English US). Then click Next to continue.

11. Select the drive and subdirectory where the mailbox(es) should be extracted to; then click Next.

    NOTE

    Even if the data will ultimately reside on another server, if the server being extracted has ample disk space, it is faster to have the mailboxes extracted from the server to a local hard drive than to try to write the exported data across a LAN or WAN connection to another server. Remember, you might be extracting gigabytes of information, and it's a lot faster to extract the information to a local hard drive than over even a 100Mb Ethernet connection.

12. Choose to change the filenames during the extract process and to save the settings of the configuration information, or just leave them as the default and select Next to continue. A summary screen displays the results of the extraction, as shown in Figure 32.4.

    Figure 32.4. Extracting information from an Exchange database.

    

To import the ExMerge information to another server, the process is very similar. Do the following:

1. Launch the exmerge.exe program and click Next through the welcome screen.

2. Choose the Export or Import (Two-Step Procedure) option, and click Next.

3. Select Step 2 to import data into an Exchange server, and click on Next.

4. Enter the name of the server where the information will be imported, and then click Next.

5. Select the storage group into which the mailboxes will be imported; then select Next.

6. Choose the mailboxes that are to have data imported (hold down the Ctrl key to select individual mailboxes, or click on the Select All button to select all mailboxes to be extracted). Click on Next to continue.

7. Select the default locale or language set used for the mailbox (such as English US). Then click Next to continue.

8. Select the drive and subdirectory where the mailbox(es) should be imported from; then click Next.

   NOTE

   If the data currently resides on another server and there is a lot of information to import (more than 1GB), you might consider copying the files onto the hard drive of the server where the data will be imported from. This can drastically improve the import time because the file read is done on a message-by-message basis, and an XCOPY or Windows Explorer transfer of files is compressed.

9. Choose to change the filenames during the extraction process and to save the settings of the configuration information, or just leave them as default and select Next to continue. After completing, a summary screen displays the results of the import.

Running the ISINTEG and ESEUTIL Utilities

When a database is determined to be corrupt, usually an administrator is directed to run the built-in utilities on Exchange to run maintenance on the databases. The utilities are the ISINTEG ("eye-ess-in-tehg") and ESEUTIL ("ee-ess-ee-u-tihl"). However, depending on the condition of the database, a very corrupt database can take several hours to run, only to result in the loss of data. Some administrators are incorrectly told to never run the utilities because they will always result in loss of data. It's typically just a lack of knowledge of how the utilities work that leads to misunderstanding the potential results of the databases.

As noted in the previous two sections, there might be better options for recovering information from a corrupt database. Instead of trying to fix a known corrupt database, simply migrating the information off a server or extracting information from corrupt databases is frequently a better fix. However, if the determination is to run the utilities, a few things should be noted:

1. The ISINTEG utility is a high-level utility that checks the consistency of the database, validating the branches of the database that handle data, data directory tables, attachment objects, and the like. Fixing the database table makes way for a more intensive data integrity check of the database.

2. The ESEUTIL utility is a low-level utility that checks the data within the database. ESEUTIL does not differentiate between a corrupt section of the database and how that section impacts mailboxes or messages. So, when a complete repair is performed using ESEUTIL , entire mailboxes can be deleted or all attachments for the entire database can be eliminated to fix the corruption. This is why running ESEUTIL to repair a database is a function of last resort.

3. To run ISINTEG on a database takes around 1 hour for every 10GB being scanned for a moderately corrupt database. The repairs are done relatively quickly, and the database is ready for more extensive scanning.

4. Running ESEUTIL on a database takes anywhere from 1 hour for every 10GB to up to 1 hour per 1GB, depending on the level of repair being performed. It is not unreasonable to see a relatively corrupt 30GB database take more than 24 hours to complete the repair.

5. ISINTEG and ESEUTIL can be performed only offline, meaning that the Exchange server is offline during the process. Users cannot access their mailboxes during the ISINTEG and ESEUTIL processes. Thus, if it takes 2040 hours of downtime to complete the repair of a database, the Move Mailbox method that can be run without bringing servers offline is frequently a more palatable solution.

6. However, if run on a regular basis, the ISINTEG and ESEUTIL utilities can clean up an Exchange database before serious corruption occurs. Administrators who get scared off performing maintenance because of the potential threat of losing data could actually minimize their chance of data corruption if the utilities are run regularly. See Chapter 19 for recommended maintenance practices.

The common parameters used for the ISINTEG and ESEUTIL utilities are as follows . For regular maintenance such as checking the database structure's integrity and performing defragmentation of the database, the following commands should be run:

 isinteg -s SERVERNAME -test allfoldertests eseutil /d priv1.edb 

When run against an Exchange Server 2003 system, the ISINTEG utility produces a summary similar to the one in Figure 32.5.

Figure 32.5. Results from an ISINTEG utility run.

NOTE

The ISINTEG and ESEUTIL utilities typically reside in the \Program Files\Exchsrvr\Bin directory of the Exchange server. The databases that are typically specified in ESEUTIL are the priv1.edb and pub1.edb . However, if an organization has multiple database and storage groups, several database might need to be checked separately for integrity.

When a database needs to be repaired, eseutil /p priv1.edb can be run. Beware: The /p repair command is a brute-force repair and deletes sections of the database to make the integrity of the database clean. A message provides an additional warning about ESEUTIL , as shown in Figure 32.6. When running the /p command in ESEUTIL , entire sections of the database might be deleted to repair and recover the state of the database.

Figure 32.6. ESEUTIL warning when a database repair ( /p ) is run.