Lesson 2: Developing Recovery Strategies | MCSE Training Kit (Exam 70-225): Microsoft Exchange 2000 Server Design and Deployment (Pro-Certification)

You should dedicate a significant portion of your disaster recovery plan to detailed step-by-step descriptions of restore procedures. Consider these procedures an insurance policy if something goes wrong with your systems. In a worst-case scenario, you don’t want to find out how to best recover your system; you just want to restore it as quickly as possible. The best way to do this is to take the instructions from the disaster recovery plan and run through the list of appropriate steps one at a time until the system is back up and running. If you are working with instructions validated in a test lab on a monthly or quarterly basis, you can be confident that you can do the job successfully, which can reduce your stress level tremendously.

This lesson explains how to develop recovery strategies for the various disasters you might encounter with an Exchange 2000 server. You can read about the retention of deleted items in the Information Store, which enables you to recover messages, mailboxes, and even entire public folder trees without retrieving a backup. You can also learn about the best ways to restore servers and about restore techniques that may become your last resort if your backup sets are corrupted.

After this lesson, you will be able to

Develop procedures based on the retention of deleted items to recover data without backups
Explain on a conceptual basis how to restore Exchange 2000 servers and databases according to various possible worst-case scenarios
Assemble a disaster recovery toolkit for the system administrators of your Exchange 2000 organization

Estimated time to complete this lesson: 90 minutes

Understanding Deleted Items Retention

Deleted items retention is a feature of the Information Store service that allows you to keep deleted messages, public folders, and mailboxes in the stores for a specified period of time. Within that time frame, accidentally deleted items can be recovered quickly without the need to restore them from a backup (see Figure 11.4).

As shown in Figure 11.4, there are two different procedures for recovering deleted items. If you have specified an appropriate item retention time for each store in its Limits tab using the Keep Deleted Items For (Days) setting, users are able to restore deleted messages and public folders without administrative intervention. In Microsoft Outlook 2000, from the Tools menu, the user simply selects the Recover Deleted Items command. This displays the Recover Deleted Items From – Deleted Items window, where deleted items can be recovered. A typical deleted items retention time is 7 days.

Figure 11.4 - Recovering deleted items without backups

Note

Users of Internet mail clients, such as OWA, Internet Message Access Protocol 4 (IMAP4), or Post Office Protocol 3 (POP3) clients, are unable to recover deleted items from the server. However, they might be available in the client’s deleted items folder.

Mailbox stores also provide a Keep Deleted Mailboxes For (Days) setting that allows you to retain entire mailboxes in the store for a period of time after deletion. Mailboxes are marked as deleted when their user accounts are removed from the domain or when their accounts are mail-disabled using the Exchange Tasks Wizard in the Active Directory Users and Computers console. The default retention time for mailboxes is 30 days. Within this interval, you can reconnect mailboxes to the same or different user accounts using the Exchange System Manager snap-in. In the console tree, expand the storage group and mailbox store of the deleted mailbox, select the Mailboxes container underneath, and, in the contents pane, right-click the deleted mailbox and select Reconnect. In the Select A New User For This Mailbox dialog box, select the user account to which you want to assign this mailbox, and the mailbox with all its data is recovered. The user can work with the reconnected mailbox immediately.

It is important to note that these recovery scenarios only work as long as the deleted items are still in the store. If they are purged beforehand, you need to restore a backup to bring back the data. If you didn’t use Exmerge or another mailbox-level backup solution to copy each item individually to tape, restore the affected mailbox or public folder store to a test server, connect to it using Outlook 2000, extract the public folder or mailbox contents in a .pst file, and pass this file to the user. You may have to first recover the item on the test system if it was backed up in a deleted state. To make sure deleted items are on the tape, enable the Do Not Permanently Delete Mailboxes And Items Until The Store Has Been Backed Up setting for your mailbox and public folder stores.

Developing Disaster Recovery Procedures

There are two general disasters that you must take into consideration when developing recovery strategies: the server is destroyed and the store is destroyed. These disasters can happen independently of each other. For instance, a crashed operating system does not necessarily mean that the Exchange 2000 database files are affected. Likewise, a destroyed mailbox store does not mean users with mailboxes in other stores cannot access their messages. You thus need to address at least three critical situations: the server is destroyed, the store is destroyed, and both the server and store are destroyed. It’s possible to break down the three critical states into further subcategories. The first two subcategories relate to offline and online backups, as outlined in Lesson 1. The third subcategory is called corrupted or no backup available. This gives you nine disaster recovery situations to deal with (see Table 11.4). You should include detailed recovery instructions for each of these situations in your disaster recovery documentation. Furthermore, include disaster recovery procedures for domain controllers. Exchange 2000 Server cannot function if the Active Directory environment is destroyed. Ask your Windows 2000 administrators to provide this information.

Table 11.4 General Disaster Recovery Situations

	File-Based Backup	Online Backup	No Backup
Server and store destroyed	Verify that all data is indeed destroyed and perform a full disaster recovery to bring back the system.	Install Windows 2000 and Exchange 2000 and then restore the databases.	Install Windows 2000, try to preserve the existing Exchange 2000 database files, and install a new Exchange 2000 server. Use low-level utilities to fix the databases. Restore the databases to a recovery server, extract the data from there, and play it back into the production system.
Server destroyed but store okay	Do not perform a disaster recovery to avoid formatting the hard disks. Restore the operating system, preserve the existing database files, restore Exchange 2000 Server, and then restore the most recent database files.	It is not necessary to restore the online backup because the database files are still available. Install the operating system, preserve the existing database files, install Exchange 2000 Server, and then restore the most recent database files.	Same as with online backups. Install the operating system, preserve the existing database files, install Exchange 2000 Server, and then restore the most recent database files.
Server okay but store destroyed	Copy the corrupted store to a secure location and then restore the database files from backup. Use the Exchange System Manager snap-in to mount the store.	Copy corrupted database files to a secure location and then restore the database files from backup. You can automatically mount the store after the backup operation.	Copy the corrupted .edb. and .stm files to a secure location and delete these files from the working directory. Mount the store to generate new databases. Use low-level utilities to fix the databases. Restore the databases to a recovery server, extract the data from there, and play it back into the production system.

It is important to understand that you must develop and validate your disaster recovery procedures in a representative test lab that is not connected to your production environment. Most organizations have specific recovery requirements that you cannot cover with standard procedures or recommendations. For example, special server hardware may require you to install third-party drivers or configure certain software settings during server recovery. Another issue arises if the organization was upgraded from an earlier version of Exchange Server, in which case recovery servers require special preparations (discussed later). Only proven procedures are trustworthy. It is a good idea to obtain formal approval of your disaster recovery strategies from the project team and sponsor.

More Info: Correcting LegacyExchangeDN Values on Recovery Servers

Most Exchange 2000 Server directory objects have a LegacyExchangeDN attribute in the form of /O=<organization>/OU=<site>/CN=<container>/CN=<subcontainer>/CN=<object>, which is used to identify items in a way that is compatible with Microsoft Exchange Server 5.5. When you upgrade to Exchange 2000 Server, the LegacyExchangeDN attribute is derived from the existing organization and site names.

When you install a new recovery server in a test lab, on the other hand, the LegacyExchangeDN attribute contains the organization name and administrative group name of your test environment. You can match the organization name to the production environment, but the administrative group name is First Administrative Group, by default. Most likely, this name does not go with your production site name. If the LegacyExchangeDN attributes are different, you can restore the databases, but you cannot mount them. Unfortunately, renaming administrative groups in the Exchange System Manager snap-in does not change the LegacyExchangeDN attribute.

The best way to solve this naming issue is to install Exchange Server 5.5 using the original organization and site names on the recovery system first, and then upgrade the installation to Exchange 2000 Server. Another option is to install a first Exchange 2000 server in First Administrative Group, create an additional administrative group with the correct name, and then install the actual recovery server into this group. You can also use LDIFDE.EXE to change the LegacyExchangeDN attribute of all relevant objects in the configuration naming context in Active Directory, if you prefer to work at the most complicated level.

Restoring a Fully Destroyed Server with File-Based Backups

If your production system is fully destroyed, you either have to recover the server to a different physical computer or to the same server with a fresh set of hard disks. In any case, strive to match the hardware configuration to the original server. If your hardware is not identical, ensure that at least the disks have the same partitioning and capacity (or more). This greatly simplifies recovery. In fact, some professional backup solutions allow you to restore a server to identical hardware in an automated process simply by booting the machine from a floppy disk and following the instructions. However, keep in mind that automated disaster recoveries tend to format hard disks. If you are not sure whether the disks that stored your messaging databases are still intact, do not use this option; otherwise, you might destroy valuable data with your undertaking.

If the hardware configuration of your recovery server doesn’t match the original machine or if you want to avoid the side effects of an automated disaster recovery, install a minimal version of Windows 2000 Server—you only need enough functionality to copy any databases to a secure location. After that, launch the Windows 2000 Backup utility (or install and use a third-party solution) to restore the actual production server. For more information, see the Windows 2000 product documentation.

Restoring a Fully Destroyed Server with Online Backups

If a file-based backup is not available, you need to reinstall the operating system manually. It is important to use the same version of Windows 2000 Server that was previously installed, such as Microsoft Windows 2000 Advanced Server, and apply any service packs—at a minimum, Windows 2000 Service Pack 1. Make sure you specify the original system drives, directories, and the old server name during installation. It is vital that you set up the same Windows 2000 components that were previously installed. The only difference is that the reinstalled server is a member of a workgroup instead of the production domain.

If you do not have a valid backup of the computer’s system state, you will need to add the new server to the production domain. If you do have a valid backup of the computer’s system state, do not add the new server to the production domain to avoid changes to the Active Directory configuration. You only need to log on to the new server, launch the Windows 2000 Backup utility (or third-party software), and restore the original system state information. Restoring the system state turns the new server into a member of the production domain and brings back the IIS metabase and other information (see Figure 11.5). Reboot the server for the changes to take effect.

Figure 11.5 - Joining a domain by restoring the system state

When the system is rebooted, you are greeted with an error message that one or more services could not be started, because Exchange 2000 Server has not been installed yet, but the corresponding services are referenced in the system configuration. If messaging databases still exist on the server’s disks, move them to a safe location. It’s always a good idea to preserve old databases. Your backup sets may turn out useless, in which case you may need to fix corrupted databases using low-level utilities, as explained later.

When you are ready to install Exchange 2000 Server, run the Setup program in /DisasterRecovery mode. It is important to note that disaster recovery requires that the old server object still exists in the organization’s configuration in Active Directory. Setup reads this information to restore the previous settings on the server, such as configuration of mailbox and public stores. When running Setup /DisasterRecovery, make sure all the components that were previously installed on the server are marked for disaster recovery on the Component Selection wizard screen.

On the other hand, if the original computer’s system state or Active Directory information was lost for some reason (for instance, because an administrator removed the server object from the Exchange organization), you cannot use Setup /DisasterRecovery. In this case, install Exchange 2000 Server the normal way and configure the system manually. It is important that you install the server in the same administrative group as the original machine and match the configuration of storage groups, mailbox stores, and public folder stores.

Restoring Databases on a Running Exchange 2000 Server with File-Based Backups

You now have reached the point where Exchange 2000 Server is running again, but the mailbox stores are still unavailable and must be recovered. If you have saved any existing database files from the server’s hard disks, as mentioned earlier, attempt to restore these first. This corresponds to a file-based backup. Make sure that the database directories, such as \Program Files\Exchsrvr\Mdbdata, are empty, and then copy the old .edb and .stm files into the correct locations. Make sure you restore .edb and .stm files together; otherwise, you cannot mount the store using the Exchange System Manager snap-in. This process may also fail for databases that originally came from a different server. In this case, right-click the problematic stores, select Properties, click the Database tab, and select the This Database Can Be Overwritten By A Restore check box. Click OK, then mount the stores and verify that the system is operational.

Restoring Databases on a Running Exchange 2000 Server with Online Backups

If the most recent database files were destroyed with the original server, but you have the most recent online backup at hand, use this backup to restore the databases. This is a straightforward process. Verify that the Exchange 2000 services are running and that the problematic stores are dismounted—this is most likely the case. Launch your backup software, such as the Windows 2000 Backup program, and restore the database files. During the restore, you need to specify a temporary location where the backup program will place transaction log and patch files. To avoid conflicts with production databases, never restore transaction log files to the original database locations. You can mount the databases automatically after the last backup set.

Note

To restore Exchange 2000 databases, you need to have the permissions of an Administrator or Backup Operator.

When restoring a normal online backup with incremental or differential backups, make sure you restore the normal backup first. Do not select the Last Backup Set check box before all incremental or differential backups have been restored. Selecting this check box triggers a process known as hard recovery, which applies the old transaction log and patch files to the restored databases. If you trigger the hard recovery too early, you have to start with the normal backup again, followed by all incremental or differential backups. Depending on the number of transaction log files that ESE must apply to the databases, the hard recovery can take a significant amount of time.

However, if you forget to select the Last Backup Set check box for your last backup, the databases cannot be mounted. This is not a critical problem. Simply restore the last backup set again with the Last Backup Set check box activated to resolve it. Alternatively, you can run a hard recovery manually by typing the eseutil /cc command from the temporary folder of the transaction log files.

The Information Store of Exchange 2000 Server allows you to restore individual databases while all other stores are mounted. However, do not restore databases from the same storage group sequentially to the same temporary folder without running hard recovery at the last backup set of each restore cycle. Otherwise, subsequent restores will overwrite the hard recovery files from previous rounds, thus preventing all databases recovered earlier from being mountable. If you restore multiple storage groups simultaneously, subfolders are created in the temporary folder automatically for each group.

More Info: Restoring KMS or SRS Databases from an Online Backup

It is possible to restore the databases of the KMS or SRS online, similar to the procedure described for the Information Store. However, before you can begin the restore operation, you need to bring the existing databases into a dismounted state. Unfortunately, it is not possible to dismount KMS or SRS databases in the Exchange System Manager snap-in. Stop the KMS or SRS service, move all existing database files from the work directory \Program Files\Exchsrvr\Kmsdata or \Srsdata to a safe location, and then start the corresponding service again. The service still starts, but in semi-running mode, allowing you to restore the databases using the online backup.

Restoring Databases Using a Corrupted Backup Set

All backup sets might turn out to be corrupted—for instance, if your backup operator saved the databases "online" using a file-based backup with the option of including opened files. Unfortunately, you most likely will lose data. The good news is that there is hope that you can bring back some data if at least a corrupted database is still available. Exchange 2000 Server comes with two powerful low-level utilities that can fix database corruption to a certain extent: ESEUTIL.EXE and ISINTEG.EXE. To fix database corruption, use ESEUTIL.EXE with the /p switch, but always back up all database files beforehand. You can read more about ESEUTIL and ISINTEG in the MicrosoftExchange 2000 Server Resource Kit.

Note

If you have to work with ESEUTIL.EXE and ISINTEG.EXE to fix database corruption, you should contact Microsoft Product Support Services for assistance.

Be forewarned that the repair process proceeds slowly, because ESEUTIL must analyze each page in the database to decide whether it can be fixed or must be purged. ESEUTIL’s performance slows even further if the temporary files are placed in a directory on another disk. Eventually, however, the database is returned to a consistent state or the process fails, in which case it might make sense to repeat the process until ESEUTIL is able to complete successfully. If it is hopeless, restore another older database version and repeat the repair process. Keep in mind that ESEUTIL purges corrupted data. Returned to a consistent state, the database may be missing internal structures that the Information Store needs to operate accurately. After running ESEUTIL.EXE, you must check the database at the Information Store level using ISINTEG.EXE, which also takes time. It can take up to several days to fix a corrupted database. As a quick stopgap to bring back the affected store as soon as possible, remove the corrupted database files from the working directory of the Information Store and mount the affected mailbox store again. The Information Store generates new, empty, and consistent database files. Your users will be able to send and receive messages; only the old data is missing. Nevertheless, work can continue while you are busy with database repairs.

It is important to understand that fixed databases should never be considered fully functional data repositories. They are not dependable because it is uncertain whether ESEUTIL and ISINTEG were able to preserve the internal structures so that the Information Store can function reliably. You run the risk of Information Store crashes possibly corrupting further stores on your server. For this reason, never restore fixed databases back into a production system. Restore them to a recovery server, reconnect the mailboxes to dummy accounts in the test environment, extract all recovered messages into .pst files using Exmerge, and then use Exmerge to play back these messages into the production system. For detailed information on how to do this, refer to the MCSE Training Kit—Microsoft Exchange 2000 Server Implementation and Administration.

Assembling a Disaster Recovery Toolkit

The advantages of developing disaster recovery procedures in a test lab are twofold. First, your project team develops procedures that truly work in your specific environment; second, you can determine exactly what hardware, software, and configuration settings are required to successfully restore a server. You must include all of these prerequisites in your disaster recovery toolkit, such as installation CDs, floppies containing third-party drivers, configuration sheets, and so forth.

Your disaster recovery toolkit should contain the following items:

A copy of the original Windows 2000 Server, Exchange 2000 Server, and Resource Kit product CDs
A CD or shared folder with Windows 2000 Service Packs, Exchange Service Packs, nonstandard drivers, and backup agents
A file containing exported recipient information in a supported LDIFDE or CSVDE format. You can find more information about the export and import of directory information via LDIFDE or CSVDE in Chapter 4, "Assessing the Current Messaging Infrastructure."
An emergency repair disk
Backups of recent system state information, which covers Active Directory, Registry, IIS metabase, and data from other system components, such as Certificate Services
Descriptions of hard drive partitions and their purposes (system drive, transaction log drives, database drives, and so forth)
Documentation about the configuration of hardware and the operating system and a copy of any other software, such as service packs or configuration changes applied to the system
Information about the configuration of Exchange 2000 Server, including installation directories, installed components, configured virtual servers and protocol settings, connector configurations, and so forth
Online backups of Exchange 2000 Server databases (Information Store, KMS, and SRS)
Recent file-based server backups, including the files of the operating system, binary files of Exchange 2000 Server, and the databases (Microsoft Mail Dirsync, message transfer agent [MTA] message queues, and so on)

Assembling a Disaster Recovery Toolkit for Proseware, Inc.

Proseware, Inc. is using an arrangement of SLR-based library systems, each with 4 TB of capacity and up to 144 GB per hour transfer rates, to back up the mailbox stores of its Exchange 2000 Server organization. "We support a very large number of Internet users in all possible time zones," explains Guy Gilbert, Head of IT Operations. "Our Exchange 2000 servers must be available 24 hours per day, 7 days per week. However, with as many servers in place as there are in our organization, the likelihood of disasters is very high. The more systems there are, the more can go wrong. Disaster recovery has a mission-critical priority in our enterprise. For this reason, we have deployed a dedicated nonproduction test lab for Lane Sacksteder and her squad of backup operators."

Guy Gilbert developed the following disaster recovery strategy for Proseware, Inc.:

Our Internet users work with OWA, which doesn’t allow the user to recover deleted items, nor do we intend to provide our users with this kind of service. We will therefore not enable the deleted items retention feature, but we will keep deleted mailboxes for 30 days. If a user leaves us and comes back within a month, we can give him or her the old mailbox with all the data.
We use high-performance computers equipped with very specific hardware that require a number of drivers not included in the Windows 2000 Datacenter Server product. Lane Sacksteder must include these drivers as well as all other required components and software in our disaster recovery toolkit. As soon as this toolkit is ready, we will perform a test recovery to ensure that nothing was overlooked.
To maximize our flexibility in disaster recovery, we will perform file-based backups of the operating system on a monthly basis and include this backup in our toolkit. However, to avoid shutting down the servers, our file-based backups will exclude the Exchange 2000 database files. Nevertheless, all offline and online backups will include the server’s system configuration. We also need to provide backups of our domain controllers, including their system state.
Based on file-based and online backups, Lane Sacksteder will develop detailed disaster recovery procedures to address the various possible scenarios. To be prepared for all imaginable situations, she also must test the database recovery using ESEUTIL and ISINTEG.

Activity: Developing Disaster Recovery Strategies

In this activity, you will develop a disaster recovery strategy for Woodgrove Bank’s Exchange 2000 environment in Switzerland. You already developed a backup plan for Woodgrove Bank during the activity of Lesson 1.

Tip

You can use Figure B.30 in Appendix B as a guideline to accomplish this activity.

Scenario: Woodgrove Bank

Woodgrove Bank operates a clustered Exchange 2000 server named ZUR-01-EX in Zurich that stores the mailboxes of all users in Switzerland. The server is configured with three mailbox stores for Basel, Bern, and Zurich. The bank relies on a 70-GB DLT drive with a transfer rate of approximately 30 GB per hour to perform daily full online backups. "Our users enjoy working in an Exchange 2000 organization. They make full use of the messaging environment and often send large messages with attachments of several megabytes," states Luis Bonifaz, Chief Information Officer. "When a user accidentally deletes an important message, I want the user to be able to recover the item to offload this job from the shoulders of my system administrators."

It is your task to outline an appropriate disaster recovery strategy for Woodgrove Bank:

Woodgrove Bank wants to enable its users to recover deleted items within 2 weeks of their deletion. What do you have to configure to enable this recovery strategy?
Woodgrove Bank does not want to perform file-based backups on a regular basis. Online backups are prepared daily and the system state is included in these backups. How does this backup strategy affect the disaster recovery options of Woodgrove Bank?

Lesson Summary

Exchange 2000 Server provides several features that can help to mitigate the risk of serious problems with user data. Accidental message, public folder, or even mailbox deletions, for instance, are not disastrous if you have enabled the deleted item retention feature. By default, deleted mailboxes are retained for 30 days. Deleted messages and public folders, on the other hand, are purged immediately by default, but you can also keep them in the store for a certain period of time if you specify a deleted item retention time in the store properties. Within that time frame, users can restore accidentally deleted items using Outlook 2000. Deleted mailboxes, on the other hand, can be reconnected to user accounts in Active Directory.

Deleted item retention cannot help you if a real disaster happens, which might entirely destroy your server or a mailbox store. If you have a file-based backup of the entire system available, you may be able to restore an entire server quickly. Remember, however, that a complete disaster recovery might not be required if only the operating system or the binary files of Exchange 2000 Server had the problem, or if only a single mailbox store was affected. If only the operating system is damaged, it is sufficient to reinstall Windows 2000 with Service Pack 1, restore the system state, and run the Setup program of Exchange 2000 Server in /DisasterRecovery mode. On the other hand, if the databases are damaged, you need to restore them using an offline or online backup—whichever contains the most recent data. If a backup set is corrupted, use the previous day’s backup. If all backup sets are useless, consider recovering the databases using the low-level utilities ESEUTIL and ISINTEG.