Disaster Recovery

When content level recovery methods do not work or when you have a complete farm failure, disaster recovery methods should be used to restore content. Be aware that disaster recovery methods may cause a loss of data. For example, if you restore a content database from a previous night's backup, all content written to that database since that backup occurred will be lost. For this reason, always attempt a restore using a content recovery method first.

Note 

All native SharePoint Products tools must be backed up to a disk drive. You must use an additional backup program, such as Windows Server 2003 Backup or Restore, to move that content to tape.

Using Central Administration Backup and Restore

The new backup and restore tool for Windows SharePoint Services 3.0 and SharePoint Server 2007 is greatly improved over the last version. You now have the ability to restore entire farms, including the configuration database. The backup and restore tool even allows restoration to an alternate SQL Server instance to permit alternate server farm restoration and testing. Unfortunately, you still do not have the ability to back up to tape or schedule backups from the GUI. You access the GUI Backup interface from Central Administration > Operations > Backup And Restore > Perform A Backup, as shown in Figure 14-5. On opening the Perform A Backup Web page, you are presented with all of the available components that can be backed up.

Figure 14-5: To back up or restore SharePoint Products content, browse to Central Administration > Operations > Backup And Restore.

In the following discussion, the term items refers to Web applications, content databases, SSPs, or content indexes and their associated search databases.

Performing a Farm-Level Backup from the GUI

1. Select the item you would like to back up. To back up the entire farm, check the box next to Farm, as shown in Figure 14-6.

2. After selecting the entire farm, all components should be highlighted.

3. Select Continue To Backup Operations.

4. If this is the first backup of the farm, check Full for the backup type.

   Important 

   You cannot perform a differential backup of the farm until at least one full backup has occurred. You also cannot perform a differential backup after adding a content database or Web application. Whenever you add any of these components to the farm, you must run another full backup before a differential backup will be successful.

5. You must enter a location to back up to. You cannot back up to tape. The backup location should be in the same domain and shared. The following accounts should have write access to the NTFS directory and File Share:

   • The account authenticated to Central Administration

   • The Farm account (should be the Central Administration application pool identity)

   • The SQL Server instance account, if you are using SQL authentication

   • The SPTimer service account, which should be the farm account in most implementations

6. Select OK.

Figure 14-6: Select the component or set of components you want to back up.

After selecting OK, the job is scheduled with Owstimer.exe (SPtimer). If your job should fail for any reason, you must manually delete the Timer Job Definition. Failing to do so prevents any further backup or restore from succeeding, whether the backup was initiated from the GUI or the command line.

Deleting a Failed Backup/Restore Job

1. Open Central Administration > Operations > Global Configuration > Timer Job Definitions.

2. Select Backup/Restore in the Title column.

3. Click on the Delete button.

Restoring Items from an Existing Backup

1. Open Central Administration > Operations > Backup And Restore > Restore From Backup.

2. Enter the location where the backup exists. There must be an Spbrtoc.xml file in the root of this backup location, or the restore will fail.

3. Select the backup that contains the item you wish to restore, and select Continue Restore Process. If you use multiple backup locations for disk management and do not see the correct backup listed, you may also change the directory from this interface from the menu.

4. From the Select Component To Restore screen, select the component to restore, such as an individual content database or an entire Web application that includes the IIS Virtual Server and associated content database(s).

5. Select Continue Restore Process.

6. Select the Type of restore you want to perform. You have two choices, as shown in Figure 14-7:

   • Selecting New Configuration allows you to restore the Web application, content database, or both as a different name and thereby recover content. In addition, you can restore the content to a different SQL Server instance.

   • The option Same Configuration actually means Overwrite. It completely overwrites any selected components, such as Web applications and content databases, with the backed-up content.

7. Select OK.

Figure 14-7: You must select New Configuration or Same Configuration to continue. Selecting Same Configuration overwrites your existing content.

If you choose to restore content to a different SQL Server instance or restore content databases with a different name to the same SQL Server instance, you must associate the content database with a Web application to retrieve the content. Never associate the restored content databases with Web applications that hosted the original content. Site collections, which are stored in those content databases, are assigned a unique identifier (UID) when created. UID conflicts are caused when associating restored content databases with a Web application with which the original content database is still associated. Therefore, always associate content databases that are restored, with the new option, to a different Web application from the one with which they were originally associated.

Important 

The only time the configuration database is backed up is during a farm-level backup. The only way to restore the configuration database is by performing a farm-level restore.

Command-Line Farm-Level Backup and Restore

You can also back up your entire farm by using stsadm.exe -o backup -directory, also known as STSADM Catastrophic Mode. Using the -directory command-line option invokes additional options that you can use to back up the entire farm or individual items, such as Web applications or content databases. However, you cannot restore object-level items such as documents. Because you cannot schedule backup jobs from the Central Administration GUI, using the command line is the best way to reliably schedule backup jobs on your server farm. The following is an example of backing up an entire server farm using Stsadm.exe in Catastrophic Mode:

 stsadm.exe -o backup -directory \\backupservername\backups\ -backupmethod full 

This example is the simplest form of backing up your entire server using the Full option. There are several optional parameters you can define when backing up, such as -backupthreads, that are useful and you may use them as well.

The parameters available using stsadm.exe -o backup -directory are as follows:

• -item The -item option must be used in conjunction with the -showtree option. An item from the list generated by showtree must be specified and the command run again omitting the showtree parameter.

• -percentage The -percentage parameter shows the progress increments on screen. For example, 10 would show the progress in 10% increments.

• -backupthreads This value should be between 1 and 10. Be careful not to use high thread counts during content indexing or other system maintenance.

• -showtree The -showtree parameter shows all items that can be backed up. This information can also be found in Central Administration Backup And Restore.

• -quiet This option suppresses output and is used when scripting backups.

For example, the following command backs up a Web application named Portal and uses five CPU threads for the backup process:

 stsadm -o backup -directory \\backupservername\backups -backupmethod full -item "Portal" -backupthreads 5 -quiet 

Automating SharePoint Products and Technologies Backups

1. Create a batch file containing a tested command-line Catastrophic backup string, as shown previously.

2. Open Scheduled Tasks in Windows Server 2003 Control Panel.

3. Add a scheduled task, and click Next.

4. Select Browse and choose the batch file you created in step 1.

5. Name the Task; SharePoint Server backup, for example.

6. Select the frequency, usually Daily, and click Next.

7. Select the Start time. Be careful not to overlap with content indexing or other scheduled system maintenance.

8. Click Next.

9. Enter the username and password to execute the scheduled task. This user must have write access to the backup share destination and be a farm administrator.

10. Click Finish.

Restoring a Server Farm from a Farm-Level Backup

1. Find the backup ID either by running

         stsadm.exe -o backuphistory -directory      \\backupserver\sharename -restore 

   or by viewing the history in Central Administration > Operations > Backup And Restore > Backup And Restore History. Every backup or restore job is assigned a unique identifier.

2. From the 12 Hive\Bin directory, execute

         stsadm.exe -o restore -directory      \\backupservername\sharename -restoremethod [ new | overwrite ]      -item <item> 

   If you do not select the item, such as a Web application or content database, the entire farm is restored.

3. If prompted, you must enter the Web application pool identity and associated password for each component restored. Alternatively, you can define these in the script using the -username and -password command-line options.

4. After restoring components to a server farm, it is always a good idea to perform an IISReset on every server in the farm.

From the command line, you also have the option to select a new database server. For example, you would execute the following command to restore databases to a new SQL Server instance named DEV:

 stsadm.exe -o restore -directory \\backupservername\sharename -restoremethod new -backupid <id> -newdatabaserver DEV 

Command-Line Site Collection Backup and Restore

If you used the previous versions of SharePoint Products, you may be familiar with the stsadm.exe -o [backup | restore] commands. Without the -directory option, Stsadm.exe backs up and restores site collections instead of farm-level components. Although using Stsadm.exe to back up and restore site collections does indeed do a complete backup of these site collections, it does not back up any configuration database information, content indexes, or search databases. It is possible to back up an SSP's site collection with Stsadm.exe, but the restoration will not be complete. For this reason, only use stsadm.exe -o [backup | restore] to back up and restore standard site collections.

To back up a site collection using Stsadm.exe, execute the following from the 12 Hive\Bin directory:

 stsadm.exe -o backup -url <Site Collection URL> -filename <backup file name> 

The URL should include the site collection name-for example, http://portal.contoso.msft/sites/accounting-if accounting is a site collection. You can also include the -overwrite option if you wish to write to the same file repeatedly. This backup file can be restored to the same server farm or to another server farm.

To restore a site collection using Stsadm.exe, execute the following from the 12 Hive\Bin directory:

 stsadm.exe -o restore -url <Site Collection URL> -filename <backup file name> 

Although it is possible to restore this site collection with a different URL, thus creating a cloned site collection in the original Web application, doing so introduces the possibility of a site collection ID conflict. When a site collection is created, it is assigned a Unique ID in the server farm. You should remove or overwrite the original site collection if you need to restore the backup to the same Web application.

Restoring Application Server Roles to the Farm

There may be times when you need to restore individual physical servers to a farm. Remember that most application server configuration data are stored in the configuration database. So, if the server farm is intact and you have only lost a single server, restoration is usually a simple matter. This section provides an overview of restoring a single physical application server to a farm.

Restoring Index Servers

Content indexes cannot be restored with an Window Server 2003 operating system backup and restore. The only way to restore content indexes is by using SharePoint Products Backup And Restore to restore the entire Shared Services Provider (SSP) that managed a single content index. Therefore, if multiple indexes existed on an Index server, you would have to restore each SSP that managed each of those individual content indexes. As part of this restore process, the associated Search database is restored as well. If either the content index or search database fails during restore, you must reset the index and re-crawl all content sources before search and indexing will resume successfully.

Resetting and Re-crawling All Content Sources After a Failure

1. Open Shared Services Provider Administration.

2. Open Search Settings.

3. Select Reset All Crawled Content, as shown at the bottom of Figure 14-8.

4. Open Content Sources And Crawl Schedules.

5. From the Quick Launch, select Start All Crawls.

Figure 14-8: Resetting all crawled content does not automate the rebuilding of the content index.

If you use a third-party tool to back up your SharePoint Products implementation, it is still wise to use the native tools to back up your Shared Services Providers; doing so ensures the ability to restore your content indexes successfully. If your indexes are quite small, you can simply add a new index server to the farm, associate it with an SSP, and use the previous numbered list to re-crawl all content sources.

Restoring a Shared Services Provider and the Associated Content Index

1. Open Central Administration > Operations > Backup And Restore > Restore From Backup.

2. Select the backup location that contains the most recent backup of this SSP.

3. Select the desired existing backup, and choose Continue Restore Process from the menu.

4. Select the SSP that managed the content index you wish to restore.

5. Select Continue Restore Process from the menu.

6. Select either a New configuration or Same (overwrite) configuration. You will usually select the Same configuration.

   When selecting the Same configuration, you must enter the farm account's user-name and password and the SSP application pool identity username and password when using SQL Server authentication. Otherwise, the restore process uses your currently logged-on account for authentication.

7. Click OK.

Restoring Query Servers

Query servers that are dedicated to the service are updated automatically after a server or server farm failure if the following are true:

• They connect to the server farm configuration database.

• The original index location exists.

• The file share for propagation is configured correctly.

If dedicated Query servers do not function after a failure, the simplest way to restore them is to remove the server from the farm, re-add the server to the farm, and follow the steps in Chapter 13, "Scaling Out to a SharePoint Technologies Server Farm," to add a Query server to a farm.

Generic Application Servers

The settings for Document Conversion Launcher Services, Document Conversions Load Balancer Service, and Excel Calculation Services server roles are stored in the configuration database. Therefore, simply adding any of these application servers back to a farm will re-enable them as application servers. Alternatively, you can rebuild these servers and run the SharePoint Products and Technologies Configuration Wizard to add them back to the server farm. Remember to enable the correct services in Central Administration > Operations > Topology And Services > Services On Server after running the wizard.

Likewise, the Windows SharePoint Services Help Search server can be restored in much the same manner. You can always select the Windows SharePoint Services Help content in a farm-level restore, but you may alternatively add a new server to the farm and enable it to service this role from the Farm Topology Web Part.

Backing Up Other Required Content

In addition to backing up all SharePoint Products content, you must also back up information that is not stored in the configuration or content databases. The following items must be backed up independently of your SharePoint Products and Technologies backups.

Operating System Files

Using the native Windows Server 2003 Backup or Restore tool, you can back up all files on the server including the system state. Verify in your complete operating system backup that you include the following:

• System State Verify you are backing up the System State that includes the Certificate Store and system logs. These are required to restore a server to a functional state, should the server fail.

• 12 Hive You must also verify you are backing up the 12 Hive at \Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\. All customized site definitions and trace logs are located there.

• SharePoint Server Binaries Configuration items, such as your Search and Indexing noise word files and thesaurus files, are located in the \program files\Microsoft Office Servers\12.0\ directory.

• Inetpub Folder(s) By default, this folder is c:\inetpub\wwwroot. Wherever this folder resides, be sure to include it in your backups because the subdirectory \wss\VirtualDirectories includes all of your Web application specific information. For example, you may have modified the Web.config file in these directories to change authentication or allow a longer timeout for uploading large files.

IIS Metabase

You must back up the IIS Metabase on each Web front-end (WFE) server in the farm. The IIS Metabase includes such information as SSL certificates, assigned IP addresses, and multiple host headers. Because this information is not stored in the configuration database, it is not restored when using the native SharePoint Products backup and restore tools.

Backing Up the IIS Metabase

1. Open Internet Information Services (IIS) Manager from Start > All Programs, Administrative Tools.

2. Right-click on the server name and select All Tasks > Backup/Restore Configuration…, as shown in Figure 14-9.

3. Select Create Backup.

4. Enter an easy-to-remember configuration backup name, such as the server name plus the backup date. For example, use WFE0106222007 for a server named WFE01 and a backup date of June 22, 2007.

5. Optionally, select Encrypt Backup Using Password. This allows the backup to become "portable," thus giving you the ability to restore the IIS Metabase to any server. Choose a simple password, such as PASSWORD, that is documented and all members of the IT staff know. This password is primarily for portability, not for security.

Figure 14-9: Right-click on the IIS Server name to begin the backup and restore process.

Note 

If you wish to restore the IIS Metabase to another server, you must change or remove machine-specific information in the backup file. This process is explained in detail at http://www.microsoft.com/technet/prodtechnol/windowsserver2003/library/IIS/.mspx?mfr=true.

Backing Up and Restoring the IIS Metabase from the Command Line

1. Either on every WFE server in the farm or on a centralized backup server, create a batch file with the command:

    cscript.exe %systemroot%\system32\iisback.vbs /backup /s Web1 /u administrator /p password /b WeeklyBackup /v NEXT_VERSION /e backuppassword 

2. In the previous command, Web1 was the target server, administrator was the username, password was the username administrator's password, WeeklyBackup was the backup type, and NEXT_VERSION appended a version number, such as 2, to the backup name. Optionally, you may use the /e flag to include a backup password, thus making the backup portable.

3. Add as many lines to this batch file as necessary to back up all WFE servers in the farm. Each backup will reside in the respective WFE server's %systemroot%\system32\inetsrv\ directory.

4. Schedule this backup from the Windows Server 2003 Control Panel using Scheduled Tasks. Simply add a scheduled task by browsing to the batch file you just created and selecting the desired schedule. Verify that the username and password that will run the task have permissions both to back up the server farm and to have write access to each WFE server.