BCP Recovery Plan Development

A complete Business Recovery Plan consists of several components that handle not only the continuation of critical business functions but also all the functions and resources that support those critical functions.

Emergency response

Emergency response teams must be identified for every possible type of disaster. These response teams need written procedures to keep critical business functions operating.

Written procedures are vital for two reasons. First, the people who perform critical functions may not be familiar with them: They may not be the same persons who perform them under usual circumstances (during a disaster, the people who ordinarily perform the function may be unavailable). Second, the procedures and processes for performing the critical functions during a disaster will probably be different than under normal conditions.

Damage assessment

When a disaster strikes, experts need to be called in to inspect the premises and determine the extent of the damage. Typically, you need experts who can assess building damage, as well as damage to any special equipment and machinery.

Depending upon the nature of the disaster, damage assessment may be performed in stages. A first assessment may be a quick walkthrough to look for obvious damage, followed by a more time-consuming and detailed assessment to look for problems that are not so easily found.

The purpose of damage assessments is to determine whether buildings and equipment can still be used, whether they can be used after some repairs, or whether they must be abandoned altogether.

Personnel safety

In any kind of disaster, the safety of personnel is the highest priority, ahead of buildings, equipment, computers, backup tapes, and so on. This is not only because of the intrinsic value of human life, but also because people - not physical assets - make the business run.

Personnel notification

The Continuity Plan must have some provisions for notifying all affected personnel that a disaster has occurred. Multiple methods for notifying key business continuity personnel are needed in cases in which public communications infrastructures are interrupted.

Not all disasters are obvious: A fire or broken water main is a local event, not a regional one. And, in an event like a tornado or flood, the state of the business is not necessarily clear to employees who live even a few miles away. Consequently, the organization needs a plan for communicating with employees, no matter what the situation.

Throughout a disaster and its recovery, management must be given regular status reports as well as updates on crucial tactical issues so that management can align resources to support critical business operations that function on a contingency basis. For instance, a manager of a corporate facilities department can loan equipment needed by critical departments so that they can keep functioning.

Backups and off-site storage

Things go wrong with hardware and software, resulting in wrecked or unreachable data. When it’s gone, it’s gone! This is why IT departments everywhere make copies of their critical data onto tapes or removable discs.

These backups must be performed regularly, usually once per day. The backup media must also be stored off-site in the event that the facility housing those systems is damaged. Having backup tapes in the data center is of little value if they’re destroyed along with their respective systems.

For systems with large amounts of data, that data must be well understood in order to determine what kinds of backups need to be performed (full, differential, incremental) and how frequently. The factors that need to be considered are

• The time that it takes to perform backups

• The effort required to restore data

• The procedures for restoring data from backups compared with other methods for re-creating the data

For example, you must consider whether restoring application software from backup tapes is faster than just installing them from release media? Also, if
a large part of the database is static, does it really need to be backed up every day?

Off-site storage of backup media and other materials (documentation, and so on) must be chosen carefully. Factors to consider include survivability of the off-site storage facility as well as the distance from the off-site facility to the data center, airports, and alternate processing sites. The facility needs to be close enough so that media retrieval doesn’t take too long ( how long depends upon the organization’s recovery needs), but not so close that the facility will be involved in the same natural disaster as the business.

 Tip   Some organizations have one or more databases that are so large that they literally can not (or, at any rate, do not) back them up to tape. Instead, they keep one or more replicated copies of the database on other computers in other cities. BCP planners need to consider this possibility when developing continuity plans.

 Instant Answer   The purpose of off-site media storage is to make up-to-date data available in the event that the primary data center is damaged.

Software escrow agreements

Your organization should consider software escrow agreements (wherein the software vendor sends a copy of its software code to a third-party escrow organization for safekeeping) with the software vendors whose applications support critical business functions. In the event that an insurmountable disaster (this could include bankruptcy) strikes the software vendor, your organization must be able to consider all options for the continued maintenance for those critical applications, including in-house support.

External communications

The Corporate Communications, External Affairs, and (if applicable) Investor Relations departments should all have plans in place for communicating the facts about a disaster to the press, customers, and public. Contingency plans for these functions are critical if the organization is to continue communicating to the outside world. Open communication during a disaster is vital so that customers, suppliers, and investors don’t panic because they don’t know the true extent of the disaster.

 Warning   Who says External Affairs is nonessential?

Suppose the headquarters building for a large company burns to the ground. (This is very unlikely in modern buildings, but stay with us.) All personnel escape unharmed. In fact, the organization is very well off because all the information in the building was duplicated and stored in an off-site facility. Nice work! However, the External Affairs department, which was housed in that building, loses everything. It takes two days to recover the capability of communicating to the outside world. However, because of this time lag, the company loses many of its customers, who feared the worst. This is an especially unfortunate and ironic circumstance because the company was actually in pretty good shape prior to the disaster.

The emergency communications plan needs to take into account the possibility that some corporate facilities or personnel may be unavailable. So even the data and procedures related to the communications plan need to be kept safe so that they’re available in any situation.

Utilities

Data-processing facilities that support time-critical business functions must keep running in the event of a power failure. Although every situation is different, the principle is not: The BCP team must determine for what period of time the data-processing facility must be able to continue operating without utility power. A power engineer can find out the length of power outages in your area and crunch the numbers to arrive at the mean time between outages. By using that information, as well as having an inventory of the data center’s equipment and environmental equipment, you can determine whether the organization needs an uninterruptible power supply (UPS) alone, or a UPS and an electric generator.

 Instant Answer   Uninterruptible power supplies (UPSes) and emergency electric generators are used to provide electric power during prolonged power outages.

 Remember   In a really long power outage (more than a day or two), it is also essential to have a plan for the replenishment of generator fuel.

Logistics and supplies

The BCP team needs to study every aspect of critical functions that must be made to continue in a disaster. Every resource that’s needed to sustain the critical operation must be identified and then considered against every possible disaster scenario to determine what special plans must be made. For instance, if a business operation relies upon a just-in-time shipment of materials for its operation and an earthquake has closed the region’s only highway
(or airport or sea/lake port), then alternative means for acquiring those materials must be determined in advance. Or, perhaps an emergency ration
of those materials needs to be stockpiled so that the business function can continue uninterrupted.

Fire and water protection

Many natural disasters disrupt public utilities, including water supplies or delivery. In the event that a disaster has interrupted water delivery, new problems arise. Your facility may not be allowed to operate without the means for fighting a fire, should one occur.

In many places, businesses could be ordered to close if they can’t prove that they can effectively fight a fire using other means, such as FM-200. Then again, if water supplies have been interrupted, you have other issues to contend with, such as drinking water and water for restrooms. Without water, you’re hosed!

Documentation

Any critical business function must be able to continue operating after a disaster strikes. An essential item for sustained operations includes all relevant documentation for every piece of equipment as well as every critical process and procedure that’s performed in a given location.

Don’t be lulled into taking for granted the emerging trend of hardware and software products not coming with any documentation. After all, many vendors deliver their documentation only over the Internet, or they charge extra for hard copy. But many types of disasters may disrupt Internet communications, thereby leaving an operation high and dry with no instructions on how to use and manage tools or applications.

At least one set of hard copy (or CD-ROM soft copy) of documentation should be stored at the same off-site storage facility that stores the organization’s backup tapes.

 Instant Answer   Continuity and recovery documentation must exist in hard copy in the event that it’s unavailable via technical means such as laptop computers.

Data processing continuity planning

Data processing facilities are so vital to businesses today that a lot of emphasis is placed on them. Generally this comes down to these variables: where and how the business will continue to sustain its data processing functions.

Because data centers are so expensive and time consuming to build, better business sense dictates having an alternate processing site available. The types of sites are

• Cold site: A cold site is basically an empty computer room with environmental facilities (UPS; heating, ventilation, and air conditioning [HVAC]; and so on) but no equipment. This is the least costly option, but more time is required to assume a workload because computers need to be brought in from somewhere and set up, and data and applications need to be loaded. Connectivity to other locations also needs to be installed.

• Warm site: A warm site is basically a cold site but with computers and communications links already in place. In order to take over production operations, the computers must be loaded with application software and business data.

• Hot site: Indisputably the most expensive option, a hot site is equipped with the same computers as the production system, with application changes, operating system changes, and even patches kept in sync with the live production system counterparts. Even business data is kept up-to-date at the hot site, with some sort of mirroring or transaction replication. Because they are trained on operating the organization’s business applications (and they have documentation), the staff there knows what to do to take over data processing operations at a moment’s notice.

• Reciprocal site: Your organization and another organization sign a reciprocal agreement that pledges to one another the availability of the other’s data center in the event of a disaster. Back in the day when data centers were rare, this was a common remedy, but it has fallen out of favor in recent years.

• Multiple data centers: Larger organizations can consider the option of running daily operations out of two or more regional data centers that are hundreds (or more) of miles apart. The advantage of this arrangement is that the organization doesn’t have to make arrangements with outside vendors for hot/warm/cold sites, and the organization’s staff is already onsite and familiar with business and computer operations.

 Instant Answer   A hot site provides the most rapid recovery ability, but it’s also the most expensive because of the effort that it takes to maintain its readiness.

Table 10-1 compares these options side by side.

Table 10-1: Data Processing Continuity Planning Site Comparison
Open table as spreadsheet

Feature	Hot Warm Site	Site	Cold Site	Multiple Data Centers
Cost	Highest	Medium	Low	No additional
Computer-equipped	Yes	Yes	No	Yes
Connectivity-equipped	Yes	Yes	No	Yes
Data-equipped	Yes	No	No	Yes
Staffed	Yes	No	No	Yes
Typical lead time to readiness	Minutes to hours	Hours to days	Days to weeks	Minutes to hours