Disaster Planning

If you ever want to see a blank look, ask your average IT manager what would happen to their servers, or indeed their entire datacentre, if the building in which it was hosted burned down or was destroyed in a massive earthquake. An alarming percentage of them will have no answer.

Disaster (Recovery) Planning is a critical and frequently overlooked part of running a service, whether it is a web application or something else (including the day-to-day operations of your business). It is a collection of documents or procedures (that have been rehearsed) for dealing with the questions that arise when one of the following happens:

• Parts of or your entire data center are destroyed in a catastrophic event.

• Your development team goes out for lunch and is hit by a bus (leaving the team seriously injured or dead).

• Your corporate headquarters burns down.

• A network attacker or disgruntled employee manages to destroy all the data on the servers for your web applications.

While many people do not like to talk about disasters and attacks, the hard reality is that such things do occur. Fortunately, it is rare. However, businesses can often not afford the downtime that an event of such magnitude would cause if they were unprepared. A business that makes millions of dollars a day would be devastated if its web applications were shut down for over a week while people unfamiliar with the setup worked to get the systems up and running again.

By preparing for these events, anticipating them with clear plans of action, and rehearsing some of the more critical portions, a little financial investment can save the business from disastrous losses when the real problem strikes.

Some of the things we might do to help us with disaster planning and recovery include

• Make sure that all data is backed up daily and taken off site to another facility, so that even if our data center is destroyed, we still have the data elsewhere.

• Have off-site, handwritten scripts on how to re-create the server environments and set up the web application. Rehearse this re-creation at least once.

• Have a full copy of the source code necessary for our web application in multiple locations.

• For larger teams, prohibit all members of the team from traveling in one vehicle, so that the team will be less affected in the event of an accident.

• Have automated tools running to make sure that server operation is normal, and have a designated "emergency operator" who will be responsible for coming in during nonbusiness hours when a problem arises.

• Make arrangements with a hardware provider to have new hardware immediately available in the case that your data center is destroyed. It would be frustrating to wait 46 weeks for new servers.