Testing Disaster Recovery Plans

Remember, you have not done a backup until you test the restore. Similarly, you do not have a disaster recovery plan until you test it. Why would you spend time developing a plan if you know it will never be tested ? Equate testing to holding a fire drill. Most municipalities around the world insist that all office buildings practice fire drills to ensure that people can get out and that all systems are working. Protecting your SLAs, as well as equipment and human life, is just as vital to your business. A successful test will give you confidence in your plan.

Every company that cares about disaster recovery needs to have a simulated recovery drill at least once a year, preferably more often. This ensures that everyone knows his or her role and responsibilities, and can execute the plan successfully. If plans have not been proactively updated as they should have been and not everything is documented, the mock drill is the time to find out, not during a real disaster.

A drill should start by simulating a system crash or that the primary site has become physically inaccessible. For example, assume that there has been a major accident at a chemical plant and your offices have been completely evacuated. At that point, the team is assembled , responsibilities are handed out, and the team sets out to the recovery site to put the system back together from the ground up. Disaster recovery could also be as simple as rebuilding a database locally.

Figuring out the logistics for the test is the complicated part. For example, is it really a valid test if everyone knows the exact time and date in advance? If you do not want it to be a complete surprise, you might give a vague time. The problem with publishing a time and date is that people can prepare. You must also notify users, management, and vendors if they will be affected or involved in the test. You should even go so far as to book flights and hotels, if necessary. In all regards, this should feel and act like the real thing.

During the drill, make notes about what is working and what is not, or where things are inaccurate. After the drill comes arguably one of the most important activities, the postmortem. This postrecovery meeting assesses what was good and what was bad. This feedback then needs to be incorporated back into the disaster recovery plan and the run book, both of which should be updated immediately so that whoever executes the plan in the future benefits from the experience, even if they were not part of the drill.