11.1 Using Backups to Protect Your Data

only for RuBoard - do not distribute or recompile

Backups are copies that you make of information that you hope you will never need. A backup can be a simple copy of a file that you put on a Zip disk and put away in the top drawer of your desk for safekeeping. If the original file is inadvertently deleted or corrupted, the backup can be retrieved after the damage is noticed.

Backups can be very simple, like the Zip disk in your desk drawer, or they can be exceedingly complex. For example, many backup systems will let you copy every file on your computer onto a 30-gigabyte magnetic tape and create a special "restore floppy." In the event that your computer is lost or stolen, you can buy a new computer and a tape drive, put the tape into the tape drive, and boot the computer from the floppy disk; the backup system will automatically restore all of your files and applications to the newly-purchased computer.

11.1.1 Make Backups!

Bugs, accidents, natural disasters, and attacks on your system cannot be predicted. Often, despite your best efforts, they can't be prevented. But if you have good backups, you at least won't lose your data and in many cases, you'll be able to restore your system to a stable state. Even if you lose your entire computer to fire, for instance with a good set of backups you can restore the information after you purchase or borrow a replacement machine. Insurance can cover the cost of a new CPU and disk drive, but your data is something that in many cases can never be replaced.

Backups and RAID

One of the best ways to protect your data from the failure of a disk is to store all of your data on two or more drives. A few years ago, storing data on multiple spinning platters was a strategy that only well-funded data centers could pursue. But now, thanks to the ever-plummeting cost of storage combined with widespread adoption of "best practices" standards, RAID (Redundant Arrays of Inexpensive Disk) systems have become commonplace on servers and are becoming standard on workstations as well. Systems based on RAID were pioneered in the 1980s. Today, RAID systems are available in four common versions:

RAID 0 (striping): These systems use two or more hard disks to simulate a single drive that has twice the capacity and twice the speed. The systems work by storing alternating blocks of data on the first drive and the second drive. RAID 0 systems actually decrease the reliability of the disk drive subsystem, because the data stored on the disk becomes unusable if either drive fails.
RAID 1 (mirroring): These systems store the same data on both drives. Data is written to both drives simultaneously; read operations can be serviced by either drive. As a result, RAID 1 systems tend to have significantly improved performance when compared with a single-drive system.
RAID 5 (parity): These systems store N drives worth of data on N+1 drives, using mathematical functions so that the data can be recovered in the event that any drive fails. That is, three drives can be used to store two drives worth of data, or six drives can be used to store five drives worth of data. RAID 5 offers a lower cost per byte than RAID 1, but it has significantly decreased performance.
RAID 0+1: These systems use four disks configured into two mirror sets of two striped drives. These systems thus use four drives to store two disks worth of data, providing the redundancy of RAID 1 with the speed improvement of RAID 0. This is the most expensive form of RAID.

If you have a RAID system, you may be tempted not to do backups at all: after all, the redundant drives protect against hardware failure. But RAID does not protect against accidental file deletion, theft, or fire. For this reason, even if you have a RAID system, you should still perform regular backups.

11.1.2 Why Make Backups?

Backups are important only if you value the work that you do on your computer. If you use your computer as a paperweight, then you don't need to make backups. Years ago, making daily backups was a common practice because computer hardware would often fail for no obvious reason. A backup was the only protection against data loss. Today, hardware failure is still a good reason to back up your system. Hard disk failures are a random process: even though a typical hard disk will now last for five years or more, an organization that has 20 or 30 hard disks can expect a significant drive failure every few months. Drives frequently fail without warning sometimes only a few days after they have been put into service. It's prudent, therefore, to back up your computer after you install its operating system and your basic load of applications. Not only will this first backup allow you to analyze your system after an attack to see what has been modified, but it will also save the time of rebuilding your system from scratch in the event of a hardware failure.

Backups are important for a number of other reasons as well:

Archival information: Backups provide archival information that lets you compare current versions of software and databases with older ones. This capability lets you determine what you've changed intentionally or by accident. It also provides an invaluable resource if you ever need to go back and reconstruct the history of a project, either as an academic exercise or to provide evidence in a court case. Being able to review multiple backups to determine how a document changed over time, when it changed, or who changed it, is probably the most important use of backups.
User error: Users especially novice users accidentally delete their files. With graphical user interfaces, it's all too easy to accidentally drag one folder on top of another with the same name. Making periodic backups makes it possible to restore files that are accidentally deleted, protecting users from their own mistakes. Mistakes aren't limited to novices, either. More than one expert has accidentally overwritten a file by issuing an incorrect editor or compiler command, or accidentally reformatting a Unix file system by typing newfs/dev/ad0c instead of newfs/dev/da0c.
System-staff error: Sometimes your system staff may make a mistake. For example, a system administrator deleting old accounts might accidentally delete an active one.
Hardware failure: Hardware breaks from time to time, often destroying data in the process: disk crashes are not unheard of. If you have a backup, you can restore the data on a different computer system.
Software failure: Many application programs, including Microsoft Word, Excel, and Access, have been known to occasionally corrupt their data files. If you have a backup and your application program suddenly deletes half of your 500 x 500-cell spreadsheet, you will be able to recover your data.
Electronic break-ins and vandalism: Computer attackers and malicious viruses frequently alter or delete data. Your backups may prove invaluable in recovering from a break-in or a virus incident.
Theft: Computers are easy to steal and all too easy to sell. Cash from your insurance company can buy you a new computer, but it can't bring back your data. Not only should you make a backup, but you should also take it out of your computer and store it in a safe place there are all too many cases of tape drives whose backups were stolen along with the computer system.
Natural disaster: Sometimes rain falls and buildings are washed away. Sometimes the earth shakes and buildings are demolished. Fires are also very effective at destroying the places where we keep our computers. Mother Nature is inventive and not always kind. As with theft, your insurance company can buy you a new computer, but it can't bring back your data.
Other disasters: Sometimes Mother Nature isn't to blame: truck bombs explode; gas pipes leak and cause explosions; and coffee spills through ventilation holes. We even know of one instance in which EPA inspectors came into a building and found asbestos in the A/C ducts, so they forced everyone to leave within 10 minutes, and then sealed the building for several months!

With all of these different uses for backups, it's not surprising that there are so many different forms of backups in use today. Here are just a few:

Copy your critical files to a floppy disk or a high-density removable magnetic or optical disk.
Copy your disk to a spare or "mirror" disk.
Make periodic Zip or "tar" archives of your important files. You can keep these backups on your primary system or you can copy them to another computer, possibly at a different location.
Make backups onto magnetic or optical tape.
Back up your files over a network or over the Internet to another computer that you own, or to an Internet backup service. Some of these services can be exceedingly sophisticated. For example, the services can examine the MD5 checksums of your files and only back up files that are "unique." Thus, if you have a thousand computers, each with a copy of Microsoft Office, none of those application files need to be copied over the network in order to add them to the backup.

11.1.3 What Should You Back Up?

There are two approaches to computer backup systems:

Back up everything that is unique to your system user accounts, data files, and important system directories that have been customized for your computer. This approach saves tape and decreases the amount of time that a backup takes; in the event of a system failure, you recover by reinstalling your computer's operating system, reloading all of the applications, and then restoring your backup tapes.
Back up everything, because restoring a complete system is easier than restoring an incomplete one, and tape/CD-ROM is cheap.

We recommend the second approach. While some of the information you back up is already "backed up" on the original distribution disks or tapes you used to load the system onto your hard disk, distribution disks or tapes sometimes get lost. Furthermore, as your system ages, programs get installed in the operating system's reserved directories as security holes get discovered and patched, and as other changes occur. If you've ever tried to restore your system after a disaster,^[1] you know how much easier the process is when everything is in the same place.

^[1] Imagine having to reapply 75 vendor "jumbo patches" or "hot fixes" by hand, plus all the little security patches you got off the Net and derived from this book, plus all the tweaks to optimize performance and imagine doing this for each system you manage. Ouch!

For this reason, we recommend that you store everything from your system (and that means everything necessary to reinstall the system from scratch every last file) onto backup media at regular, predefined intervals. How often you do this depends on the speed of your backup equipment and the amount of storage space allocated for backups. You might want to do a total backup once a week, or you might want to do it only twice a year. But please do it!

11.1.4 Types of Backups

There are three basic types of backups:

Level-zero backup: Makes a copy of your original system. When your system is first installed, before people have started to use it, back up every file and program on the system. Such a backup can be invaluable after a break-in.^[2]

^[2] We recommend that you also do such a backup immediately after you restore your system after recovering from a break-in. Even if you have left a hole open and the intruder returns, you'll save a lot of time if you are able to fix the hole in the backup, rather than starting from scratch again.
Full backup: Makes a copy to the backup device of every file on your computer. This method is similar to a day-zero backup, except that you do it on a regular basis.
Incremental backup: Makes a copy to the backup device of only those items in a filesystem that have been modified after a particular event (such as the application of a vendor patch) or date (such as the date of the last full backup).

Full backups and incremental backups work together. A common backup strategy is:

Make a full backup on the first day of every other week.
Make an incremental backup every evening of everything that has been modified since the last full backup.

Most administrators of large systems plan and store their backups by disk drive or disk partition. Different partitions usually require different backup strategies. Some partitions, such as your system partition (if it is separate), should probably be backed up whenever you make a change to them, on the theory that every change that you make to them is too important to lose. You should use full backups with these systems, rather than incremental backups, because they are only usable in their entirety.

On the other hand, partitions that are used for keeping user files are more amenable to incremental backups. Partitions that are used solely for storing application programs really only need to be backed up when new programs are installed or when the configuration of existing programs is changed.

When you make incremental backups, use a rotating set of backup tapes.^[3] The backup you do tonight shouldn't write over the tape you used for your backup last night. Otherwise, if your computer crashes in the middle of tonight's backup, you would lose the data on the disk, the data in tonight's backup (because it is incomplete), and the data in last night's backup (because you partially overwrote it with tonight's backup). Ideally, perform an incremental backup once a night, and have a different tape for every night of the week, as shown in Figure 11-1.

^[3] Of course, all types also rotate around a spindle. "Rotating" means that the tapes are rotated with each other according to a schedule.

Figure 11-1. An incremental backup

11.1.5 Guarding Against Media Failure

You can use two distinct sets of backup tapes to create a tandem backup. With this backup strategy, you create two complete backups (call them A and B) on successive backup occasions. Then, when you perform your first incremental backup, the "A incremental," you back up all of the files that were created or modified after the last A backup, even if they are on the B backup. The second time you perform an incremental backup, the "B incremental," you write out all of the files that were created or modified since the last B backup (even if they are on the A incremental backup.) This system protects you against media failure, because every file is backed up in two locations. It does, however, double the amount of time that you will spend performing backups.

Some kinds of tapes in particular, 4mm or 8mm video tape and Digital Audio Tape (DAT) cannot be reused repeatedly without degrading the quality of the backup. If you use the same tape cartridge for more than a fixed number of backups (usually, 50 or 100), you should get a new one. Be certain to see what the vendor recommends and don't push that limit. The few pennies you may save by using a tape beyond its useful range will not offset the cost of a major loss.

Try to restore a few files chosen at random from your backups each time, to make sure that your equipment and software are functioning properly. Stories abound about computer centers that have lost disk drives and gone to their backup tapes, only to find them all unreadable. This scenario can occur as a result of bad tapes, improper backup procedures, faulty software, operator error (see the sidebar), or other problems.

At least once a year, you should attempt to restore your entire system completely from backups to ensure that your entire backup system is working properly. Starting with a different, unconfigured computer, see if you can restore all of your tapes and get the new computer operational. Sometimes you will discover that some critical file is missing from your backup tapes. These practice trials are the best times to discover a problem and fix it.

A Classic Case of Backup Horror

Sometimes, the weakest link in the backup chain is the human responsible for making the backup. Even when everything is automated and requires little thought, things can go badly awry. The following was presented to one of the authors as a true story. The names and agency have been omitted for obvious reasons.

It seems that a government agency had hired a new night operator to do the backups of their Unix systems. The operator indicated that she had prior computer operations experience. Even if she hadn't, that was okay little was needed in this job because the backup was largely the result of an automated script. All the operator had to do was log in at the terminal in the machine room located next to the tape cabinet, start up a command script, and follow the directions. The large disk array would then be backed up with the correct options.

All went fine for several months, until one morning, the system administrator met the operator leaving. She was asked how the job was going. "Fine," she replied. Then the system administrator asked if she needed some extra tapes to go with the tapes she was using every night he noticed that the disks were getting nearer to full capacity as they approached the end of the fiscal year. He was met by a blank stare and the chilling reply, "What tapes?"

Further investigation revealed that the operator didn't know she was responsible for selecting tapes from the cabinet and mounting them. When she started the command file (using the Unix dump program), it would pause while mapping the sectors on disk that it needed to write to tape. She would wait a few minutes, see no message, and assume that the backup was proceeding. She would then retire to the lounge to read.

Meanwhile, the tape program would, after some time, begin prompting the operator to mount a tape and press the return key. No tape was forthcoming, however, and the mandatory security software installed on the system logged out the terminal and cleared the screen after 60 minutes of no typing. The operator would come back some hours later and see no error messages of any kind.

The panicked supervisor immediately started day-zero dumps of all the computer's disks. Fortunately, the system didn't crash during the process. Procedures were changed, and the operator was given more complete training.

How do you know if the people doing your backups are doing them correctly?

A related exercise that can prove valuable is to pick a file at random, once a week or once a month, and try to restore it. Not only will this reveal if the backups are comprehensive, but the exercise of doing the restoration may also provide some insight.

We have heard many stories about how the tape drive used to make the backup tapes had a speed or alignment problem. Such a problem results in the tapes being readable by the drive that made them, but unreadable on every other tape drive in the world! Be sure that you try loading your tapes on other drives when you check them.

11.1.6 How Long Should You Keep a Backup?

It may take a week or a month to realize that a file has been deleted. Therefore, you should keep some backup tapes for a week, some for a month, and some for several months. Many organizations make yearly or quarterly backups that they archive indefinitely. After all, tape or CD-ROM is cheap, and del is forever. Keeping a yearly or a biannual backup "forever" is a small investment in the event that it should ever be needed again.

You may wish to keep on your system an index or listing of the names of the files on your backup tapes. This way, if you ever need to restore a file, you can find the right tape to use by scanning the index, rather than by reading in every single tape. Having a printed copy of these indexes is also a good idea, especially if you keep the online index on a system that may need to be restored!

If you keep your backups for a long period of time, you should be sure to migrate the data on your backups each time you purchase a new backup system. Otherwise, you might find yourself stuck with a lot of tapes that can't be read by anyone, anywhere. This happened in the late 1980s to the MIT Artificial Intelligence Laboratory, which had a collection of research reports and projects from the 1970s on seven-track tape. One day, the lab started a project to put all of the old work online once more. The only problem was that there didn't appear to be a working seven-track tape drive anywhere in the country that the lab could use to restore the data.

11.1.7 Security for Backups

Backups pose a double problem for computer security. On the one hand, your backup tape is your safety net; ideally, it should be kept far away from your computer system so that a local disaster cannot ruin both. On the other hand, the backup contains a complete copy of every file on your system, so the backup itself must be carefully protected.

11.1.7.1 Physical security for backups

If you use tape drives to make backups, be sure to take the tape out of the drive. One company in San Francisco that made backups every day never bothered removing the cartridge tape from their drive: when their computer was stolen over a long weekend by professional thieves who went through a false ceiling in their office building, they lost everything. "The lesson is that the removable storage media is much safer when you remove it from the drive," said an employee after the incident.

Do not store your backup tapes in the same room as your computer system. Any disaster that might damage or destroy your computers is likely to damage or destroy anything in the immediate vicinity of those computers as well. This rule applies to fire, flood, explosion, and building collapse.

You may wish to consider investing in a fireproof safe to protect your backup tapes. However, the safe should be placed off site, rather than right next to your computer system. While fireproof safes do protect against fire and theft, they don't protect your data against explosion, many kinds of water damage, and building collapse.

Be certain that any safe you use for storing backups is actually designed for storing your form of media. One of the fireproof lockboxes from the neighborhood discount store might not be magnetically safe for your tapes. It might be heat-resistant enough for storing paper, but not for storing magnetic tape, which cannot withstand the same high temperatures. Also, some of the generic fire-resistant boxes for paper are designed with a liquid in the walls that evaporates or foams when exposed to heat, to help protect paper inside. Unfortunately, these chemicals can damage the plastic in magnetic tape or CD-ROMs.

11.1.7.2 Write-protect your backups

After you have removed a backup tape from a drive, do yourself a favor and flip the write-protect switch. A write-protected tape cannot be accidentally erased.

If you are using the tape for incremental backups, you can flip the write-protect switch when you remove the tape, and then flip it again when you reinsert the tape later. If you forget to unprotect the tape, your software will probably give you an error and let you try again. On the other hand, having the tape write-protected will save your data if you accidentally put the wrong tape in the tape drive, or run a program on the wrong tape.

11.1.7.3 Data security for backups

File protections and passwords protect the information stored on your computer's hard disk, but anybody who has your backup tapes can restore your files (and read the information contained in them) on another computer. For this reason, keep your backup tapes under lock and key.

Several years ago, an employee at a computer magazine pocketed a 4mm cartridge backup tape that was on the desk of the system manager. When the employee got the tape home, he discovered that it contained hundreds of megabytes of personal files, articles in progress, customer and advertising lists, contracts, and detailed business plans for a new venture that the magazine's parent company was planning. The tape also included tens of thousands of dollars worth of computer application programs, many of which were branded with the magazine's name and license numbers. Quite a find for an insider who was setting up a competing publication!

When you transfer your backup tapes from your computer to the backup location, protect the tapes at least as well as you normally protect the computers themselves. Letting a messenger carry the tapes from building to building may not be appropriate if the material on the tapes is sensitive. Getting information from a tape by bribing an underpaid courier, or by knocking him unconscious and stealing it, is usually easier and cheaper than breaching a firewall, cracking some passwords, and avoiding detection online.

The use of encryption can dramatically improve security for backup tapes. However, if you do choose to encrypt your backup tapes, be sure that the encryption key is known by more than one person. You may wish to escrow your key. Otherwise, the backups may be worthless if the only person with the key forgets it, becomes incapacitated, or decides to hold your data for ransom.

Here are some recommendations for storing a backup tape's encryption key:

Change your keys infrequently if you change them at all. If you do change your keys, you must remember the old ones as well as the new, which probably means writing them all down in the same place. So you don't really get any security from changing the keys in the first place. Physical security of your backup tape should be your first line of defense.
Store copies of the key on pieces of paper in envelopes. Give the envelopes to each member of your organization's board of directors, or chief officers.
If your organization uses an encryption system such as PGP that allows a message to be encrypted for multiple recipients, encrypt and distribute the backup encryption key so that it can be decrypted by anyone on the board.
Alternately, you might consider a secret-sharing system, so that the key can be decrypted by any two or three board members working together, but not by any board member on his own.

11.1.8 Legal Issues

Finally, some firms should be careful about backing up too much information, or holding it for too long. Recently, backup tapes have become targets in lawsuits and criminal investigations. Backup tapes can be obtained by subpoena or during discovery in lawsuits. If your organization has a policy regarding the destruction of old paper files, you should extend this policy to backup tapes as well.

You may wish to segregate potentially sensitive data so that it is stored on separate backup tapes. For example, you can store applications on one tape, pending cases on another tape, and library files and archives on a third.

Back up your data, but back up with caution.

11.1.9 Deciding upon a Backup Strategy

The key to deciding upon a good strategy for backups is to understand the importance and time-sensitivity of your data. As a start, we suggest that the answers to the following questions will help you plan your backups:

How quickly do you need to resume operations after a complete loss of the main system?
How quickly do you need to resume operations after a partial loss?
Can you perform restores while the system is "live"?
Can you perform backups while the system is "live"?
What data do you need restored first? Next? Last?
Of the users you must listen to, who will complain the most if their data is not available?
What will cause the biggest loss if it is not available?
Who loses data most often from equipment or human failures?
How many spare copies of the backups must you have to feel safe?
How long do you need to keep each backup?
How much are you willing or able to spend?

only for RuBoard - do not distribute or recompile