Implementing a Backup Strategy

As previously mentioned, the ideal backup strategy is to do a full backup of all the filesystems frequently. This way, should you need to restore a single file or the whole system, you need to access only the latest backup tape set and go from there. Unfortunately, daily full backups are only possible for small systems where you have enough low system usage time (a backup "window") to create a complete backup. It also becomes expensive to continue purchasing new tapes (because none of the existing tapes can be reused during the archive-retaining time period). For most systems, a combination of full and incremental backups coupled with a tape rotation scheme, where a given set of tapes is reused, is the best option.

NOTE

Always schedule your system backup to take place during a time of little or no user activityfor two reasons. First, backup procedures take up system resources such as CPU cycles and put a high demand on hard disk access. This could degrade the system performance, and in some extreme cases, the backup process can consume considerable resources, resulting in a temporarily denial of service. Second, when users are on the system, there will always be opened files, which are not backed up (unless your backup software has an "open file agent" that handles them). Therefore, to back up as many changed files as possible, during the time that your backup job runs, you should shut down any applications, such as inventory database programs, that keep files open constantly, and you should also restrict user access to files. (Also see the "Database Backups: Cold or Hot?" section later in this chapter.)

The main drawback to incremental backups appears when you need to perform a full system restore. You need to first restore the last full backup and then apply all the incremental backups from that point onward. Therefore, you save some time during the backup process, but the restore phase takes a little longer. In the case of a partial restore, you can easily do that from the incremental backup, but you would have to scan through a number of media to locate the one where the desired data is stored.

There are two types of incremental backups: backup files changed since the last complete backup (often referred to as differential backups) or backup files changed since the last incremental backup. Assume that you have set up a backup schedule as listed in Table 10.1.

Table 10.1. Sample Differential Backup Schedule

DAY OF WEEK	BACKUP TYPE
Sunday	FB, Full backup
Monday	Differential backup of files changed since Sunday's full backup
Tuesday	Differential backup of files changed since Sunday's full backup
Wednesday	Differential backup of files changed since Sunday's full backup
Thursday	Differential backup of files changed since Sunday's full backup
Friday	Differential backup of files changed since Sunday's full backup
Saturday	No backup (assuming that no one uses the system on weekends)

If you need to restore a file lost on Thursday, you need to access only one tape: either the differential tape created on Wednesday (if the file was changed during the current week) or the full backup tape created on Sunday (if the file was not changed during the current week). To fully restore the system, you need only two tapes: the full backup tape and the latest differential tape. Under this schedule, the backup time gets longer as the week progresses because more and more files need to be backed up. However, it makes restoring files simple. This example is a simplification of the Grandfather-Father-Son rotation method.

NOTE

The main drawback of differential backups is that, as the week progresses, you have more and more changed files to back up as you are backing up files changed since the last full backup. Therefore, it is likely that by Friday, your backup time will take twice as long as it did on Monday.

Table 10.2 shows a different backup schedule. This one does a full backup at the beginning of the month, a weekly incremental on Mondays, and a daily incremental for the rest of the week.

Table 10.2. Another Backup Schedule Example

DAY	BACKUP TYPE
First of Month	FB, Full backup
Every Monday	Differential backup of files changed since the last full backup
Tuesday	Incremental backup of files changed since Monday
Wednesday	Incremental backup of files changed since Tuesday
Thursday	Incremental backup of files changed since Wednesday
Friday	Incremental backup of files changed since Thursday
Saturday	No backup (assuming that no one uses the system on weekends)
Sunday	No backup (assuming that no one uses the system on weekends)

Using this schedule, restoring files is a little more complicated than it was in the previous example. For instance, to restore a file you lost, you need to do the following:

1.	Use the full backup tape if the file wasn't changed during the month.
2.	Use the latest differential tape if the file was changed in the previous week but not during the current week.
3.	Use the appropriate incremental tape from the current week if the file was changed this week.

The advantage of this sample schedule is that it takes less time per day for the backups because it backs up only those files changed from the previous workday. The downside is that a little more work is required to restore a file.

The preceding two examples do not take into account multiple tape sets that would be necessary to go back to data from the previous week or month. The Grandfather-Father-Son and Tower of Hanoi rotation systems described in the following sections, on the other hand, use multiple tape sets. These two rotation methods are among the most often used by backup software.

Grandfather-Father-Son Rotation Method

The Grandfather-Father-Son rotation scheme (GFS for short) uses three "generations" of tapes (hence, the name), as illustrated in Table 10.3. It uses a total of 21 tapes. Of these 21 tapes, 4 are daily tape sets labeled Monday, Tuesday, Wednesday, and Thursday. Another 4 tapes are weekly tape sets labeled Friday1, Friday2, Friday3, and Friday4; for months that have five Fridays, a fifth weekly tape set labeled Friday5 is used. Also, 12 tapes labeled January, February, and so on through December act as monthly tapes.

Table 10.3. GFS Tape Rotation Scheme

	WEEK1	WEEK2	WEEK3	WEEK4
Daily	Monday	Monday	Monday	Monday
Daily	Tuesday	Tuesday	Tuesday	Tuesday
Daily	Wednesday	Wednesday	Wednesday	Wednesday
Daily	Thursday	Thursday	Thursday	Thursday
Weekly	Friday1	Friday2	Friday3	Friday4
Monthly				January
	WEEK5	WEEK6	WEEK7	WEEK8
Daily	Monday	Monday	Monday	Monday
Daily	Tuesday	Tuesday	Tuesday	Tuesday
Daily	Wednesday	Wednesday	Wednesday	Wednesday
Daily	Thursday	Thursday	Thursday	Thursday
Weekly	Friday1	Friday2	Friday3	Friday4
Monthly				February

This rotation scheme recycles the daily tapes the following week (the "sons" because they have the shortest life span), the weekly backup tapes after five weeks (the "fathers"), and the monthly tapes the following year (the "grandfathers").

NOTE

The monthly tapes are full backups, whereas the daily and weekly tapes are incrementals. As to which type of incremental backup (weekly or daily) you use, the choice is up to you. However, you should base your decision on these factors: how large a backup window you have, the amount of data to back up, and the throughput of your backup device.

CAUTION

The daily tapes get the most use; therefore, they are most prone to failure. Check these tapes regularly and often for wear-and-tear before using them.

Tower of Hanoi Rotation Method

The Tower of Hanoi rotation scheme is named after an ancient mathematical game of the same name. The rotation scheme is sometimes referred to as the ABACABA rotation method, based on the frequency with which tapes are rotated. Five or more tapes are needed in this implementation. To simplify the discussion, five tapes labeled A, B, C, D, and E are used.

NOTE

The French mathematician Edouard Lucas invented the Tower of Hanoi game, sometimes referred to as the Tower of Brahma or the End of the World Puzzle, in 1883.

The basic idea is that each of the five tapes is used at different rotation intervals. For example, tape A is used every other day; tape B, every fourth day; tape C, every eighth day; and tapes D and E, every sixteenth day. Typically, tapes A, B, and C are incremental backups, and tapes D and E are full backups. Table 10.4 shows the rotation pattern.

Table 10.4. Tower of Hanoi Tape Rotation Scheme

WEEK1	WEEK2	WEEK3	WEEK4
M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F
A,B,A,C,A	B,A,D,A,B	A,C,A,B,A	E,A,B,A,C
WEEK5	WEEK6	WEEK7	WEEK8
M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F
A,B,A,D,A	B,A,C,A,B	A,E,A,B,A	C,A,B,A,D
WEEK9	WEEK10	WEEK11	WEEK12
M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F	M,T,W,Th,F
A,B,A,C,A	B,A,D,A,B	A,C,A,B,A	E,A,B,A,C

Notice that the pattern recycles itself every 31 days (one month), with the use of either tape D or E between the cycles. If you use fewer than five tapes, the cycle repeats itself every 15 days, which doesn't "map" nicely to the requirement of monthly backups. In the case where five tapes are used (as in the example presented here), tapes D and E are alternated in their usage within the cycle, so they are used once every 16 days. This difference is shown in bold in Table 10.4.

Some Tips and Tricks

Having chosen a backup media rotation scheme does not mean you now have a viable backup strategy. You also need to decide what to back up, how often to back up, and how best to keep track and safeguard your backup tapes. The following are some points to consider:

• Automate your backup. Most backup software allows you to set up a schedule so that the process initiates itself periodically without manual intervention. This is important because if you need to manually start the backup procedure daily, inevitably there will be a day that you forget because you don't have time, or something happens and prevents you from doing it. And, as Murphy's Law will have it, that is the one day you will need a backup!

  If you are using one of the Linux utilities, such as tar, for your backup, you can always automate it using one of the other Linux tools, such as at or cron. They are discussed later in this chapter.

  TIP

  You might want to get a backup device that holds multiple tapes (or whatever medium)about twice as much as you require for a backup jobso that you have the option of not changing the tapes for one day.

  
  

• Back up every file. Do not limit backups to just documents or certain files; you will inevitably need one that was not backed up. Also, having a backup of every file, especially the system files, allows you to rebuild your entire server quickly, should there be a need. Having a pristine copy of your system utilities gives you a way to determine whether an intruder has installed any rootkits on your system. Of course, if your old backups contained already-compromised files, you wouldn't necessarily know by comparing existing files to those on the backup. (Refer to Chapter 12, "Intrusion Detection" for more information about rootkits.)

  You can elect to back up documents and files that change frequently (such as those in /home, /etc, and perhaps /usr/local) in your daily incremental backup, and include system files in the full backup. In general, you can safely exclude /tmp, /var/tmp, and /usr/var/tmp from being backed up because they usually only contain temporary files.

  If you take intrusion detection seriously, you should also backup your /var/log directory. The log files can be very useful for intrusion situations and for general troubleshooting. However, log files tend to be large, so perhaps you should compress themusing gzip, for instancebefore backing them up. Alternatively, you can use a central log server (see Chapter 13, "System Security") so the backup can exclude /var/log should backup media capacity become an issue.

  WARNING

  You should exclude /proc from your backups because it is not a true disk filesystem. Rather, /proc is really just a way for the kernel to provide you with information about the operating system and system memory. For instance, /proc/kcore is a pseudo-file containing the contents of your entire physical memory; backing it up is rather a waste of time and backup media storage space! (More information about /proc can be found in Chapter 7, "System Management and Monitoring.")

  You might also want to avoid backing up the /mnt filesystem, unless you need to back up the files from your CD-ROM device, floppy drive, network file shares, or other mounted devices.

  
  

• Make copies of your backup. Storage media will fail, especially tapes, after prolonged and repeated use. It does not hurt to have multiple copies (or "generations") of your backups, even if they are older copies. When needed, an old copy is better than no copy at all.

  Some backup software includes a tape-to-tape copy feature that you can use to make a duplicate of your backup without having to actually perform another backup.

• Keep offsite copies. You never know when a fire, flood, theft, or some natural disaster will make your office inaccessible for days or weeks and your offsite copy is your only readily accessible copy. One option is to keep your current weekly backup onsite (in a fire-resistant safe, for instance) and send the previous week's backup to a secure offsite location.

  TIP

  There are companies that specialize in secure and climate-controlled storage for both documents and backup media. They often provide a courier service to pick up your new set of tapes and drop off an older set for reuse. Check your local yellow pages. In a pinch, a bank safety deposit box can be a good alternative.

  
  

• Verify your backups. You need to know whether you can ever restore from your backups. Most backup software has a verification feature; although it adds to the backup time, use it whenever possible. Also, periodically restore a few files at random and verify them yourself.

• Label all media. Be sure to label and date all mediatapes, DVDs, whateverused in a backup. If you have to use multiple items, make sure they are numbered sequentially. If you send backups offsite, document them so you know which tapes are where.

• Keep track of where your files are located. As you can see from the Tower of Hanoi discussion, keeping track of which tape to use when can be complicated. Labeling all your media is certainly a starting point. Fortunately, many backup programs logically label the media so that they can detect whether the right one has been inserted. At the same time, the backup software keeps track of what files have been backed up on which tape using its own database. Make sure that this database is backed up as well.

• Back up your system before making major changes. When you upgrade your system or make any configuration changes, you should definitely make a backup of at least the / (root), /usr, and /home filesystems (if they are not on the same disk partition), if not a full backup. Although such failures don't happen often, it is possible for a critical library or package not to upgrade properly, crippling your system.

NOTE

A root filesystem generally contains everything needed to support a functional Linux system, such as the following:

• Minimum set of directories, such as /dev, /proc, /bin, /sbin, /etc, /lib, /tmp

• Basic set of utilities, such as sh, ls, cp, mv, and so on

• Minimum set of configuration files, such as rc, inittab, fstab, and so on

• Device files, such as /dev/hd*, /dev/tty*, /dev/fd0, and so on

• Runtime libraries to provide basic functions used by utilities

Before we discuss the actual backup tools, there is one more topic to consider as part of your backup strategy: how best to back up a database application, such as Oracle or MySQL.

Database Backups: Cold or Hot?

When you are backing up files belonging to a database application, such as Oracle or MySQL, or applications that constantly keep certain files open, such as Lotus Notes, you need to give some extra thought than you would when backing up typical documents, such as OpenOffice files. There are two methods of performing a backup on a database: cold and hot. In a cold backup, an application is taken offline, which means there's no user access to the data, and the data is backed up; this is the way backups are normally done. In a hot backup, on the other hand, the application remains online, and user access is retained while the backup is performed.

A cold backup is usually the optimal solution for those applications that can tolerate multiple hours of downtime to perform the backup. Some applications that used to be backed up cold have now grown so large that the backup cannot be completed during the allotted time window. If a cold backup is still desired, one way is to take a point-in-time "snapshot" of the data, and within a matter of minutes (depending on the size of the data files involved), the application is brought online. The snapshot can then be mounted back onto the application server, or mounted directly to the backup server, and backed up. Total downtime for the application in such a case is the time required to stop the application, perform the snapshot, and then restart the application.

NOTE

To take a snapshot of an application's database, either you need an application that provides this feature, or you need to obtain additional software and/or hardware.

TIP

It is possible to create a snapshot device that is an alias for an existing Logical Volume (LV). The snapshot device, which can be accessed read-only, contains a point-in-time image of the LV; in other words, while applications continue to change the data on the LV, this logical device contains the unchanging image of the LV at the time when the snapshot was created. This makes it possible for you to do a consistent backup without shutting anything down or using any special software. This method is independent of any software because it happens in the LV Manager abstraction layer. SUSE has included a Logical Volume Manager since SUSE LINUX 6.3. For details on performing backups using LVM snapshots, see http://www.tldp.org/HOWTO/LVM-HOWTO/index.html.

If you want to perform a hot backup on an application that has constantly open files, the application must have a hot backup feature, and the backup software needs hot backup support for the specific application. Generally speaking, in hot backup mode, instead of writing to the live data, the application queues up the updates in a special file so the backup software can get a complete backup of the database. The special file is backed up next. After this is done, the application is then allowed to apply the queued-up changes to the database, thus bringing everything up to date.

Therefore, to decide whether you should perform a cold or hot backup of your database application files, you need to take the following factors into consideration:

• Can the application data files be backed up cold and not violate their integrity? If not, does the application have a hot backup feature?

• Does your backup software support the hot backup option of that particular application?

• How much of a downtime window do you have to back up this application's data files? Perhaps the window is wide enough for a cold backup, which would make life a lot easier.

If you have small downtime window but have sufficient disk space, perhaps using the Logical Volume snapshot feature is an option.