Hack 50. Make Disk-to-Disk Backups for Large Drives
Today's hard drives are large enough that you could spend the rest of your life backing them up to tape. Putting drive trays in your servers and using removable drives as a backup destination provides a modern solution. Some of us are old, and therefore remember when magnetic tape was the de facto backup medium for any computer system. Disk drives were small, and tapes were comparatively large. Nowadays, the reverse is generally true disk drives are huge, and few tapes can hold more than a fraction of a drive's capacity. But these facts shouldn't be used as an excuse to skip doing backups! Backups are still necessary, and they may be more critical today than ever, given that the failure of a single drive can easily cause you to lose multiple partitions and hundreds of gigabytes of data. Luckily, dynamic device buses such as USB and FireWire ( a.k.a. IEEE 1094) and adaptors for inexpensive ATA drives to these connection technologies provide inexpensive ways of making any media removable without disassembling your system. Large, removable, rewritable media can truly simplify life for you (and your operators, if you're lucky enough to have some). A clever combination of removable media and a good backup strategy will make it easy for you to adapt disk drives to your systems to create large, fast, removable media devices that can solve your backup woes and also get you home in time for dinner (today's dinner, even). If you're fortunate enough to work somewhere that can buy the latest, partial terabyte backup tape technology, I'm proud to know you. This hack is for the rest of us. 5.6.1. Convenient Removable Media Technologies for BackupsDepending on the type of interfaces available on your servers, an easy way to roll your own removable media is to purchase external drive cases that provide USB or FireWire interfaces, but in which you can insert today's largest IDE or SATA disk drives. Because both USB and FireWire support dynamic device detection, you can simply attach a new external drive to your server and power it up, and the system will assign it a device identifier. If you don't know every possible device on your system, you can always check the tail of your system's logfile, /var/log/messages, to determine the name of the device associated with the drive you've just attached. Depending on how your system is configured, you may also need to insert modules such as uhci_hcd, ehci_hcd, and usb_storage in order to get your system to recognize new USB storage devices, or ohci1394 for FireWire devices.
Empty external drive cases with USB and/or FireWire interfaces start at around $35 on eBay or from your local computer vendor, but can run much higher if you decide you want a case that holds multiple drives. I was a Boy Scout eons ago and have been a sysadmin for a long time, and I like to "be prepared." I therefore further hedge my external drive options by putting drive trays in the external cases, so that I can quickly and easily swap drives in and out of the external cases without having to look for a screwdriver in a time of crisis. Figure 5-7 shows a sample drive tray. Drive trays come with a small rack that you mount in a standard drive bay and a drive tray into which you insert your hard drive. This combination makes it easy to swap hard drives in and out of the external drive case without opening it. I also put drive racks in the standard drive bays in my servers so that I can quickly add or replace drives as needed.
Figure 5-7. A removable drive rack with drive tray inserted![]() 5.6.2. Choosing the Right Backup CommandOnce you have a mechanism for attaching removable storage devices to your system and have a few large drives ready, it's important to think through the mechanism that you'll use for backups. Most traditional Unix backups are done using specialized backup and restore commands called dump and restore, but these commands take advantage of built-in knowledge about filesystem internals and therefore aren't portable across all of the different filesystems available for Linux. (A version of these commands for ext2/ext3 filesystems is available at http://dump.sourceforge.net.) Another shortcoming of the traditional dump/restore commands for Unix/Linux is that they reflect their origins in the days of mag tapes by creating output data in their own formats in single output files (or, traditionally, a stream written to tape). This is also true of more generic archiving commands that are also often used for backups, such as tar, cpio, and pax.
So I can use the same backup scripts and commands regardless of the type of Linux filesystem that I'm backing up, I prefer to use file-and directory-level commands such as cp rather than filesystem-level commands. This is easy to do when doing disk-to-disk backups, because the backup medium is actually a disk that contains a filesystem that I mount before starting the backup. After mounting the drive, I use a script that invokes cp to keep the backup drive synchronized with the contents of the filesystem that I'm backing up, using a cp command such as the following: # cp dpRux /home /mnt/home-backup As you can see from this example, the script creates mount points for the backup filesystems that indicate their purpose, which makes it easier for other sysadmins to know why a specific drive is mounted on any given system. I use names that append the string backup to the name of the filesystem that I'm backing uptherefore, /mnt/home-backup is used as a mount point for the backup filesystem for the filesystem mounted as /home. You're welcome to choose your own naming convention, but this seems intuitive to me. The cp options that I use have the following implications:
5.6.3. The CodeThe actual script that I use to do these sorts of backups is the following (feel free to use or modify it if you'd like): #!/bin/bash # # wvh's simple backup script using cp # if [ $# != 2 ] ; then echo " Usage: cp_backup partition backup-device" echo " Example: cp_backup /home /dev/sda1" exit fi VERBOSE="no" STDOPTS="-dpRux" LOGFILE="/var/log/backup/simple.log" TARGETBASE=`echo $1 | sed -e 's;^\/;;' -e 's;\/;-;g'` FULLTARGET="/mnt/"$TARGETBASE"-backup" DATE=`date` export BACKUPTASK="$1 to $2" trap cleanup 1 2 3 6 cleanup() { echo " Uh-oh, caught signal: tidying up…" | tee -a $LOGFILE DATE=`date` umount $FULLTARGET echo "Aborted simple backups of $BACKUPTASK $DATE" | tee -a $LOGFILE exit 1 } if [ ! -d /var/log/backup ] ; then mkdir -p /var/log/backup fi echo "Starting simple backups of $BACKUPTASK at $DATE" | tee -a $LOGFILE if [ ! -d $FULLTARGET ] ; then echo " Creating mountpoint $FULLTARGET" | tee -a $LOGFILE mkdir -p $FULLTARGET fi MOUNTED=`df | grep $FULLTARGET` if [ "x$MOUNTED" != "x" ] ; then echo " Something is already mounted at $FULLTARGET - exiting" | tee -a $LOGFILE exit fi mount $2 $FULLTARGET if [ x$? != "x0" ] ; then echo " Mount of backup volume $2 failed - exiting" | tee -a $LOGFILE exit fi # # This block keeps copies of important system files on all backup volumes # in a special directory called .123_admin. They're small, it's only slow # once, and I'm paranoid. # if [ ! -d $FULLTARGET"/.123_admin" ] ; then mkdir -p $FULLTARGET"/.123_admin/conf" fi echo " Backing up system files to $FULLTARGET/.123_admin" | tee -a $LOGFILE cd /etc cp -u passwd group shadow $FULLTARGET"/.123_admin" if [ -d sysconfig ] ; then cp -uR sysconfig $FULLTARGET"/.123_admin" fi find . -name "*.conf" -print | while read file ; do cp -u $file $FULLTARGET"/.123_admin/conf" done # # Now we actually do the cp backups # DATE=`date` echo " Starting actual backup of $BACKUPTASK at $DATE" | tee -a $LOGFILE cd $1 if [ x$VERBOSE != "xno" ] ; then cp $STDOPTS"v" . $FULLTARGET else cp $STDOPTS . $FULLTARGET fi umount $FULLTARGET DATE=`date` echo "Completed simple backups of $BACKUPTASK at $DATE" | tee -a $LOGFILE You'll note that I don't log each file that's being backed up, though that would be easy to do if running the script in verbose mode by using the tee command to clone the cp command's output to the logfile. The traditional Unix/Linux dump and restore commands use the file /etc/dumpdates to figure out which full and incremental backups to use in order to restore a specific file or filesystem, but this isn't necessary in this case because we're copying the updated files from the specified partition to a full backup of that partition, not just doing an incremental backup in traditional Unix/Linux terms. 5.6.4. Running the CodeIf you're following along at home, you can use this script by entering it in your favorite text editor, saving it to a file called cp_backup in /usr/local/bin, making it executable (chmod 755 /usr/local/bin/cp_backup), and then executing it (after making sure that you've mounted a spare disk as a backup target, and that the spare disk is the same size as or larger than the filesystem that you want to back up). For example, to back up the partition mounted as /mnt/music on my system (which contains 100% legally purchased music in digital form) to a 250-GB disk containing the single partition /dev/sda1, I would use the following command: # /usr/local/bin/cp_backup /mnt/music /dev/sda1 You can even automate these sorts of backups by adding an entry that executes them to root's crontab file. As the root user or via sudo, execute the crontab e command and append a line like the following to the end of the file: 0 2 * * * $/usr/local/bin/cp_backup /mnt/music /dev/sda1 This will run the cp_backup script to back up /mnt/music to /dev/sda1 every night at 2 A.M. 5.6.5. Choosing What to Back UpThe previous sections explained why disk-to-disk backups are the smartest choice for low-cost backups of today's huge disk drives, and advocated file-and directory-level commands as an easy backup mechanism that is independent of the actual format of the filesystem that houses the data you're backing up. Keeping a large number of spare drives around can be costly, though, so I try to minimize the number of filesystems that I back up. The traditional Unix/Linux dump command does this through entries in the /etc/fstab file that identify whether the filesystem should be backed up or notif the entry in the next-to-last column in /etc/fstab is non-zero, the filesystem will be backed up. My general rule is to only back up filesystems that contain user data. Standard Linux filesystems such as / and /usr can easily be recreated from the distribution media or from partition images [Hack #49]. Since the backup script I use keeps copies of system configuration files, I'm not that worried about preserving system configuration information. 5.6.6. Summary and TipsThis hack provides an overview of doing modern backups and a script that I use to do them on most of the systems I deploy. To use this approach, the target devices that you're backing up to have to have at least as much space as the filesystem that you're backing up, and you'll have to preen or wipe the daily backup devices every so often (generally after a full backup) in order to minimize the number of copies of files and directories that have been deleted from the live filesystem but still exist on the backup drives. If your systems use logical volumes that span multiple disks, you'll have to use equivalent, multi-disk backup devices, but they can often be simpler, cheaper devices than those that house your live data. For example, if you're backing up filesystems that live on a RAID array, you don't have to have a RAID backup deviceyou can get away with sets of drives that are large enough to hold the data itself, not its mirrors or checksum disks. |