Synchronizing Files: rsync

Synchronizing Files: `rsync`

Having a backup of critical files is the best way to avert disaster, but restoring a backup can be time consuming, and creating an exact mirror is often impractical on a day-to-day basis. In many cases, the best solution is to synchronize files between two or more machines as changes are made. Starting in Mac OS X 10.2, Apple has included rsync , an Open Source utility designed to make folder synchronization fast, scriptable, and painless.

NOTE

The version of rsync included in Mac OS X 10.2 does not include support for HFS+ resource forks, but a work-alike HFS+ compatible version is available for download through the Open Darwin Web site CVS, at http://www.opendarwin.org/cgi-bin/cvsweb.cgi/proj/rsync_hfs/.

Rsync can operate locally to synchronize files between directories on a single machine, or can work over a network. An especially nice feature of the software is its ability to use several different transport mechanisms for network transfers. The rsync utility, for example, can operate as a server and be used to host rsync -accessible directories on remote machines, but if setting up an additional dedicated server process isn't desirable, you don't have to. An alternative (and often overlooked) method of providing rsync access is through a remote shell such as SSH. If your Mac OS X machines are running SSH, they are ready to use rsync immediately ”without any additional software or configuration.

Using `rsync`

The rsync syntax is simple: rsync [options] <source> <destination> . For example, to synchronize the contents of the folder /Users/jray/source with /Users/jray/destination , I would use the following:

 %  rsync -va source/ destination  rsync: building file list... rsync: 6 files to consider. ./ Icon apache_pb.gif index.html macosxlogo.gif web_share.gif wrote 25079 bytes  read 100 bytes  50358.00 bytes/sec total size is 24711  speedup is 0.98

In this example, the -v (verbose) and -a (archive) options are included so that rsync displays files being copied and includes all attributes (permissions, owner, and so forth) of the original files. Running the command again produces slightly different output, because rsync doesn't need to recopy the files:

 %  rsync -va source/ destination  rsync: building file list... rsync: 6 files to consider. wrote 172 bytes  read 20 bytes  128.00 bytes/sec total size is 24711  speedup is 128.70

Here, no files were copied because the directories are already in sync.

NOTE

Trailing slashes mean something to rsync . If a path is given with a trailing slash, it refers to the contents of the named directory. If the slash is excluded, it means the directory itself.

Although synchronizing local directories may be useful, the true power of rsync is revealed when it is used over a remote network connection. To do this, all that is needed is a running SSH daemon on the remote side, and an account with access to the directory you want to sync. If you've ever used scp , you'll recognize the syntax for an SSH-tunneled rsync immediately. For example, assume I want to synchronize the directory /Users/jray/Tools located on a remote server www.poisontooth.com with a local directory Tools ”and, at the same time, compress the data as it is sent:

 %  rsync -vaze ssh  jray@www.poisontooth.com:/Users/jray/Tools/ Tools  jray@www.poisontooth.com's password:  ******  receiving file list ... done ./ htdigsearch/ adduser authuser.pl backupdisdain.pl cgiinput.pl deluser footer.html htdigsearch/README.txt htdigsearch/format.html htdigsearch/long.html htdigsearch/name.config htdigsearch/nothing.html htdigsearch/sample.conf htdigsearch/search.php htdigsearch/short.html htdigsearch/syntax.html indexsites.pl johnclient.pl johnscron ptail.pm runindex.pl setupconfig.pl ticker.tar wrote 448 bytes  read 216031 bytes  18824.26 bytes/sec total size is 1262138  speedup is 5.83

This example introduces the use of -e ssh to specify that ssh should be used as the remote shell through which rsync will connect and -z , which will compress data in real time to increase the transfer rate. There are a number of additional switches that can be used with rsync to change its behavior. Table 20.6 documents many of the useful options.

Table 20.6. Useful `rsync` Options

Option	Description
`-v`	Use verbose output.
`-a`	Recursively copy files, preserving as much information as possible.
`-c`	Calculate a checksum before sending and after receiving to verify file integrity.
`-b`	Create a backup of existing files. Any file that would be replaced is renamed with a `~` (tilde) extension.
`--suffix=` `<string>`	Set the string used as a suffix when using the -b option. By default, the suffix is `~` .
`-u`	Skip any files that are already newer (have a more recent date stamp) at the destination.
`-n`	Perform a "dry run" of the operation, displaying what would be copied, but without making any changes.
`--existing`	Do not copy any files that don't already exist on the destination.
`--delete`	Delete files on the destination that are not in the source.
`--max-delete=` `<#>`	Set the maximum number of files that can be deleted (to avoid disaster).
`--delete-after`	Delete files only after they have been successfully transferred.
`--exclude=` `<pattern>`	Exclude files from the operation based on a pattern, such as `*.dmg` .
`--include=` `<pattern>`	Do not exclude files that match the given pattern (used in conjunction with `--exclude` ).
`-z`	Compress transferred data.
`-p`	Transfer file permissions. Implied by `-a` .
`-o`	Transfer file ownership (must be root). Implied by `-a` .
`-g`	Transfer file group membership. Implied by `-a` .
`-t`	Transfer file modification times. Implied by `-a` .

Be certain to read through the rsync man page for additional information on command usage ”the include and exclude options can be used to generate complex file selection rulesets beyond simple wildcards. Also, if you'd like to run a dedicated rsync server rather than using an SSH, you can view a tutorial on the setup of an rsync daemon at http://www. macosxhints .com/article.php?story=20021023063424701.

Other Options

Although Retrospect is "king" of Mac OS X backups , there are a number of other utilities that may or may not suit your needs. If you've reached this point and still don't see something you like, visit these product sites for a few more options:

FWB Backup Toolkit . An incremental backup solution, lacking some of the features of Retrospect, but also lacking the price tag. http://www.fwb.com/html/backup_toolkit.html.
Tivoli Storage Manager Client . A client for IBM's Tivoli enterprise backup system. Although not a backup solution on its own, it does help integrate Mac OS X into popular corporate backup systems. http://www. ibmlink .ibm.com/usalets& parms =H_202-078.
Synchronize! X/Synchronize! X Pro . Award winning synchronization products that can be used to create scheduled bootable backups. http://www.qdea.com.
ChronoSync . Synchronizes the contents of drives . Features rules for selecting files, full Applescript support, and the ability to sync across network volumes . http://www.econtechnologies.com/.
FoldersSynchronizer . Quick and easy application to synchronize the contents of drives and folders. http://www.softobe.com/products/flsy/pp.html.
BackupScript . A 100% Applescript-based backup solution. Although not nearly as robust as the other tools, the Applescript can be easily modified as you see fit. http://www.railheaddesign.com/pages/software/backupscript/backupscript.html.
Carbon Copy Cloner . Carbon Copy Cloner, as mentioned earlier in the chapter, provides a GUI for the ditto command. Using CCC, you can create bootable mirror volumes of your Mac OS X computer. http://www.bombich.com/software/ccc.html.
Synk X . Open source and written in Cocoa, Synk X provides synchronization services to Mac OS X and can keep multiple volumes up to date. Synk X provides powerful Applescripting features and intelligent change recognition, such as backing up files even if the only change is an altered permission setting. Download Synk X from http://mypage.uniserve.ca/~rvoth/synk4_target.htm.

Although half a dozen tools may seem quite a few to a Mac user (hey, we're used to a few quality applications!), the choices are still very limited. We are currently without an all-in-one incremental backup utility that can be executed and scripted from the command line. Although Retrospect is schedulable and quite flexible, it does not benefit from the BSD underpinnings of Mac OS X. In time, solutions for (and by) Unix administrators are likely to appear, but for now, Macintosh backup software is slowly trying to catch up with the radical changes between OS 9 and OS X.

Top