81 Synchronizing Directories with FTP
#81 Synchronizing Directories with FTP
One of my most common uses for ftp is to ensure that a local copy of a directory is synchronized with a remote copy on a web server. The fancy name for this is content mirroring . The basic idea is simple: Move into a specific local directory, specify a remote server and remote directory, and then ensure that anything that's changed in one directory is copied to the other, as needed.
This book offers two scripts for FTP syncing: ftpsyncup and ftpsyncdown . The first uploads all files in the current directory to the remote directory, while the latter does the opposite and is presented next , as Script #82. Unless you're starting afresh on a new client system and thus need to acquire the latest versions of files from a server, you'll most likely use ftpsyncup far, far more often than its sibling, because people rarely work directly on files located on servers.
The Code
#!/bin/sh # ftpsyncup - Given a target directory on an ftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/ftpsyncup.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage:#!/bin/sh # ftpsyncup - Given a target directory on an ftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/ftpsyncup.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage: $0 user@host { remotedir }" >&2 exit 1 fi user ="$(echo $1 cut -d@ -f1)" server="$(echo $1 cut -d@ -f2)" echo " open $server" > $tempfile echo "user $user" >> $tempfile if [ $# -gt 1 ] ; then echo "cd $2" >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) fi done else for filename in $(find . - newer $timestamp -type f -print) do echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 0 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! ftp -n < $tempfile ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0user@host { remotedir }" >&2 exit 1 fi user="$(echo cut -d@ -f1)" server="$(echo cut -d@ -f2)" echo "open $server" > $tempfile echo "user $user" >> $tempfile if [ $# -gt 1 ] ; then echo "cd " >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) fi done else for filename in $(find . -newer $timestamp -type f -print) do echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "#!/bin/sh # ftpsyncup - Given a target directory on an ftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/ftpsyncup.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage: $0 user@host { remotedir }" >&2 exit 1 fi user ="$(echo $1 cut -d@ -f1)" server="$(echo $1 cut -d@ -f2)" echo " open $server" > $tempfile echo "user $user" >> $tempfile if [ $# -gt 1 ] ; then echo "cd $2" >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) fi done else for filename in $(find . - newer $timestamp -type f -print) do echo "put \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 0 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! ftp -n < $tempfile ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0: No files require uploading to $server" >&2 exit 0 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! ftp -n < $tempfile ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0
How It Works
The ftpsyncup script uses the .timestamp file to ascertain which files in the current directory have changed since the last time ftpsyncup synchronized with the remote system. If .timestamp isn't present, ftpsyncup automatically uploads everything in the current directory.
The actual upload of files occurs in the conditional statement at the end of the script, which tests to see whether the transfer worked:
if ! ftp -n < $tempfile ; then
| Caution | Be warned that some versions of Unix include an ftp program that doesn't properly return a nonzero failure code to the shell when a transfer fails. If you have such an ftp program, the conditional statement just shown will always return false, and the touch $timestamp statement will never execute. If you find that to be the case, remove the conditional block completely, leaving just the following: |
ftp -n < $tempfile touch $timestamp
Upon completion, the .timestamp file is either created or updated, depending on whether it exists.
Running the Script
To run this script, set up a directory on the remote server that you want to have mirror the contents of a local directory using ftp , and then synchronize the files in the current directory by invoking ftpsyncup with the account name, server name, and remote directory.
It would be quite easy to either drop this shell invocation directly into a cron job or to create a sync alias that remembers the command-line arguments, as shown in the "Running the Script" section of Script #83, Synchronizing Files with SFTP.
The Results
$ ftpsyncup taylor@intuitive.com archive Synchronizing Up: Found 33 files in local sync folder. Password: Done. All files synchronized up with intuitive.com
The Password: prompt is from within the ftp program itself, and on this Linux system, the entire interaction is quite succinct and graceful . The second time the command is invoked, it properly reports nothing to do:
$ ftpsyncup taylor@intuitive.com archive ftpsyncup: No files require uploading to intuitive.com
Hacking the Script
The ftpsyncup script uploads only files, ignoring directories. To rectify this, you could have each subdirectory within the working directory on the local system detected in the for filename in loop, add a mkdir command to the $tempfile file, and then invoke another call to ftpsyncup with the name of the new remote subdirectory at the end of the current script. You'd still need to ensure that you aren't irreversibly stepping into subdirectories, but that can be managed by invoking subsequent calls to ftpsyncup in subshells.
The problem with this solution is that it's really beginning to push the edges of what's logical to include in a shell script. If you have ncftp , for example, you'll find that it has built-in support for recursive put commands; rewriting these scripts to utilize that ncftp capability makes a lot more sense than continuing to struggle with the more primitive ftp command.
| When to rewrite your script in a "real" programming language | Any shell script that's grown to more than 150 lines or so would probably be better written in a more sophisticated language, whether Perl, C, C++, or even Java. The longest script in this entire book is only 149 lines long (Script #53, Validating User crontab Entries ). Your cutoff may vary, and there are some situations in which you must solve the problem within a shell script, but they're few and far between. Think carefully about whether you can solve the problem more efficiently in a more sophisticated development environment if you find your script is bursting at the seams and hundreds of lines long. |
