83 Synchronizing Files with SFTP
#83 Synchronizing Files with SFTP
While the ftp program is quite widely available, it's really something you should avoid like the plague. There are two reasons for this. First, ftp servers are notorious for being buggy and having security holes, and second, and much more problematic , ftp transfers all data between the server and client in the clear. This means that when you transmit files to your server, your account name and password are sent along without any encryption, making it relatively trivial for someone with a packet sniffer to glean this vital information. That's bad. Very bad.
Instead, all modern servers should support the considerably more secure ssh (secure shell) package, a login and file transfer pair that supports end-to-end encryption. The file transfer element of the encrypted transfer is sftp , and it's even more primitive than ftp , but we can still rewrite ftpsyncup to work with sftp , as shown in this script.
| Downloading an ssh package | If you don't have ssh on your system, complain to your vendor and administrative team. There's no excuse . You can also obtain the package and install it yourself by starting at http://www.openssh.com/ |
The Code
#!/bin/sh # sftpsync - Given a target directory on an sftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/sftpsync.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage:#!/bin/sh # sftpsync - Given a target directory on an sftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/sftpsync.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage: $0 user@host { remotedir }" >&2 exit 1 fi user ="$(echo $1 cut -d@ -f1)" server="$(echo $1 cut -d@ -f2)" if [ $# -gt 1 ] ; then echo "cd $2" >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put -P \"$filename\"" >> $ tempfile count=$(($count + 1)) fi done else for filename in $(find . - newer $timestamp -type f -print) do echo "put -P \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 1 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! sftp -b $tempfile "$user@$server" ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0user@host { remotedir }" >&2 exit 1 fi user="$(echo cut -d@ -f1)" server="$(echo cut -d@ -f2)" if [ $# -gt 1 ] ; then echo "cd " >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put -P \"$filename\"" >> $tempfile count=$(($count + 1)) fi done else for filename in $(find . -newer $timestamp -type f -print) do echo "put -P \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "#!/bin/sh # sftpsync - Given a target directory on an sftp server, makes sure that # all new or modified files are uploaded to the remote system. Uses # a timestamp file ingeniously called .timestamp to keep track. timestamp=".timestamp" tempfile="/tmp/sftpsync.$$" count=0 trap "/bin/rm -f $tempfile" 0 1 15 # zap tempfile on exit &sigs if [ $# -eq 0 ] ; then echo "Usage: $0 user@host { remotedir }" >&2 exit 1 fi user ="$(echo $1 cut -d@ -f1)" server="$(echo $1 cut -d@ -f2)" if [ $# -gt 1 ] ; then echo "cd $2" >> $tempfile fi if [ ! -f $timestamp ] ; then # no timestamp file, upload all files for filename in * do if [ -f "$filename" ] ; then echo "put -P \"$filename\"" >> $ tempfile count=$(($count + 1)) fi done else for filename in $(find . - newer $timestamp -type f -print) do echo "put -P \"$filename\"" >> $tempfile count=$(($count + 1)) done fi if [ $count -eq 0 ] ; then echo "$0: No files require uploading to $server" >&2 exit 1 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! sftp -b $tempfile "$user@$server" ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0: No files require uploading to $server" >&2 exit 1 fi echo "quit" >> $tempfile echo "Synchronizing: Found $count files in local folder to upload." if ! sftp -b $tempfile "$user@$server" ; then echo "Done. All files synchronized up with $server" touch $timestamp fi exit 0
How It Works
Like ftp , sftp allows a series of commands to be fed to it as a pipe or input redirect, which makes this script rather simple to write: Almost the entire script focuses on building the sequence of commands necessary to upload changed files. At the end, the sequence of commands is fed to the sftp program for execution.
As with Scripts #81 and #82, if you have a version of sftp that doesn't properly return a nonzero failure code to the shell when a transfer fails, simply remove the conditional block at the end of the script, leaving only
sftp -b $tempfile "$user@$server" touch $timestamp
Because sftp requires the account to be specified as user@host , it's actually a bit simpler than the equivalent ftp script shown in Script #81, ftpsyncup . Also notice the - P flag added to the put commands; it causes sftp to retain the local permission, creation, and modification times for all files transferred.
Running the Script
This script is simple to run: Move into the local source directory, ensure that the target directory exists, and invoke the script with your username, server name, and remote directory. For simple situations, I have an alias called ssync (source sync) that moves into the directory I need to keep in sync and invokes sftpsync automatically:
alias ssync=sftpsync taylor@intuitive.com /wicked/scripts
The "Hacking the Script" section shows a more sophisticated wrapper that makes the synchronization script even more helpful.
The Results
$ sftpsync taylor@intuitive.com /wicked/scripts Synchronizing: Found 2 files in local folder to upload. Connecting to intuitive.com... taylortaylor@intuitive.com's password: sftp> cd /wicked/scripts sftp> put -P "./003-normdate.sh" Uploading ./003-normdate.sh to /usr/home/taylor/usr/local/etc/httpd/htdocs/ intuitive/wicked/scripts/003-normdate.sh sftp> put -P "./004-nicenumber.sh" Uploading ./004-nicenumber.sh to /usr/home/taylor/usr/local/etc/httpd/htdocs/ intuitive/wicked/scripts/004-nicenumber.sh sftp> quit Done. All files synchronized up with intuitive.com
Hacking the Script
The wrapper script that I use to invoke sftpsync is a tremendously useful script, and I have used it throughout the development of this book to ensure that the copies of the scripts in the web archive (see http://www.intuitive.com/wicked/ ) are exactly in sync with those on my own servers, all the while adroitly sidestepping the insecurities of the ftp protocol.
This wrapper, ssync , contains all the necessary logic for moving to the right local directory (see the variable localsource ) and creating a file archive that has the latest versions of all the files in a so-called tarball (named for the tar , tape archive, command that's used to build it). The last line of the script calls sftpsync :
#!/bin/sh # ssync - If anything's changed, creates a tarball and syncs a remote # directory via sftp using sftpsync. sftpacct="taylor@intuitive.com" tarballname="AllFiles.tgz" localsource="$HOME/Desktop/Wicked Cool Scripts/scripts" remotedir="/wicked/scripts" timestamp=".timestamp" count=0 sftpsync="$HOME/bin/sftpsync" # First off, let's see if the local dir exists and has files if [ ! -d "$localsource" ] ; then echo "#!/bin/sh # ssync - If anything's changed, creates a tarball and syncs a remote # directory via sftp using sftpsync. sftpacct="taylor@intuitive.com" tarballname="AllFiles.tgz" localsource="$HOME/Desktop/Wicked Cool Scripts/scripts" remotedir="/wicked/scripts" timestamp=".timestamp" count=0 sftpsync="$HOME/bin/sftpsync" # First off, let's see if the local dir exists and has files if [ ! -d "$localsource" ] ; then echo "$0: Error: directory $localsource doesn't exist?" >&2 exit 1 fi cd "$localsource" # Now let's count files to ensure something's changed: if [ ! -f $timestamp ] ; then for filename in * do if [ -f "$filename" ] ; then count=$(($count + 1)) fi done else count=$(find . -newer $timestamp -type f -print wc -l) fi if [ $count -eq 0 ] ; then echo "$(basename $0): No files found in $localsource to sync with remote."; exit 0 fi echo "Making tarball archive file for upload" tar -czf $tarballname ./* # Done! Now let's switch to the sftpsync script exec $sftpsync $sftpacct $remotedir: Error: directory $localsource doesn't exist?" >&2 exit 1 fi cd "$localsource" # Now let's count files to ensure something's changed: if [ ! -f $timestamp ] ; then for filename in * do if [ -f "$filename" ] ; then count=$(($count + 1)) fi done else count=$(find . -newer $timestamp -type f -print wc -l) fi if [ $count -eq 0 ] ; then echo "$(basename#!/bin/sh # ssync - If anything's changed, creates a tarball and syncs a remote # directory via sftp using sftpsync. sftpacct="taylor@intuitive.com" tarballname="AllFiles.tgz" localsource="$HOME/Desktop/Wicked Cool Scripts/scripts" remotedir="/wicked/scripts" timestamp=".timestamp" count=0 sftpsync="$HOME/bin/sftpsync" # First off, let's see if the local dir exists and has files if [ ! -d "$localsource" ] ; then echo "$0: Error: directory $localsource doesn't exist?" >&2 exit 1 fi cd "$localsource" # Now let's count files to ensure something's changed: if [ ! -f $timestamp ] ; then for filename in * do if [ -f "$filename" ] ; then count=$(($count + 1)) fi done else count=$(find . -newer $timestamp -type f -print wc -l) fi if [ $count -eq 0 ] ; then echo "$(basename $0): No files found in $localsource to sync with remote."; exit 0 fi echo "Making tarball archive file for upload" tar -czf $tarballname ./* # Done! Now let's switch to the sftpsync script exec $sftpsync $sftpacct $remotedir): No files found in $localsource to sync with remote."; exit 0 fi echo "Making tarball archive file for upload" tar -czf $tarballname ./* # Done! Now let's switch to the sftpsync script exec $sftpsync $sftpacct $remotedir
With one command, a new archive file is created, if necessary, and all files (including the new archive, of course) are uploaded to the server as needed:
$ ssync Making tarball archive file for upload Synchronizing: Found 2 files in local folder to upload. Connecting to intuitive.com... taylor@intuitive.com's password: sftp> cd shellhacks/scripts sftp> put -P "./AllFiles.tgz" Uploading ./AllFiles.tgz to shellhacks/scripts/AllFiles.tgz sftp> put -P "./ssync" Uploading ./ssync to shellhacks/scripts/ssync sftp> quit Done. All files synchronized up with intuitive.com
This script can doubtless be hacked further. One obvious tweak would be to have ssync invoked from a cron job every few hours during the work day so that the files on a remote backup server are invisibly synchronized to your local files without any human intervention.
