59 Downloading Files via FTP | Wicked Cool Shell Scripts

#59 Downloading Files via FTP

One of the original killer apps of the Internet was file transfer, and the king of file transfer programs is ftp , the File Transfer Protocol. At some fundamental level, all Internet interaction is based upon file transfer, whether it's a web browser requesting an HTML document and its accompanying graphic files, a chat server relaying lines of discussion back and forth, or an email message traveling from one end of the earth to the other.

The original ftp program still lingers on, and while its interface is quite crude, it's powerful, capable, and well worth taking advantage of with some good scripts. There are plenty of newer ftp programs around, notably ncftp (see http://www.ncftp.org/), but with some shell script wrappers, ftp does just fine for uploading and downloading files.

For example, a typical use for ftp is to download files from the Internet. Quite often, the files will be located on anonymous FTP servers and will have URLs similar to ftp://someserver/ path /filename . A perfect use for a scripted ftp .

The Code

 #!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and tries to obtain the #    file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then   echo "Usage:  #!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and tries to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF open $server user ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0 
 ftp://..." >&2   exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo   cut -c1-6)" != "ftp://" ] ; then   echo "  #!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and tries to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF open $server user ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0 
 : Malformed url. I need it to start with ftp://" >&2;   exit 1 fi server="$(echo   cut -d/ -f3)" filename="$(echo   cut -d/ -f4-)" basefile="$(basename $filename)" echo  #!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and tries to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF open $server user ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0 
 : Downloading $basefile from server $server ftp -n << EOF open $server user ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then   ls -l $basefile fi exit 0

How It Works

The heart of this script is the sequence of commands fed to the ftp program:

 ftp -n << EOF open $server user ftp $anonpass get $filename $basefile quit EOF

This script illustrates the essence of a batch file: It prepares a sequence of instructions that it then feeds to a separate program, in this case ftp . Here we specify the server connection to open, specify the anonymous user ( ftp ) and whatever default password is specified in the script configuration (typically your email address), and then get the specified file from the FTP site and quit the transfer.

Running the Script

In use, this script is simple and straightforward: Just fully specify an ftp URL, and it'll download the specified file to the current working directory.

The Results

 $  ftpget ftp://ftp.ncftp.com/ncftp/ncftp-3.1.5-src.tar.bz2  ftpget: Downloading ncftp-3.1.5-src.tar.bz2 from server ftp.ncftp.com -rw-rw-r--    1 taylor   taylor     394777 Jan  6 08:26 ncftp-3.1.5-src.tar.bz2

Some versions of ftp are more verbose than others, and because it's not too uncommon to find a slight mismatch in the client and server protocol, those verbose versions of ftp can spit out scary-sounding but safely ignored errors, like Unimplemented command. For example, here's the same script run within Mac OS X:

 $  ftpget ftp://ftp.ncftp.com/ncftp/ncftp-3.1.5-src.tar.bz2  055-ftpget.sh: Downloading ncftp-3.1.5-src.tar.bz2 from server ftp.ncftp.com Connected to ncftp.com. 220 ncftpd.com NcFTPd Server (licensed copy) ready. 331 Guest login ok, send your complete e-mail address as password. 230-You are user #10 of 16 simultaneous users allowed. 230- 230 Logged in anonymously. Remote system type is UNIX. Using binary mode to transfer files. local: ncftp-3.1.5-src.tar.bz2 remote: ncftp/ncftp-3.1.5-src.tar.bz2 502 Unimplemented command. 227 Entering Passive Mode (209,197,102,38,212,218) 150 Data connection accepted from 12.253.112.102:49236; transfer starting for ncftp-3.1.5-src.tar.bz2 (394777 bytes). 100% ***********************************************   385 KB  266.14 KB/s 00:00 ETA 226 Transfer completed. 394777 bytes received in 00:01 (262.39 KB/s) 221 Goodbye. -rw-r--r--  1 taylor  staff  394777 Oct 13 20:32 ncftp-3.1.5-src.tar.bz2

If your ftp is excessively verbose, you can quiet it down by adding a - V flag to the ftp invocation (that is, instead of ftp - n , use ftp - nV ).

An alternative to ftpget

Worth noting is that there's a popular utility called curl that performs the same task as ftpget . If you have curl available, it's a superior alternative to this script, but because we're going to build upon the ideas embodied in ftpget for more sophisticated ftp interactions later in this book, you'll benefit from studying the code here.

Hacking the Script

This script can be expanded to uncompress the downloaded file automatically (see Script #37, Working with Compressed Files , for an example of how to do this).

You can also tweak this script just a bit and end up with a simple tool for uploading a specified file to an FTP server. If the server supports anonymous connections (few do nowadays, thanks to skript kiddies and other delinquents, but that's another story), all you really have to do is specify a destination directory on the command line (or in the script) and change the get to a put in the main script:

 ftp -n << EOF open $server user ftp $anonpass cd $destdir put $filename quit EOF

To work with a password-protected account, you could hard-code your password into the script ” a very bad idea ” or you could have the script prompt for the password interactively. To do that, turn off echoing before a read statement, and then turn it back on when you're done:

 echo -n "Password for ${user}: " stty -echo read password stty echo echo ""

A smarter way to prompt for a password, however, is to just let the ftp program do the work itself, as demonstrated in Script #81, Synchronizing Directories with FTP.