59 Downloading Files via FTP

#59 Downloading Files via FTP

One of the original killer apps of the Internet was file transfer, and the king of file transfer programs is ftp , the File Transfer Protocol. At some fundamental level, all Internet interaction is based upon file transfer, whether it's a web browser requesting an HTML document and its accompanying graphic files, a chat server relaying lines of discussion back and forth, or an email message traveling from one end of the earth to the other.

The original ftp program still lingers on, and while its interface is quite crude, it's powerful, capable, and well worth taking advantage of with some good scripts. There are plenty of newer ftp programs around, notably ncftp (see http://www.ncftp.org/), but with some shell script wrappers, ftp does just fine for uploading and downloading files.

For example, a typical use for ftp is to download files from the Internet. Quite often, the files will be located on anonymous FTP servers and will have URLs similar to ftp://someserver/ path /filename . A perfect use for a scripted ftp .

The Code

#!/bin/sh

# ftpget - Given an ftp-style URL, unwraps it and tries to obtain the
#    file using anonymous ftp.

anonpass="$LOGNAME@$(hostname)"

if [ $# -ne 1 ] ; then
  echo "Usage:
#!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and

tries

to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF

open

$server

user

ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0

ftp://..." >&2
  exit 1
fi

# Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz

if [ "$(echo   cut -c1-6)" != "ftp://" ] ; then
  echo "
#!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and

tries

to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF

open

$server

user

ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0

: Malformed url. I need it to start with ftp://" >&2;
  exit 1
fi

server="$(echo   cut -d/ -f3)"
filename="$(echo   cut -d/ -f4-)"
basefile="$(basename $filename)"

echo
#!/bin/sh # ftpget - Given an ftp-style URL, unwraps it and

tries

to obtain the # file using anonymous ftp. anonpass="$LOGNAME@$(hostname)" if [ $# -ne 1 ] ; then echo "Usage: $0 ftp://..." >&2 exit 1 fi # Typical URL: ftp://ftp.ncftp.com/2.7.1/ncftpd-2.7.1.tar.gz if [ "$(echo $1  cut -c1-6)" != "ftp://" ] ; then echo "$0: Malformed url. I need it to start with ftp://" >&2; exit 1 fi server="$(echo $1  cut -d/ -f3)" filename="$(echo $1  cut -d/ -f4-)" basefile="$(basename $filename)" echo ${0}: Downloading $basefile from server $server ftp -n << EOF

open

$server

user

ftp $anonpass get $filename $basefile quit EOF if [ $? -eq 0 ] ; then ls -l $basefile fi exit 0

: Downloading $basefile from server $server

ftp -n << EOF
open $server
user ftp $anonpass
get $filename $basefile
quit
EOF

if [ $? -eq 0 ] ; then
  ls -l $basefile
fi

exit 0

How It Works

The heart of this script is the sequence of commands fed to the ftp program:

ftp -n << EOF
open $server
user ftp $anonpass
get $filename $basefile
quit
EOF

This script illustrates the essence of a batch file: It prepares a sequence of instructions that it then feeds to a separate program, in this case ftp . Here we specify the server connection to open, specify the anonymous user ( ftp ) and whatever default password is specified in the script configuration (typically your email address), and then get the specified file from the FTP site and quit the transfer.

Running the Script

In use, this script is simple and straightforward: Just fully specify an ftp URL, and it'll download the specified file to the current working directory.

The Results

$

ftpget ftp://ftp.ncftp.com/ncftp/ncftp-3.1.5-src.tar.bz2

ftpget: Downloading ncftp-3.1.5-src.tar.bz2 from server ftp.ncftp.com
-rw-rw-r--    1 taylor   taylor     394777 Jan  6 08:26 ncftp-3.1.5-src.tar.bz2

Some versions of ftp are more verbose than others, and because it's not too uncommon to find a slight mismatch in the client and server protocol, those verbose versions of ftp can spit out scary-sounding but safely ignored errors, like Unimplemented command. For example, here's the same script run within Mac OS X:

$

ftpget ftp://ftp.ncftp.com/ncftp/ncftp-3.1.5-src.tar.bz2

055-ftpget.sh: Downloading ncftp-3.1.5-src.tar.bz2 from server ftp.ncftp.com
Connected to ncftp.com.
220 ncftpd.com NcFTPd Server (licensed copy) ready.
331 Guest login ok, send your complete e-mail address as password.
230-You are user #10 of 16 simultaneous users allowed.
230-
230 Logged in anonymously.
Remote system type is UNIX.
Using binary mode to transfer files.
local: ncftp-3.1.5-src.tar.bz2 remote: ncftp/ncftp-3.1.5-src.tar.bz2
502 Unimplemented command.
227 Entering Passive Mode (209,197,102,38,212,218)
150 Data connection accepted from 12.253.112.102:49236; transfer starting for
ncftp-3.1.5-src.tar.bz2 (394777 bytes).
100% ***********************************************   385 KB  266.14 KB/s
00:00 ETA
226 Transfer completed.
394777 bytes received in 00:01 (262.39 KB/s)
221 Goodbye.
-rw-r--r--  1 taylor  staff  394777 Oct 13 20:32 ncftp-3.1.5-src.tar.bz2

If your ftp is excessively verbose, you can quiet it down by adding a - V flag to the ftp invocation (that is, instead of ftp - n , use ftp - nV ).

An alternative to ftpget

Worth noting is that there's a popular utility called curl that performs the same task as ftpget . If you have curl available, it's a superior alternative to this script, but because we're going to build upon the ideas embodied in ftpget for more sophisticated ftp interactions later in this book, you'll benefit from studying the code here.

Hacking the Script

This script can be expanded to uncompress the downloaded file automatically (see Script #37, Working with Compressed Files , for an example of how to do this).

You can also tweak this script just a bit and end up with a simple tool for uploading a specified file to an FTP server. If the server supports anonymous connections (few do nowadays, thanks to skript kiddies and other delinquents, but that's another story), all you really have to do is specify a destination directory on the command line (or in the script) and change the get to a put in the main script:

ftp -n << EOF
open $server
user ftp $anonpass
cd $destdir
put $filename
quit
EOF

To work with a password-protected account, you could hard-code your password into the script ” a very bad idea ” or you could have the script prompt for the password interactively. To do that, turn off echoing before a read statement, and then turn it back on when you're done:

echo -n "Password for ${user}: "
stty -echo
read password
stty echo
echo ""

A smarter way to prompt for a password, however, is to just let the ftp program do the work itself, as demonstrated in Script #81, Synchronizing Directories with FTP.