Project95.Networking Tips

Project 95. Networking Tips

"How do I automate FTP transfers in a script?"

This project gives a few tips on grabbing files from across a network. It covers the curl command. It shows how curl is used to grab files from a Web site by using the HTTP protocol or from an FTP server by using the FTP protocol. It also shows how you might look up dictionary definitions by using the DICT protocol.

The curl command transfers data to or from a server by using one of many protocols: HTTP, FTP, DICT, GOPHER, TELNET, LDAP, HTTPS, or FTPS. We'll show how to transfer files and data by using the HTTP, FTP, and DICT protocols.

The curl command can be incorporated into scripts to fetch files without user intervention. It's a very comprehensive tool, but we'll touch on only the basics.

Grab via HTTP

Let's grab a page from a Web site. We grab the index page from Jan's site by issuing the following command.

$ curl http://jan.1dot1.com

If you try this command, you'll see raw HTML code written to your Terminal window. To capture the output in a file named jan.html, type

$ curl -o jan.html http://jan.1dot1.com   % Total % Received % Xferd Average Speed Time  Time ...                              Dload  Upload Total Spent... 100 2873  0  2873    0     0  6707      0 --:-:- --:-:...

We may fetch any page from the site. In the next example, we fetch the page object.php from the directory tails. By specifying option -O, we tell curl to write the fetched page to a file of the same name (object.php).

$ curl -O http://jan.1dot1.com/tails/object.php

The curl command also supports the HTTP get and post methods, and can fetch and set cookies. As ever, man curl is your friend when you require more information on curl.

Tip

If you experience difficulties connecting to FTP servers, make sure that you have enabled Passive Mode by checking Use Passive FTP Mode in the Proxies tab of each network port configuration shown in the System Preferences Network pane.

Grab and Upload via FTP

curl provides an easy way to download and upload files to an FTP server. Because it requires no user interaction, this functionality is easily incorporated into shell scripts to automate FTP transfers.

Let's grab from an anonymous FTP server (one that does not require you to have an account and password). In this example, we'll simply list the files and directories at the root of the anonymous FTP account.

$ curl ftp://ftp.apple.com drwxrwxrwx  3 ftpprod ftpprod  102 May  7 2003 Apple_... drwxrwxr-x 20 ftpprod ftpprod  680 Mar  8 2005 developer drwxrwxr-x 37 ftpprod ftpprod 1258 May 18 2004 emagic drwxrwxr-x 11 ftpprod ftpprod  374 Mar  9 2004 filemaker drwxrwxrwx 10 ftpprod ftpprod  340 Apr  7 2003 research

To list a directory such as research, name the directory, and add a trailing slash.

$ curl ftp://ftp.apple.com/research/ drwxrwxrwx  5 ftpprod  ftpprod  170 Nov 12  1997 eg drwxrwxrwx  4 ftpprod  ftpprod  136 Nov 19  1997 goe ...

To grab a file such as /research/goe/goe4mac.sea (whatever that might be), type

$ curl -O ftp://ftp.apple.com/research/goe/goe4mac.sea

The options -o and -O have the same meanings as they did for the HTTP protocol.

Connect to an FTP Account

To connect to an FTP account that requires a username and password, we simply supply the necessary information on the command line. To grab a file named test from our account saruman on the FTP server ftp.mayo-family.com, we would type

$ curl -O ftp://saruman:[e-mail]/test

Learn More

To prevent a command line that contains sensitive information, such as a password, from entering the Bash command-line history, refer to Project 48.

Upload Files

FTP servers let you upload files as well as download them (though usually only if you have a proper account). Upload a file with the option -T.

Here, we upload a file named test to our FTP home directory.

$ curl -T test ftp://saruman:[e-mail]

To rename the file at its destination, type

$ curl -T test ftp://saruman:[e-mail]/new

We may also upload the file to a specific directory by specifying a pathname after the server name. To upload a file named test to the directory Documents and call it t01, type

$ curl -T test ftp://saruman:[e-mail]/ ¬     Documents/t01

Use option -v to see the FTP transcript that curl executes in performing its allotted task.

$ curl -v -T test ftp://saruman:[e-mail]/ ¬    Documents/t01 * About to connect() to carcharoth.mayo-family.com port 21 ... > USER saruman < 331 Password required for saruman. > PASS pass ... > CWD Documents < 250 CWD command successful. ... > STOR t01 < 150 Opening BINARY mode data connection for 't01'. ...

Learn More

Refer to Project 52 to learn about Bash functions.

Query a Dictionary

A site named dict.org (among others) understands the DICT dictionary protocol. Look up the definition of apple by typing

$ curl dict://dict.org/d:apple 220 aspen.miranda.org dictd 1.9.15/rf on Linux... 250 ok 150 2 definitions retrieved 151 "Apple" gcide "The Collaborative International Dictionary of English v.0.48" Apple \Ap"ple\ ([a^]p"p'l), n. [OE. appel, AS. [ae]ppel,    [ae]pl; akin to Fries. & D. appel, OHG, aphul, aphol, G. ...

We might define a handy Bash function to encapsulate this functionality by typing

$ defn () { curl dict://dict.org/d:$*; }

and use it to look up Unix by typing

$ defn Unix 220 aspen.miranda.org dictd 1.9.15/rf on Linux ... 250 ok 150 1 definitions retrieved 151 "UNIX" wn "WordNet (r) 2.0" UNIX      n : trademark for a powerful operating system           [syn: {UNIX system}, {UNIX operating system}] ...