Download Sequential Files and Internet Resources

curl

At first blush, wget and curl seem similar: Both download files non-interactively. They each have one large difference distinguishing them, however, among many smaller ones: curl supports sequences and sets in specifying what to download, which wget does not, while wget supports recursion, a feature missing from curl.

Note

The programs have plenty of other differences. The full list of curl's features can be seen at "FeaturesWhat Can curl Do" (http://curl.haxx.se/docs/features.html), while some of wget's are listed at "Overview" (www.gnu.org/software/wget/manual/html_node/Overview.html#Overview). The cURL site has a chart comparing curl to other, similar programs at "Compare cURL Features with Other FTP+HTTP Tools" (http://curl.haxx.se/docs/comparison-table.html); while informative, the chart is (unsurprisingly) a bit biased toward curl.

Here's an example that uses curl's capability to support sequences in specifying what to download. The excellent National Public Radio show This American Life makes archives of all of its shows available for download on its parent website in Real Audio format (why they chose Real and not a more open format is a mystery). If you want to download 10 of these Real Audio files, just use the following:

$ curl -O http://www.wbez.org/ta/[1-10].rm [1/10]: http://www.wbez.org/ta/1.rm --> 1.rm --_curl_--http://www.wbez.org/ta/1.rm 

Notice how you used [1-10].rm to specify that you wanted to download 1.rm, 2.rm, 3.rm, and so on. If WBEZ had instead named the files one.rm, two.rm, and three.rm, for example, you could have used a part set instead:

$ curl -O http://www.wbez.org/ta/{one,two,three}.rm 

The -O (or --remote-name) option is absolutely required. If you don't use it, curl writes the output of the download to STDOUT, which means that your terminal will quickly fill with unusable goobledygook. The -O asks curl to write out what it downloads to a file, and to use the name of the file being downloaded as the local filename as well.

We've barely scratched the surface of curl. Its man page, while not as long as wget's, is also full of useful information that you need to read if you're going to maximize your use of curl. Consider it required reading.