Customizing Download Sources

Many pieces of popular software are available scattered all over the Internet. If you want to download the source code to the KDE window manager, for example, a manual search would find dozens of mirrors. One of them is probably very close to you. Some collections of software are mirrored as a whole and have a collective name. For example, official GNU software is generally distributed on GNU mirrors, many pieces of software are available in the widely redistributed Sunsite collection, and so on. Many ports use these mirrors as the authoritative source for distfiles.

The location of these mirror collections changes over time, and this month's most reliable mirror will not necessarily be good six months from now. Continually updating all the ports that use these sites would be a very difficult task. Instead, the OpenBSD ports team uses a series of macros to define these sites and has the ports call the macros to learn where to get the software. You can use these macros yourself to choose convenient places to download the software.

Each port Makefile includes a definition for MASTER_SITE, a URL where the source code can be downloaded. For example, the Makefile for /usr/ports/ shells/tcsh contains the following line:

 MASTER_SITES=    ftp://ftp.astron.com/pub/tcsh/ 

If you go look at this FTP site, you'll find the source code for this particular release of tcsh.

Other ports rely on a mirror site macro to get their download locations.

 MASTER_SITES=          ${MASTER_SITE_WINDOWMAKER:=source/release/} 

The WindowMaker window manager source code repository is widely mirrored, contains a great deal of software, and is well known enough to rate having its own name. You can call something "an official WindowMaker mirror," and people will generally know what you're talking about.

OpenBSD includes a set of macros that defines the common mirror collections and lists servers that mirror that content. A port will try the mirrors in order until it can find one where it can download the code. As with any collection of mirror sites, some will respond more quickly to you than others. You may find that the mirror site at the top of the list is painfully slow and that there's a more responsive mirror that you'd like to try first. That's fairly easy with OpenBSD.

If you look at the file /usr/ports/infrastructure/templates/ network.conf.template, you'll find a whole list of MASTER_SITE_ variables, including MASTER_SITE_WINDOWMAKER, MASTER_SITE_KDE, MASTER_SITE_SUNSITE, and, yes, MASTER_SITE_OPENBSD. Don't directly edit this file — it's considered an OpenBSD system file and will be overwritten when you upgrade your system. Instead, copy it to /usr/ports/db/network.conf and make your changes to the copy.

For example, you'll find the following entry in network.conf:

 ... MASTER_SITE_SOURCEFORGE+=       \         http://us.dl.sourceforge.net/ \         http://eu.dl.sourceforge.net/ \         ftp://us.dl.sourceforge.net/pub/sourceforge/ \         ftp://ftp.kddlabs.co.jp/sourceforge/ \         ftp://ftp.chg.ru/pub/sourceforge/ ... 

This is a list of the SourceForge mirrors that the OpenBSD ports system knows about. When you install a port that downloads from a SourceForge site, the port first tries to grab the software from http://us.dl.sourceforge.net/. If the software is not available there, it then tries http://eu.dl.sourceforge.net/, and then proceeds on down the list to other U.S., Japanese, and Russian SourceForge mirrors. If you happen to be in Europe, you might rightly think that trying a US site first is rather dumb. Edit the list and put your preferred mirror site first, making sure to preserve the trailing slash on the URL and the final trailing backslash on the line, so that the port will try the other sites if the first one doesn't work. You can also add a mirror not on the list, if that works well for you.