4.2. Creating Your APT Repository
In Chapter 3, "SUSE's Update Systems and rsync Mirrors," you learned how to create an update repository for computers loaded with SUSE Linux by using appropriate rsync commands. You don't need and do not have to configure a complete repository on your local network. Debian includes packages for the widest variety of architectures, so you need to know how to limit the downloadsto help manage the demands required on your time, bandwidth, and disk space.
4.2.1. Debian Repository Mirror Options
There are several different tools that you can use to create a local repository for apt-based updates for distributions such as Debian Linux. Most of these tools use either rsync or other commands that follow the same principles. The advantage of such commands is during updates; only changes to individual packages are downloaded, so updates are quicker. The options are outlined in the following list:
These are just a few of the available packages which can help you create a local repository for your network. For more information, see www.debian.org/mirror/ftpmirror.
4.2.2. A Complete Debian Repository
Most Debian repositories include directories for each of the active distribution versions, as well as a substantial number of different computer architectures. Unless there are computers on your network that run different Debian distributions on each of the available architectures, you might not want to mirror the complete repository. If you do, you might need well over 100GB of free space on your hard drive. Downloading this much data may overload most small business Internet connections for days.
If you want to configure a complete Debian mirror to share on the Internet, the people behind Debian ask that you register your mirror at www.debian.org/mirror/submit.
The major Debian directories correspond to available distributions and architectures. Before you create a mirror, it may be helpful to review what's available in each category on the mirror of your choice. Where there are several names for the same distribution, you may find more than one directory, each pointing to the same set of packages. The current major Debian distributions include Debian Woody, Debian Sarge, Debian Etch, and Debian Sid.
Under each distribution directory, Debian maintainers and developers build and incorporate binary and source packages for a number of different architectures, as noted in Table 4-5.
If the right distribution or architecture is not available on your preferred mirror, you'll need to select and configure a different mirror.
A Basic rsync Script
Debian makes a basic rsync script available at www.debian.org/mirror/ftpmirror. The script helps you create a mirror for the distribution and architecture of your choice. Debian recommends that you run the rsync command with at least the following switches:
If you don't have computers with all the Debian architectures on your network, you might not want to download every directory from your preferred Debian mirror. The following --exclude switches make it possible to limit your download. As you can see from the rsync man page, the following switches --exclude the noted directories and packages with the noted extensions. Be careful. While a Debian mirror is supposed to use directories, such as binary-i386, your selected mirror may not use the same directories. If in doubt, search around using the rsync commands described in Chapter 3.
--exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-i386/ --exclude *_i386.deb --exclude binary-ia64/ --exclude *_ia64.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb
Don't include every switch and option in this list; if you do, you're excluding the directory and package extensions associated with every current Debian architecture. You've read about some of these directories along with those related to major repositories earlier in this chapter. See Table 4-2 for some of the options.
4.2.3. Creating a Debian Mirror
You may not want to use rsync or the associated script just described. You can use the packages you've downloaded and installed on your own computer with commands such as apt-get -u dselect-upgrade. If you've followed the instructions associated with the dpkg-scanpackages command described earlier in this chapter, you can create a local repository with those packages.
In this section, we'll also use two alternative tools. One is debmirror, which allows you to download and synchronize part of a mirror to your own repository with a minimum of configuration. The other is apt-mirror, which uses configuration files similar to the standard /etc/apt/sources.list configuration file.
Before you change the /etc/apt/sources.list to point to the local mirror on your network, you should test your configuration on at least one client. Back up that client before updating your system.
Using Local Packages
If you've upgraded your system with download packages, you may still have them stored in the default download directory, /var/cache/apt/archives/. With these packages and the dpkg-scanpackages command, you can create your own local repository.
Assuming you've installed the Debian apache2 Web server (and associated packages), you could configure a repository on Debian Linux (Sarge) with the following commands:
cd /var/cache/apt/archives dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz mkdir /var/www/apache2-default/pub cp -ar /var/cache/apt/archives/* /var/www/apache2-default/pub/
Don't forget the dot (.) after dpkg-scanpackages; it applies the command to all files in the current directory.
You could then configure your Apache Web server for that specialized repository. To do so, you could point your clients to that repository. Test it first! If you were to point a client to a repository on web.example.com, you would add the following command to your /etc/apt/sources.list:
If you limit clients to that repository, you deny access to those Debian mirrors available on the Internet. But in some cases, that may be exactly what you want to do.
The debmirror package, developed by Joey Hess and Joerg Wendland, is very simple, which makes it a popular option among many Debian users. Before you can configure debmirror, you should download this package, available from the main Sarge repository. Assuming that you've configured /etc/apt/sources.list appropriately, you can download and install it with the following commands:
apt-get update apt-get install debmirror
As you can see from a dpkg -L debmirror command, this package contains six files. Only two of them are important. The /usr/bin/debmirror file is a script written in Perl that performs three basic steps:
These steps are based on default settings in /usr/share/doc/debmirror/debmirror.conf, which downloads a complete mirror associated with the i386 architecture from ftp.debian.org. But as we've emphasized before, it's best for all concerned if you download from a mirror closer to your location.
To override these settings, make a copy of this file in /etc/debmirror.conf. If you configure debmirror, all you need to do is run the debmirror command to synchronize the mirror to the directory of your choice. For example, the following command synchronizes the mirror to the Debian directory associated with a standard FTP server:
That looks easy. Yes, you can add options as defined in the debmirror man page. But it isn't necessary. First, copy the configuration file:
cp /usr/share/doc/debmirror/debmirror.conf /etc/
Now open /etc/debmirror.conf in the text editor of your choice. The settings that you see in this file should correspond to defaults, which you can change to meet your own requirements. Let's examine these defaults, starting with the output options, which correspond to the -v, -p, and -debug switches. At least until I'm sure that the settings work, I like to change the $verbose variable:
$verbose=0; $progress=0; $debug=0;
If you don't want to change these variables, you can use the switches associated with debmirror. Next, there are download options. While download mirrors generally support anonymous access, you should change the host to the Debian mirror of your choice. Make sure that the $remoteroot directory corresponds to the Debian mirror, and make sure that you're accessing the correct distribution (woody, sarge, etch, or sid):
$host="ftp.debian.org" ; $user="anonymous"; $remoteroot=" /debian"; $download_method="ftp"; @dists="sid";
Make sure that you download the correct sections as defined in Table 4-2, and work with the appropriate architecture. The following default downloads all base sections (except non-US). While you're testing debmirror, you may want to limit the download to one specific section:
The settings that follow correspond to the --skippackages switch, which avoids repeating the download of packages or source files; --getcontents, which downloads the compressed contents archives; and --source, which downloads source packages and avoids limits on the number of files to download. I normally change the $do_source variable because I don't normally download source packages:
$skippackages=0; $getcontents=0; $do_source=1; $max_batch=0;
You can configure how debmirror checks downloaded files. The following settings check for a Release.gpg file, a Release file, and report all errors. However, the defaults do not check MD5 sums of each file:
$ignore_release_gpg=0; $ignore_release=0; $check_md5sums=0; $ignore_small_errors=0;
By default, debmirror deletes files and directories no longer on the remote mirror during the download process, not after the download is complete:
If you're using debmirror on a rsync server, the default options download up to 200 files at a time and use the rsync -aIL and --partial options. Downloads are performed in passive mode; the dry run option is disabled.
$rsync_batch=200; $rsync_options="-aIL partial"; $passive=0; $dry_run=0;
When you've configured /etc/debmirror.conf to your liking, you can test the result, by downloading your selections to the directory of your choice with a command, such as
Afterwards, you can check the result in the appropriate /srv/ftp/pub subdirectories, and download as complete a mirror as you need for your network.
There are several other excellent packages that can help you create a local mirror for your network. The other one we'll explore here is apt-mirror; for more information, see http://apt-mirror.sourceforge.net. One advantage of apt-mirror is the preconfigured daily cron job which keeps your mirror up to date. As of this writing, apt-mirror is not available from the standard Debian repositories. Therefore, if you want to download apt-mirror, you'll need to download it from its Source Forge home page or use the associated apt repository.
To install apt-mirror using the apt interface, you'll need to take the following steps:
Now you can configure how apt-mirror works. The configuration file, mirror.list, by default, is installed in the /etc/apt/ directory. Let's examine what you should do with this configuration file.
By default, the base directory is /var/spool/apt-mirror. The mirror files are downloaded to the mirror subdirectory. Indexes are downloaded to the skel subdirectory. Logs, MD5 checksums, and URLs are stored in the var subdirectory. If you want to change these defaults, activate and modify the following commands appropriately in mirror.list:
# set base_path /var/spool/apt-mirror # set mirror_path $base_path/mirror # set skel_path $base_path/skel # set var_path $base_path/var
The default version of this file points to Woody and Sid mirrors in Finland. Unless you're located in or near that country, and have Debian Woody and Sid workstations on your network, you should change at least some of the defaults. Remember, the current standard is Debian Sarge.
The apt-mirror package requires the use of the HTTP protocol. While most Debian mirrors are configured for FTP, many work using identical URLs with HTTP. For example, while the Debian mirror list points to ftp://debian.oregonstate.edu/debian, you can use http://debian.oregonstate.edu/debian. Type the URL in your browser to make sure it works on the mirror of your choice.
If you've updated your /etc/apt/sources.list to point to faster mirrors closer to your location, you can generally copy this list to your /etc/apt/mirror.list file. For example, the following list may be suitable for many users in the Northwest region of the USA:
deb http://debian.oregonstate.edu/debian stable main contrib non-free deb http://security.debian.org/ stable/updates main contrib non-free deb http://debian.oregonstate.edu/debian stable main/debian-installer
Because security updates are sporadic, Debian encourages direct updates from security.debian.org. There are few public mirrors available for Debian security updates. Nevertheless, you may still want to mirror those updates for your own network for all the reasons described in this book.
The remaining lines in this file come into play only if you're upgrading your version of Debian Linux, such as from Woody to Sarge. They allow apt-mirror to delete those directories associated with the older version of Debian Linux. Naturally, you should modify the URLs to match those that you use in other parts of this file.
If you've properly configured /etc/apt/mirror.list, all you need to do is run apt-mirror to start the mirror synchronization process. Some trial and error may be required, as your favorite mirror may not include a critical database file, such as Packages.gz and Sources.gz.
When you run apt-mirror, it starts two threads for each of the directories you've specified in your /etc/apt/mirror.list, as shown here:
Downloading 18 index files using 18 threads... Begin time: Wed Apr 6 10:55:31 2005 ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... End time: Wed Apr 6 10:55:51 2005 Proceed indexes: [PP] 9458573242 bytes will be downloaded into archive. Downloading 9629 archive files using 20 threads... Begin time: Wed Apr 6 10:55:55 2005 ... ...
Note the number of bytes to be downloaded. From what I specified in my /etc/apt/mirror.list, that's over 9GB! That download will take some time.
After you've run apt-mirror, keep the mirror on your network up to date. To this end, the apt-mirror package includes a daily cron job. If you activate the embedded command in /etc/cron.d/apt-mirror, it runs apt-mirror by default at 4 a.m. every day. Make sure the update time is appropriate, relative to any other cron jobs on your system. Don't worry, apt-mirror downloads only those packages that have changed since the last update.
Finally, update the clients on your network to get their updates from your local mirror. That means revising the /etc/apt/sources.list configuration file on each of your Debian client computers.