Section 4.2. Creating Your APT Repository

4.2. Creating Your APT Repository

In Chapter 3, "SUSE's Update Systems and rsync Mirrors," you learned how to create an update repository for computers loaded with SUSE Linux by using appropriate rsync commands. You don't need and do not have to configure a complete repository on your local network. Debian includes packages for the widest variety of architectures, so you need to know how to limit the downloadsto help manage the demands required on your time, bandwidth, and disk space.

4.2.1. Debian Repository Mirror Options

There are several different tools that you can use to create a local repository for apt-based updates for distributions such as Debian Linux. Most of these tools use either rsync or other commands that follow the same principles. The advantage of such commands is during updates; only changes to individual packages are downloaded, so updates are quicker. The options are outlined in the following list:

Anonymous FTP was the traditional method for setting up a mirror. With the right client, you can select the directories to mirror and limit updates to newer packages. However, any package that is installed must be completely installed.

The basic rsync command may be a bit complex, but when configured you can repeat the same command again to keep your mirror up to date. Shortly, you'll examine some appropriate switches to use with rsync.

You can configure a partial mirror. For more information, see people.debian.org/~debacle/mirror.html.

The apt-proxy package is functionally similar to the Red Hat Network Proxy Server described in Chapter 2, "Consolidating Patches on a Red Hat/Fedora Network." As of this writing, however, apt-proxy is available only for the now obsolete Debian Woody. While work on apt-proxy version 2 is in progress, there is no working Debian version available. For the latest information, see apt-proxy.sourceforge.net.

While you might also read about apt-cacher and the aptcached daemon, some of the domain names associated with these daemons, including www.apt-cacher.org, were unused as of this writing.

The debmirror package can help you download complete or partial mirrors. You can read more about the associated commands later in this chapter.

The apt-mirror package is one more option for creating the mirror that you need for your local network. You can read more about the associated commands later in this chapter.

These are just a few of the available packages which can help you create a local repository for your network. For more information, see www.debian.org/mirror/ftpmirror.

4.2.2. A Complete Debian Repository

Most Debian repositories include directories for each of the active distribution versions, as well as a substantial number of different computer architectures. Unless there are computers on your network that run different Debian distributions on each of the available architectures, you might not want to mirror the complete repository. If you do, you might need well over 100GB of free space on your hard drive. Downloading this much data may overload most small business Internet connections for days.

Note

If you want to configure a complete Debian mirror to share on the Internet, the people behind Debian ask that you register your mirror at www.debian.org/mirror/submit.

Debian Directories

The major Debian directories correspond to available distributions and architectures. Before you create a mirror, it may be helpful to review what's available in each category on the mirror of your choice. Where there are several names for the same distribution, you may find more than one directory, each pointing to the same set of packages. The current major Debian distributions include Debian Woody, Debian Sarge, Debian Etch, and Debian Sid.

Debian Woody is now the obsolete distribution. As of this writing, the current version of this distribution is 3.0r6. There may be directories on a Debian mirror known as woody, oldstable, or Debian3.0r6. (Debian Potato came before Debian Woody.)

Debian Sarge, as of this writing, is now the most recent stable version of Debian Linux. As of this writing, the current version of this distribution is 3.1r0. There may be directories on a Debian mirror known as sarge, stable, or Debian3.1r0.

Debian Etch, as of this writing, is now the beta, or testing version of Debian Linux. There may be directories on a Debian mirror known as etch or testing.

Debian Sid is the developmental version of Debian Linux. There may be directories on a Debian mirror known as sid, or unstable.

Under each distribution directory, Debian maintainers and developers build and incorporate binary and source packages for a number of different architectures, as noted in Table 4-5.

Table 4-5. Debian architectures

Architecture	Description
alpha	For computers with the HP (formerly Compaq/Digital) CPU.
arm	Associated with computers with Advanced (formerly Acorn) RISC (Reduced Instruction Set Computing) Machine processors; commonly available on handheld computers.
hppa	Short for Hewlett-Packard Precision Architecture; associated with HP's line of RISC CPUs.
i386	The standard architecture associated with 32-bit Intel and compatible CPUs.
ia64	Associated with Itanium and Itanium-2 64-bit CPUs developed by Intel.
m68k	The architecture for Motorola's 68000 series of CPUs; not developed beyond 32-bit.
mips	Another RISC CPU commonly used in consumer electronics. Sometimes associated with SGI Indy/Indigo2 workstations. MIPS is short for Microprocessor without Interlocked Pipeline Stages.
mipsel	Also for the MIPS CPU; but customized for HP/Cobalt workstations.
powerpc	One more RISC CPU, developed by Motorola, IBM, and Apple; available in a variety of different computers.
s390	Also known as the Z-series of CPUs; developed by IBM originally for the System 390 servers.
sparc	Associated with the Sparc CPUs developed for SUN workstations.

If the right distribution or architecture is not available on your preferred mirror, you'll need to select and configure a different mirror.

A Basic rsync Script

Debian makes a basic rsync script available at www.debian.org/mirror/ftpmirror. The script helps you create a mirror for the distribution and architecture of your choice. Debian recommends that you run the rsync command with at least the following switches:

--recursive Downloads and synchronizes all files from subdirectories.

--times Preserves the date and time associated with each file.

--links Recreates any existing symlinks from the mirror.

--hard-links Preserves any existing hard links from the mirror.

--delete Removes any files on your computer that no longer exist on the target mirror. (If you have sufficient space for multiple versions of a package, use --delete-after.)

If you don't have computers with all the Debian architectures on your network, you might not want to download every directory from your preferred Debian mirror. The following --exclude switches make it possible to limit your download. As you can see from the rsync man page, the following switches --exclude the noted directories and packages with the noted extensions. Be careful. While a Debian mirror is supposed to use directories, such as binary-i386, your selected mirror may not use the same directories. If in doubt, search around using the rsync commands described in Chapter 3.

--exclude binary-alpha/ --exclude *_alpha.deb --exclude binary-arm/ --exclude *_arm.deb --exclude binary-hppa/ --exclude *_hppa.deb --exclude binary-i386/ --exclude *_i386.deb --exclude binary-ia64/ --exclude *_ia64.deb --exclude binary-m68k/ --exclude *_m68k.deb --exclude binary-mips/ --exclude *_mips.deb --exclude binary-mipsel/ --exclude *_mipsel.deb --exclude binary-powerpc/ --exclude *_powerpc.deb --exclude binary-s390/ --exclude *_s390.deb --exclude binary-sparc/ --exclude *_sparc.deb 

Don't include every switch and option in this list; if you do, you're excluding the directory and package extensions associated with every current Debian architecture. You've read about some of these directories along with those related to major repositories earlier in this chapter. See Table 4-2 for some of the options.

4.2.3. Creating a Debian Mirror

You may not want to use rsync or the associated script just described. You can use the packages you've downloaded and installed on your own computer with commands such as apt-get -u dselect-upgrade. If you've followed the instructions associated with the dpkg-scanpackages command described earlier in this chapter, you can create a local repository with those packages.

In this section, we'll also use two alternative tools. One is debmirror, which allows you to download and synchronize part of a mirror to your own repository with a minimum of configuration. The other is apt-mirror, which uses configuration files similar to the standard /etc/apt/sources.list configuration file.

Note

Before you change the /etc/apt/sources.list to point to the local mirror on your network, you should test your configuration on at least one client. Back up that client before updating your system.

Using Local Packages

If you've upgraded your system with download packages, you may still have them stored in the default download directory, /var/cache/apt/archives/. With these packages and the dpkg-scanpackages command, you can create your own local repository.

Assuming you've installed the Debian apache2 Web server (and associated packages), you could configure a repository on Debian Linux (Sarge) with the following commands:

cd /var/cache/apt/archives dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz mkdir /var/www/apache2-default/pub cp -ar /var/cache/apt/archives/*  /var/www/apache2-default/pub/ 

Don't forget the dot (.) after dpkg-scanpackages; it applies the command to all files in the current directory.

You could then configure your Apache Web server for that specialized repository. To do so, you could point your clients to that repository. Test it first! If you were to point a client to a repository on web.example.com, you would add the following command to your /etc/apt/sources.list:

deb http://web.example.com/pub/ 

If you limit clients to that repository, you deny access to those Debian mirrors available on the Internet. But in some cases, that may be exactly what you want to do.

Configuring debmirror

The debmirror package, developed by Joey Hess and Joerg Wendland, is very simple, which makes it a popular option among many Debian users. Before you can configure debmirror, you should download this package, available from the main Sarge repository. Assuming that you've configured /etc/apt/sources.list appropriately, you can download and install it with the following commands:

apt-get update apt-get install debmirror 

As you can see from a dpkg -L debmirror command, this package contains six files. Only two of them are important. The /usr/bin/debmirror file is a script written in Perl that performs three basic steps:

1.	Download packages and source files associated with your selected architecture.
2.	Delete any local files and directories not on the remote mirror.
3.	Download all other files associated with your selected architecture.

These steps are based on default settings in /usr/share/doc/debmirror/debmirror.conf, which downloads a complete mirror associated with the i386 architecture from ftp.debian.org. But as we've emphasized before, it's best for all concerned if you download from a mirror closer to your location.

To override these settings, make a copy of this file in /etc/debmirror.conf. If you configure debmirror, all you need to do is run the debmirror command to synchronize the mirror to the directory of your choice. For example, the following command synchronizes the mirror to the Debian directory associated with a standard FTP server:

debmirror /srv/ftp/pub 

That looks easy. Yes, you can add options as defined in the debmirror man page. But it isn't necessary. First, copy the configuration file:

cp /usr/share/doc/debmirror/debmirror.conf /etc/ 

Now open /etc/debmirror.conf in the text editor of your choice. The settings that you see in this file should correspond to defaults, which you can change to meet your own requirements. Let's examine these defaults, starting with the output options, which correspond to the -v, -p, and -debug switches. At least until I'm sure that the settings work, I like to change the $verbose variable:

$verbose=0; $progress=0; $debug=0; 

If you don't want to change these variables, you can use the switches associated with debmirror. Next, there are download options. While download mirrors generally support anonymous access, you should change the host to the Debian mirror of your choice. Make sure that the $remoteroot directory corresponds to the Debian mirror, and make sure that you're accessing the correct distribution (woody, sarge, etch, or sid):

$host="ftp.debian.org" ; $user="anonymous"; $remoteroot=" /debian"; $download_method="ftp"; @dists="sid"; 

Make sure that you download the correct sections as defined in Table 4-2, and work with the appropriate architecture. The following default downloads all base sections (except non-US). While you're testing debmirror, you may want to limit the download to one specific section:

@sections="main,main/debian-installer,contrib,non-free"; 

The settings that follow correspond to the --skippackages switch, which avoids repeating the download of packages or source files; --getcontents, which downloads the compressed contents archives; and --source, which downloads source packages and avoids limits on the number of files to download. I normally change the $do_source variable because I don't normally download source packages:

$skippackages=0; $getcontents=0; $do_source=1; $max_batch=0; 

You can configure how debmirror checks downloaded files. The following settings check for a Release.gpg file, a Release file, and report all errors. However, the defaults do not check MD5 sums of each file:

$ignore_release_gpg=0; $ignore_release=0; $check_md5sums=0; $ignore_small_errors=0; 

By default, debmirror deletes files and directories no longer on the remote mirror during the download process, not after the download is complete:

$cleanup=1; $post_cleanup=0; 

If you're using debmirror on a rsync server, the default options download up to 200 files at a time and use the rsync -aIL and --partial options. Downloads are performed in passive mode; the dry run option is disabled.

$rsync_batch=200; $rsync_options="-aIL partial"; $passive=0; $dry_run=0; 

When you've configured /etc/debmirror.conf to your liking, you can test the result, by downloading your selections to the directory of your choice with a command, such as

debmirror /srv/ftp/pub 

Afterwards, you can check the result in the appropriate /srv/ftp/pub subdirectories, and download as complete a mirror as you need for your network.

Configuring apt-mirror

There are several other excellent packages that can help you create a local mirror for your network. The other one we'll explore here is apt-mirror; for more information, see http://apt-mirror.sourceforge.net. One advantage of apt-mirror is the preconfigured daily cron job which keeps your mirror up to date. As of this writing, apt-mirror is not available from the standard Debian repositories. Therefore, if you want to download apt-mirror, you'll need to download it from its Source Forge home page or use the associated apt repository.

To install apt-mirror using the apt interface, you'll need to take the following steps:

1.	Add the apt-mirror repository to your /etc/apt/sources.list file. To do so, add the following line: deb http://apt-mirror.sourceforge.net/ apt-mirror/
2.	Update your database: apt-get update
3.	Install apt-mirror: apt-get install apt-mirror

Now you can configure how apt-mirror works. The configuration file, mirror.list, by default, is installed in the /etc/apt/ directory. Let's examine what you should do with this configuration file.

By default, the base directory is /var/spool/apt-mirror. The mirror files are downloaded to the mirror subdirectory. Indexes are downloaded to the skel subdirectory. Logs, MD5 checksums, and URLs are stored in the var subdirectory. If you want to change these defaults, activate and modify the following commands appropriately in mirror.list:

# set base_path /var/spool/apt-mirror # set mirror_path $base_path/mirror # set skel_path $base_path/skel # set var_path $base_path/var 

The default version of this file points to Woody and Sid mirrors in Finland. Unless you're located in or near that country, and have Debian Woody and Sid workstations on your network, you should change at least some of the defaults. Remember, the current standard is Debian Sarge.

Note

The apt-mirror package requires the use of the HTTP protocol. While most Debian mirrors are configured for FTP, many work using identical URLs with HTTP. For example, while the Debian mirror list points to ftp://debian.oregonstate.edu/debian, you can use http://debian.oregonstate.edu/debian. Type the URL in your browser to make sure it works on the mirror of your choice.

If you've updated your /etc/apt/sources.list to point to faster mirrors closer to your location, you can generally copy this list to your /etc/apt/mirror.list file. For example, the following list may be suitable for many users in the Northwest region of the USA:

deb http://debian.oregonstate.edu/debian stable main contrib non-free deb http://security.debian.org/ stable/updates main contrib non-free deb http://debian.oregonstate.edu/debian stable main/debian-installer 

Note

Because security updates are sporadic, Debian encourages direct updates from security.debian.org. There are few public mirrors available for Debian security updates. Nevertheless, you may still want to mirror those updates for your own network for all the reasons described in this book.

The remaining lines in this file come into play only if you're upgrading your version of Debian Linux, such as from Woody to Sarge. They allow apt-mirror to delete those directories associated with the older version of Debian Linux. Naturally, you should modify the URLs to match those that you use in other parts of this file.

If you've properly configured /etc/apt/mirror.list, all you need to do is run apt-mirror to start the mirror synchronization process. Some trial and error may be required, as your favorite mirror may not include a critical database file, such as Packages.gz and Sources.gz.

When you run apt-mirror, it starts two threads for each of the directories you've specified in your /etc/apt/mirror.list, as shown here:

Downloading 18 index files using 18 threads... Begin time: Wed Apr  6 10:55:31 2005 [18]... [17]... [16]... [15]... [14]... [13]... [12]... [11]... [10]... [9]... [8]... [7]... [6]... [5]... [4]... [3]... [2]... [1]... [0]... End time: Wed Apr  6 10:55:51 2005 Proceed indexes: [PP] 9458573242 bytes will be downloaded into archive. Downloading 9629 archive files using 20 threads... Begin time: Wed Apr 6 10:55:55 2005 [20]... [19]... 

Note the number of bytes to be downloaded. From what I specified in my /etc/apt/mirror.list, that's over 9GB! That download will take some time.

After you've run apt-mirror, keep the mirror on your network up to date. To this end, the apt-mirror package includes a daily cron job. If you activate the embedded command in /etc/cron.d/apt-mirror, it runs apt-mirror by default at 4 a.m. every day. Make sure the update time is appropriate, relative to any other cron jobs on your system. Don't worry, apt-mirror downloads only those packages that have changed since the last update.

Finally, update the clients on your network to get their updates from your local mirror. That means revising the /etc/apt/sources.list configuration file on each of your Debian client computers.