2.3. The Basic Sandbox: chroot

As Unix systems evolved over time, more and more defects were found in critical system software. When software ran with elevated privileges, those defects could often be parlayed into security compromises. Sometimes a program might divulge the contents of a protected file, and sometimes it would execute a system call with unauthorized parameters. The change root (chroot(2)) system call was invented as a defense against this sort of attack. If the privileged software goes awry, this secondary defense tries to limit the damage that might result.

Be aware that both chroot and jail have two different sets of manpages. There are chroot(2) and jail(2) system calls as well as chroot(8) and jail(8) commands.

The principles behind chroot are simple. A process running in a chrooted environment sees a normal filesystem, but it in fact has a virtual root directory. The goal is to prevent the process from accessing files outside its sandbox. So, if ntpd runs in a chrooted environment, for example, and an exploit is discovered that causes it to overwrite a file, files in the real filesystem should be protected. The daemon perceives a / directory and will write relative to that directory, but on the real filesystem, the directory is something like /var/ntpd, and the daemon cannot actually reach the real / directory.

The archetypal use of chroot is for the FTP daemon, ftpd(8). It allows anonymous users to traverse part of the filesystem and download files. For obvious reasons, it should not allow them to traverse the entire real filesystem anonymously. To run correctly, however, the daemon needs more than just the files it is supposed to serve. It needs files like /etc/passwd and /etc/group to map numeric UIDs and GIDs to user login names and groups, /etc/localtime to display times in the correct time zone, and /etc/motd to define a "message of the day" to users who connect. In order to get the FTP daemon to run correctly in a chroot environment, then, it takes some configuration and planning.

2.3.1. Creating a chroot Environment

It takes a bit of work, and sometimes some in-depth knowledge of the software you're installing to properly build a chroot environment for it. Figure 2-1 shows an example of a real filesystem, with a chroot virtual filesystem beginning at a /jail directory on a real filesystem. Most software that is amenable to chroot will tell you, in their manpages or other documentation, what they require in order to run correctly. If they don't give you explicit instructions, expect to spend some time in trial-and-error attempts. We discuss some techniques for determining software needs in the section "Managing jails."

Figure 2-1. A chroot filesystem

First, you must create a directory hierarchy with all the subdirectories that need to exist. In this case, /jail with subdirectories bin, etc, lib, usr, and web. We're chrooting a web server, so we're going to put all the web server's binaries, configuration, and data in the web directory.

Then copy the software and all its data into the chroot area. Depending on how the software will execute, this usually requires copying shared libraries (e.g., the C run-time library /usr/lib/libc.so.4), configuration files, and datafiles. If the software normally expects its configuration files to be in /etc/software.conf, then install the file in /jail/etc/software.conf.

Most software capable of taking advantage of chroot will save you some of this effort. Either by statically linking at compile time, or by dynamically loading all its shared libraries before calling chroot, it reduces the number of files that have to be stored in special chroot filesystems. BIND 9, for example, does exactly this, alleviating the need to copy a bunch of dependent libraries into a virtual filesystem.

2.3.2. An Example: chrooting ntpd

When a program does not have native support for chroot(2), the way BIND and ftpd do, you can use the chroot(8) command to impose the restrictions on it. To show how this is done, we will take a program in FreeBSD, ntpd, that does not support chroot and launch it chrooted. There have been historically very few bugs with ntpd relative to its ubiquity, and OpenBSD has already imposed privilege separation on its ntpd, so this is a somewhat contrived example. We start out by creating a /jail/ntpd virtual filesystem. Normally we would create a separate user associated with ntpd and make all the files in the jail owned by that user (OpenBSD has an _ntpd user for this purpose). In this case, we can't do that because only root can change the clock. The ntpd process needs to run as root.

If the process must run as root to set the clock, what is the _ntpd user for? The OpenBSD team wrote their own NTP daemon that prominently features privilege separation. It drops its privileges for most mundane operations (like DNS lookup and parsing network messages). This is an excellent example of how a simple design and the application of a security principle can obviate the need for complex, error-prone configurations.

First, let's just take a crack at it. We know a few things have to exist in our chroot environment: the /usr/sbin/ntpd binary and its configuration file /etc/ntp.conf. We make our virtual root directory with these two files and run the command with proper syntax just to see what happens, as shown in Example 2-5.

Example 2-5. Make an ntpd chroot environment and see what happens

% sudo mkdir -p /jail/ntpd/usr/sbin /jail/ntpd/etc % sudo cp /usr/sbin/ntpd /jail/ntpd/usr/sbin % sudo cp /etc/ntp.conf /jail/etc/ntp.conf % sudo chroot /jail/ntpd /usr/sbin/ntpd ELF interpreter /libexec/ld-elf.so.1 not found

This tells us two things: first, the program is dynamically linked; second, we have to track down its library dependencies. If ntpd were statically linked, all the instructions would be stored in the binary file. The operating system would essentially load the ntpd file into memory and execute it. Because it is dynamically linked, it borrows instructions from various other libraries. Normally dynamic linking is a good thing. It makes the programs smaller on disk and in RAM. This lets your system benefit more from its caches because the same RAM pages and disk blocks are being used by multiple programs. When dealing with a chroot (or, as we will see, with a jail), dynamic linking makes it a little harder to isolate the program. We have to identify every library that ntpd loads, and we have to put a copy of the library in the virtual filesystem.

Rather than find all the libraries by trial and error, use ldd(1) to interrogate the binary as shown in Example 2-6.

Example 2-6. Using ldd to determine library dependencies

% ldd /usr/sbin/ntpd /usr/sbin/ntpd:         libm.so.3 => /lib/libm.so.3 (0x280ae000)         libmd.so.2 => /lib/libmd.so.2 (0x280c8000)         libcrypto.so.3 => /lib/libcrypto.so.3 (0x280d2000)         libc.so.5 => /lib/libc.so.5 (0x281c7000)

Copy /libexec/ld-elf.so.1 and these other library files into /jail/ntpd and try again. You'll see that it works! Now, ntpd is a pretty straightforward program, so chrooting it was equally straightforward. For other programs, you often need more files than just the shared libraries. If ntpd had complained about other missing files, we would have used some of the techniques described in the next section, "Finding Other Dependencies."

You might wonder if you gained anything by chrooting ntpd. While chroot is potentially escapable, especially if the process runs as root, you did create an additional hurdle for an attacker who would exploit your ntpd. He must not only create a successful exploit for the daemon, but he also has to customize his attack to escape your chroot environment. This is just another layer of defense, in line with our defense-in- depth principle.

A dedicated hacker who is specifically targeting your system for some reason might spend the extra time and effort to break out of a chroot. The so-called "script kiddies," on the other hand, who run automated discovery and exploit tools will probably pass over your system since they'd rather spend their time on low hanging fruit. Ultimately, you have to consider the level of hostility you face and decide whether the increase in security was worth the effort of chrooting.

2.3.3. Finding Other Dependencies

If you try to run programs in a chrooted environment and they fail mysteriously, the ktrace(1) and kdump(1) commands come in handy.

To use ktrace, simply put it on the command line first, like you would for sudo(8). A file named ktrace.out will be produced in the directory in which you run kTRace. You can then use the kdump command to parse that file. The first things to look for are files that the program you're trying to chroot is attempting to open. Look for the name-to-inode (NAMI) translation (e.g., run kdump | less and then search interactively for "NAMI"). This makes a good starting point because you often see files that you know don't exist and you can work on installing those files into your virtual filesystem.

See Table 2-4 for some red herrings that you don't need to track down.

If the NAMI translations don't give you enough of a clue to figure out what files you need in the virtual filesystem, look more interactively at the kdump output for more information. Example 2-7 shows some of the output from trying to open a file named foo that is not readable by the current user:

Example 2-7. kdump output

52240 cat      CALL  open(0xbfbffc8c,0,0) 52240 cat      NAMI  "foo" 52240 cat      RET   open -1 errno 13 Permission denied

Often the kinds of problems you run into when trying to install software into chroot are configuration files or shared libraries that do not exist in the right path. By running kdump and ktrace multiple times, you can zero in on the files you need and where they need to be. If you're working with a daemon process like Sendmail or Apache that forks child processes, you can run ktrace -di to follow its descendents. Also check for application-specific options that may prevent the program from running in daemon mode. Many daemons have options to stay in the foreground and log extra information.

2.3.3.1 Sorting through kdump's output

You don't have to track down each and every file that the program tries to open. Sometimes, for example with libcrypto.so, you'll see the program try opening it in a variety of different directories until it finally finds it. The library is ultimately found, so it really doesn't matter where it's found. Other files, like /etc/malloc.conf, don't actually matter most of the time. They should only exist if you really are trying to modify malloc(2)'s behavior in your chroot or jail environment. Table 2-4 lists files that you'll frequently need in either a jail or chroot environment. It also lists some files that you might see in the output of kdump that aren't usually important. If your program dies because it can't open a file, look at the last error in the kdump output and work backward from there. Don't start with files that aren't found if the program keeps running after failing to find the file.

Table 2-4. Files related to chroot and jail

Filename	Description
/lib/ld-elf.so.1 (FreeBSD) /usr/libexec/ld.so (OpenBSD)	The runtime loader for the operating system. If you intend to run any dynamically linked programs, you will have to provide this file in this location inside the virtual environment. You will also need one or more dynamic libraries.
/lib/libc.so.5 /usr/lib/libc_r.so.5 /usr/lib/libstdc++.so.4 /lib/libm.so.3 /lib/libcrypto.so.3 /lib/libcrypt.so.2 /usr/lib/libpthread.so.1	Dynamic libraries that you frequently need on FreeBSD, including the C and C++ runtime libraries, the standard math library, the cryptography library, and pthread library. The libc_r.so.5 library is a reentrant version of the standard C runtime environment that is used by programs that use FreeBSD's native threads instead of pthreads.
/usr/lib/libc.so.34.1 /usr/lib/libstdc++.so.33.0 /usr/lib/libm.so.2.0 /usr/lib/libcrypto.so.11.0 /usr/lib/libpthread.so.6.0	Dynamic libraries that you frequently need on OpenBSD, including the C and C++ runtime libraries, the standard math library, the cryptography library, and pthread library.
/etc/pwd.db	A hashed version of the traditional /etc/passwd file. Used frequently to map numeric UIDs to names. Does not actually contain encrypted versions of passwords.
/etc/spwd.db	A hashed version of the /etc/master.passwd file, just like pwd.db, but has encrypted versions of passwords. Might be needed if software in your chroot environment needs to authenticate users (e.g., an IMAP daemon).
/etc/localtime	Specifies the local time zone. Allows log entries to carry a properly localized timestamp.
/var/run/ld-elf.so.hints	A hints file used by the runtime loader (ld.so). It speeds up the loading of programs slightly, if it exists, but you don't have to have it.
/etc/libmap.conf (FreeBSD only)	An optional file that changes how ld.so loads dynamic libraries. Again, you might use it in unusual circumstances, but most normal installations won't use it and it will show up as a file not found in the kTRace output.
/etc/resolv.conf	Sometimes you might want DNS resolution to act differently inside your chroot or jail than for the rest of the operating system, or you might have software (e.g., Postfix) that requires this file.
/etc/malloc.conf	This file controls options for the malloc(2) system call. See the malloc.conf(5) manpage for more information.
/dev/null /dev/random /dev/urandom /dev/srandom /dev/zero /dev/mem /dev/kmem	These are device files that you commonly need in a chroot environment. Don't create them all just in case. Create just the ones you need.

2.3.3.2 Making device nodes

If you run ls -l on the device you're interested in, you'll see everything you need to know to make a copy of it. For example, if your software needs /dev/random in /jail/dev, Example 2-8 shows how you look up the major and minor modes and then run the mknod(8) command with the right parameters.

Example 2-8. Making devices in chroot environments

FreeBSD% ls -l /dev/random crw-rw-rw-  1 root  wheel  249,   0 Nov 25 14:04 /dev/random FreeBSD% sudo mknod /jail/dev/random c 249 0 OpenBSD% ls -l /dev/random crw-r--r--  1 root  wheel   45,   0 Nov  3 17:15 /dev/random OpenBSD% sudo mknod /jail/dev/random c 45 0

2.3.4. Limitations of chroot

For almost as long as it has been around, chroot has been the focal point of an arms race between hackers and programmers in the Unix world. The programmers try to make chroot inescapable (hence their frequent use of the term "jail" to refer to it), and hackers invent increasingly complex ways of escaping.

For instance, directory and file handling in chrooted processes has traditionally been a source of chroot escape tricks. Directories and files that were open prior to calling chroot(2) remain open and available to software after calling chroot. If the software is not fastidious about closing them all before calling chroot, it may be possible later to use open directories to break out. Another technique for reading files outside the chroot environment involves calling mknod(8) to create a disk device file (e.g., an rwd0a or da0s1a node) and then opening it and finding the superblock for the filesystem. The primary superblock is always at a well-known offset (sector 32) from the start of the disk partition. The process can traverse the full filesystem of that device by manually decoding inodes and seeking to the right blocks on the disk. The inode number for the root directory is also always 2. This means that processes can always tell, simply by looking at the inode number of the root directory, whether or not they are seeing a virtualized filesystem.

Even though a chrooted process sees a virtualized filesystem, it is otherwise unrestricted. This means that, if it runs as root, it can do almost anything it wants. For instance, a chrooted process can still send signals to other processes using kill(2), or create device nodes using mknod(2). Under FreeBSD, you can set the sysctl variables security.bsd.see_other_uids=0 and security.bsd.see_other_gids=0 to hide processes that do not have the same UID or GID as the chroot'ed process. But this only helps a little. If the chroot'ed process knows the PID of another process it wants to signal, it can still use kill() to send the signal. These sysctl variables make it harder to find processes, but they don't limit the interprocess interactions. By default, FreeBSD allows all chrooted processes to see all other processes on the system. On OpenBSD this level of control is not supported.

As a rule, chroot is most effective when the chrooted process closes all its open files and directories and drops all its privileges as soon as possible after calling chroot(2). It is worth mentioning that shells are notoriously hard to chroot, even shells that innately try to be restrictive, like rsh, rbash, and rksh. With their willingness to manipulate files and filehandles, plus their rich built-in commands and scripting capabilities, shells are hard to limit.