2.4. Jail: Beyond chroot

The well-known defects and deficiencies in chroot(2) motivated FreeBSD developers to develop a more thorough sandbox, which they call jail(2). Whereas chroot works the same on both OpenBSD and FreeBSD, jail exists only on FreeBSD.

Jail Versus chroot

For many years, the only technology available for putting processes in sandboxes was the chroot mechanism. Since "chroot" is a somewhat hard to pronounce, the term "jail" is often used. The terms chroot and jail are interchangeable to most people. You will frequently see documentation and literature talk about "putting a process in a chroot jail" or "jailing a process with chroot." It's somewhat unfortunate, then, that the authors of the jail(2) system call in FreeBSD chose the term they did. In FreeBSD, jail and chroot are totally different implementations that achieve similar goals. Because this text deals significantly with both technologies, we are very careful with our usage. When we text say chroot, we refer to the chroot system call and when we say jail, we mean the jail system call.

2.4.1. New Limitations

Poul-Henning Kamp and Robert Watson created a whole new system of sandboxing processes. They actually modified the FreeBSD kernel in a variety of places to specifically close loopholes that had been historically used to escape chroot environments. The intent of a jail is to create a different environment, and one that cannot be so easily escaped. While the kernel is minimally aware of chrooted programs, it has dozens of checks in all sorts of subsystems that are related to keeping programs in a jail.

2.4.1.1 Limited process interaction

One of the most significant and interesting ways that jail is different is that it creates a whole new identifier associated with a process and its permissions: the jail ID. All processes have always had an effective user ID (UID) and group ID (GID). With the advent of jails, all processes now have a jail ID (JID) as well. Traditionally one process could affect another process if the UIDs of the two processes matched. This is true even in chroot environments. In a system using jails, however, the JID must also match. Even though a process with JID 2 has UID 0 (i.e., a root-equivalent process in jail 2), it cannot send signals to, read information from, or even detect the existence of a process that has a different JID. As you might expect, JID 0 corresponds to the main operating system. A process with UID 0 and JID 0 can kill any process whether the victim process is jailed or not.

2.4.1.2 Limited access to network resources

Jail addresses network issues that are completely unaddressed by chroot. Although a privileged chrooted process sees a virtual filesystem, its access to the network is unmediated. It can send and receive arbitrary data, listen(2) on a socket(2), create raw sockets, spoof packets, and perform similar nefarious activity. Every jail, on the other hand, has a single IP address and hostname associated with it. Processes in that jail will only be able to send packets with the IP address assigned to the jail. Ethernet MAC address spoofing is not possible within a jail, either. Jailed processes cannot access promiscuous mode on Ethernet drivers (or the bpf packet filtering device in general). The kernel explicitly checks the process's jail ID before allowing such things. Raw sockets, MAC spoofing, and such activities are only permitted to non-jailed processes. These restrictions on networking set jail apart from chroot already, but the improvements do not stop there.

2.4.1.3 Devices and mknod

Jail and chroot both create a virtual filesystem, but with jail, the process's ability to perform disk-related activities is severely restricted. A jailed process cannot call mknod(2) (meaning mknod(8) also fails) to make device nodes. Because of this restriction on making devices, all the devices must be made in the jail's /dev directory before launching the jail. Although processes cannot create new device nodes, they can read any devices that already exist in the jail. Thus, it's very important to create only the device nodes that the processes will need. Chances are good that no process that needs a jail will also need access to raw SCSI devices (such as pass(4)), for instance. You might be surprised to find that /dev/mem and /dev/kmem are needed, since they appear to create a quick path to information disclosure. As it turns out, the kernel device drivers related to those pseudodevices don't allow jailed processes to read or write the memory areas associated with processes having a different JID.

In FreeBSD 5.x, the old MAKEDEV way of handling devices is gone in favor of devfs. /etc/devfs.conf controls the permissions for each device in the /dev directory and can hide device nodes that are undesirable for one reason or another. When you make your jail, you can create an /etc/devfs.conf file in it that will create just the right devices that your processes need. You can also expect the normal devices to appear, and simply hide the ones that you don't want visible in your jail. If you're thinking about making this filesystem immutable, good for you.

2.4.2. Creating Jail Environments

There are two different styles of jails: "thin" and "fat." Thin jails are just like chroot, but better. You set them up the same way you set up a chroot environment, but you get all the additional protections from jail. Fat jails, on the other hand, are almost like virtual systems. You install a subset of the operating system in them and then you can use and manage them like a separate host. We talk about both, but it's safe to say that thin jails and chroot really differ mainly in the syntax of the command you use to launch them. Everything we described in "Creating a chroot Environment" applies to creating a thin jail. The rest of this section explores creating fat jails and the motivations for doing so.

The manpage for jail(8) covers creating jails very well. In particular they offer the following sage advice:

It is a lot easier to start with a "fat" jail and remove things until it stops working than it is to start with a "thin" jail and add things until it works.

Given that as a starting point, there are a couple of realistic ways to create reasonable fat jails: We can build the whole OS from scratch or we can use the binaries on a distribution CD.

2.4.2.1 Building jails from source

The manpage for jail(8) recommends the procedure shown in Example 2-9 for building a jail from source. We build it in /jail/master because, as we discuss in "Make a builder jail," it's often useful to have a fat jail where you build all the software that will run in your other jails.

Example 2-9. Building a jail from source

% D= /jail/master % cd /usr/src % sudo mkdir -p $D % sudo make world DESTDIR=$D % cd etc % sudo make distribution DESTDIR=$D % sudo mount_devfs devfs $D/dev % cd $D % sudo ln -sf dev/null kernel

If you take this approach, make sure you use your /etc/make.conf file (see Chapter 4 for more about /etc/make.conf) to exclude a lot of the optional modules (e.g., sendmail, BIND, ISDN4BSD, etc.) that you don't need in your jails. This may lead you to create a couple of make.conf files, one for building your system in general and one for building jails.

2.4.2.2 Installing from a distribution CD

If you have your distribution CD mounted at /cdrom, you'll find the base OS distribution at /cdrom/base. There is an install.sh shell script that obeys the DESTDIR environment variable. Example 2-10 shows how to install the base OS into a jail from the CD.

Example 2-10. Creating a jail from a distribution CD

% cd /cdrom/base % sudo sh install.sh DESTDIR=/jail/master

You can add other distributions located in /cdrom/distribution_name in the same way.

2.4.3. Launching Jails

Launching jail requires a few more command-line arguments than chroot because the jail system call requires more arguments than does chroot. In addition to the virtual root of the filesystem, you must also supply the one IP address and hostname that the jail will use. Ideally, this hostname appears in DNS matched to the IP address in use for the jail; but, if you use an /etc/hosts entry, it will work fine. Lastly, you must provide a command to run in the jail.

2.4.3.1 Fat jails as virtual machines

You may get to the point where you have an /etc directory, sshd, and a variety of other binaries installed in your jail. In this case, you can run /etc/rc to start up something that looks and feels like another instance of FreeBSD but isn't really. In this case, after the jail is up and running, you can actually use ssh to log into it and manage it. Since processes you launch will be children of sshd, which is a child of the jailed /bin/sh, they all have the same JID. This means they all see the same virtual filesystem, use the same IP address, and can interact with each other.

Of course, if there are processes in your jail that listen on the network, attackers can attack the jail in much the same ways as they can any other host. Still, even if they actually succeed in logging in, the amount of damage they can do is significantly limited. They probably can't see any part of the filesystem other than what is in the virtual filesystem, and they can't interact with any processes running outside the jail. Of course, they might be able to use the jail environment to participate in distributed denial-of-service (DDoS) attacks, but the complex network address spoofing that some DDoS attacks use won't work.

2.4.3.2 Jail security options

There are special sysctl variables that control certain behaviors of jails. They control whether jailed process can affect the jail's hostname, use certain network protocols, and whether or not System V interprocess communication (IPC) is allowed. These options have security ramifications that might need to be relaxed in order for certain software to work as expected.

Normally, jails only have permissions to work with local sockets, routing sockets, and IPv4 sockets. Only these sections of the networking part of the kernel are fully jail aware. Thus, it's possible that a jailed program with access to a different protocol family (e.g., IPX or ATM) could do malicious things with those protocols. The security.jail.socket_unixiproute_only variable controls whether jails can access all protocols, or just the jail-safe protocol families. Change the default value to 0 to enable the non-jail-safe protocols.

System V IPC allows processes to communicate through several mechanisms and to use a variety of synchronization primitives like semaphores. Unfortunately, this is another section of the kernel that is not jail aware in a sophisticated way. System V IPC is not permitted from within a jail by default, but you can enable this functionality by setting security.jail.sysvipc_allowed to 1. Unfortunately, it's an all or nothing proposition. If enabled, a jailed process can see all semaphores, shared memory segments, queues, and other System V IPC data structures. This means that root-equivalent processes will have access to all System V IPC data structures, jailed or not. Such broad access allows a jailed process to have effects outside its own jail, which obviously has security implications and weakens one of the goals of the jail in the first place. In general this option should retain its default disabled status unless you have a specific reason to enable it.

2.4.3.3 Managing jails

A variety of commands make working with jails easier. For instance, the output of ps(1) does not indicate which jail a particular process is in, but the jls(1) command does. After you know a JID, you can use jexec(1) to add a new process to an already running jail. If you need to kill a process, you can do so from outside the jail (if your shell has JID 0, of course). If you want to kill an entire jail, you can combine the kill(1) command (assuming /bin/kill or /bin/sh exists in the jail) with jexec. For example, you can kill every process in jail 3 by running sudo jexec 3 kill -1.

2.4.4. Installing Software in Jail

This section of the text discusses the FreeBSD packages and ports system extensively. If you are unfamiliar with them, see Chapter 4 of the FreeBSD Handbook. We also talk a lot about them in Chapter 4 of this book.

When you try to put a program in a jail, especially if you are trying to install only the minimal subset of files it needs, you may have difficulties getting the right files in the right places, as we described previously. Instead of hunting down dependencies, consider using the ports system to create packages that can be installed in the jail. For custom software, you have a couple of options: you can just wing it, or you can leverage the ports system to make a new port of your custom software, and then all the packaging and tracking comes along for free.

2.4.4.1 Make a builder jail

If you make your software from source, using the ports tree, you can still install the software into the jails using packages. The best option is to create a fat jail that looks just like the jail where you will be installing the new software, except that it has a compiler and the ports system installed in it. Build software inside that jail and then run make package. The FreeBSD ports system cannot make a package from your newly built binaries unless they are first installed in the main operating system, so you definitely want to do this in your builder jail to avoid interfering with your base OS.

For this example, we assume that the target jail is rooted at /jail/web and there is a special building jail named "master" already configured at /jail/master. The whole session would look something like Example 2-11.

Example 2-11. Using a jail to build software for other jails

main-system% sudo ifconfig lo0 alias 127.0.0.2 main-system% sudo jail 127.0.0.2 master /bin/sh builder% cd /usr/ports/www/apache2 builder% make [... make messages omitted ...] builder% sudo make install builder% sudo make package builder% sudo kill -1 main-system% sudo pkg_add -R /jail/web \         /jail/master/usr/ports/www/apache2/apache-2.55.tgz

Note that we have to create an IP address and associate it with some network interface (the loopback, in this example) even though we won't be doing any networking.

2.4.4.2 Install from binary package

Copy the package file you've just made (and the pkg_add executable and associated dependencies, if they're not already in the jail) into the jail's filesystem. If the jail is not currently running, you can install the software by using the jail command to run pkg_add, as shown in Example 2-12. Notice that the first thing to do is to copy the package file into some location that will be visible inside the jail.

Example 2-12. Installing into a jail from a package file

% sudo cp ports/packages/All/apache-2.55.tgz /jail/web/tmp % sudo jail 10.1.0.2 www.example.com /jail/web pkg_add /tmp/apache-2.55.tgz

2.4.4.3 Getting custom software installed in a jail

If you download some source code that is not supported by the ports and packages infrastructure, you have a few options for getting it in jail. One option is to make a port out of it. It takes a little effort to start with, but it's effort well spent. Once your software is correctly installed through a port, it is very little work to keep it up to date. You can use the portupgrade program to maintain it, and you can run make package to make a binary installable package out if it. All this functionality is free after you create the port infrastructure.

The other option is to figure out its dependencies through trial and error. The section "Finding Other Dependencies" discusses a lot of good techniques you can use to find and satisfy dependencies. Remember that you can also RTFM and look at the source code. If you're trying to jail some interesting software, you're probably not the only person to try it. Check the mailing lists both for your operating system and for the application itself and see if anyone has had luck or can at least warn you about the problems you'll face. You'll probably find it most helpful to do this trial-and-error work inside your builder jail, because it presumably looks the most like the jail in which your software will ultimately run.

2.4.5. NFS-Based Jails

Jails offer the possibility of creating lots of virtual environments for specialized jobs and segregating them into separate independent sandboxes. However, in order to run correctly, a lot of the operating system might need to be installed in the jail. This creates a scalability problem for large numbers of jails. Not only do they require copies of the operating system files, but those copies must all be kept synchronized and updated. Updating a large number of jails is difficult, as is detecting changes in them. You can use read-only NFS to solve several of these issues, making dozens or hundreds of jails scalable.

2.4.5.1 Creating a single NFS master jail

Create a master jail filesystem that contains the right set of files for your applications. If you plan to use just one master for dozens or hundreds of jails, then this may end up including a superset of files that are not needed in all jails. Share this filesystem as read-only to localhost (127.0.0.1) only. Likewise, make a temporary area that every jail can write to for its /tmp and /var partitions. Share these as read/write to localhost also. Have each jail mount the single read-only master jail root filesystem, and writable /var and /tmp filesystems from localhost.

If you use this approach, you probably want to create a "package master" jail of some kind into which you can freely install software and create packages. The master jail in this example is too important to screw up.

There are many advantages to the read-only NFS master jail. All the upgrades can be done in just one place and take effect immediately. The filesystem is read-only from the point of view of the jail. Even if the jail is compromised, a rogue process cannot overwrite the NFS-mounted files. Lastly, disk caching in the operating system can still help with performance. If you make 10 copies of all your binaries and data, then you will get lower disk cache performance than if you have 10 jails all requesting the same binary off the disk.