Section 2.4. Putting Apache in Jail

2.4. Putting Apache in Jail

Even the most secure software installations get broken into. Sometimes, this is because you get the attention of a skilled and persistent attacker. Sometimes, a new vulnerability is discovered, and an attacker uses it before the server is patched. Once an intruder gets in, his next step is to look for local vulnerability and become superuser. When this happens, the whole system becomes contaminated, and the only solution is to reinstall everything.

Our aim is to contain the intrusion to just a part of the system, and we do this with the help of the chroot(2) system call. This system call allows restrictions to be put on a process, limiting its access to the filesystem. It works by choosing a folder to become the new filesystem root. Once the system call is executed, a process cannot go back (in most cases, and provided the jail was properly constructed).

The root user can almost always break out of jail. The key to building an escape-proof jail environment is not to allow any root processes to exist inside the jail. You must also not have a process outside jail running as the same user as a process inside jail. Under some circumstances, an attacker may jump from one process to another and break out of jail. That's one of the reasons why I have insisted on having a separate account for Apache.

The term chroot is often interchangeably used with the term jail. The term can be used as a verb and noun. If you say Apache is chrooted, for example, you are saying that Apache was put in jail, typically via use of the chroot binary or the chroot(2) system call. On Linux systems, the meanings of chroot and jail are close enough. BSD systems have a separate jail( ) call, which implements additional security mechanisms. For more details about the jail( ) call, see the following: http://docs.freebsd.org/44doc/papers/jail/jail.html.

Incorporating the jail mechanism (using either chroot(2) or jail( )) into your web server defense gives the following advantages:

Containment: If the intruder breaks in through the server, he will only be able to access files in the restricted file system. Unable to touch other files, he will be unable to alter them or harm the data in any way.
No shell: Most exploits need shells (mostly /bin/sh) to be fully operative. While you cannot remove a shell from the operating system, you can remove it from a jail environment.
Limited tool availability: Once inside, the intruder will need tools to progress further. To begin with, he will need a shell. If a shell isn't available he will need to find ways to bring one in from the inside. The intruder will also need a compiler. Many black hat tools are not used as binaries. Instead, these tools are uploaded to the server in source and compiled on the spot. Even many automated attack tools compile programs. The best example is the Apache Slapper Worm (see the sidebar Apache Slapper Worm).
Absence of suid root binaries: Getting out of a jail is possible if you have the privileges of the root user. Since all the effort we put into the construction of a jail would be meaningless if we allowed suid root binaries, make sure you do not put such files into the jail.

The chroot(2) call was not originally designed as a security measure. Its use for security is essentially a hack, and will be replaced as the server virtualization technologies advance. For Linux, that will happen once these efforts become part of a mainstream kernel. Though server virtualization falls out of the scope of this book, some information on this subject is provided in Chapter 9.

The following sections describe various approaches to putting Apache in jail. First, an example demonstrating use of the original chroot binary to put a process in jail is shown. That example demonstrates the issues that typically come up when attempting to put a process in jail and briefly documents tools that are useful for solving these issues. Next, the steps required for creating a jail and putting Apache in it using chroot are shown. This is followed by the simpler chroot(2) approach, which can be used in some limited situations. Finally, the use of mod_security or mod_chroot to chroot Apache is presented.

Apache Slapper Worm

The Apache Slapper Worm (http://www.cert.org/advisories/CA-2002-27.html) is arguably the worst thing to happen to the Apache web server as far as security goes. It uses vulnerabilities in the OpenSSL subsystem (http://www.cert.org/advisories/CA-2002-23.html) to break into a system running Apache. It proceeds to infect other systems and calls back home to become a part of a distributed denial of service (DDoS) network. Some variants install a backdoor, listening on a TCP/IP port. The worm only works on Linux systems running on the Intel architecture.

The behavior of this worm serves as an excellent case study and a good example of how some of the techniques we used to secure Apache help in real life.

The worm uses a probing request to determine the web server make and version from the Server response header and attacks the servers it knows are vulnerable. A fake server signature would, therefore, protect from this worm. Subsequent worm mutations stopped using the probing request, but the initial version did and this still serves as an important point.
If a vulnerable system is found, the worm source code is uploaded (to /tmp) and compiled. The worm would not spread to a system without a compiler, to a system where the server is running from a jail, or to a system where code execution in the /tmp directory is disabled (for example, by mounting the partition with a noexec flag).

Proper firewall configuration, as discussed in Chapter 9, would stop the worm from spreading and would prevent the attacker from going into the server through the backdoor.

2.4.1. Tools of the chroot Trade

Before you venture into chroot land you must become aware of several tools and techniques you will need to make things work and to troubleshoot problems when they appear. The general problem you will encounter is that programs do not expect to be run without full access to the filesystem. They assume certain files are present and they do not check error codes of system calls they assume always succeed. As a result, these programs fail without an error message. You must use diagnostic tools such as those described below to find out what has gone wrong.

2.4.1.1 Sample use of the chroot binary

The chroot binary takes a path to the new filesystem root as its first parameter and takes the name of another binary to run in that jail as its second parameter. First, we need to create the folder that will become the jail:

# mkdir /chroot

Then, we specify the jail (as the chroot first parameter) and try (and fail) to run a shell in the jail:

# chroot /chroot /bin/bash chroot: /bin/bash: No such file or directory

The above command fails because chroot corners itself into the jail as its first action and attempts to run /bin/bash second. Since the jail contains nothing, chroot complains about being unable to find the binary to execute. Copy the shell into the jail and try (and fail) again:

# mkdir /chroot/bin # cp /bin/bash /chroot/bin/bash # chroot /chroot /bin/bash chroot: /bin/bash: No such file or directory

How can that be when you just copied the shell into jail?

# ls -al /chroot/bin/bash -rwxr-xr-x   1 root    root      605504 Mar 28 14:23 /chroot/bin/bash

The bash shell is compiled to depend on several shared libraries, and the Linux kernel prints out the same error message whether the problem is that the target file does not exist or that any of the shared libraries it depends on do not exist. To move beyond this problem, we need the tool from the next section.

2.4.1.2 Using ldd to discover dependencies

The ldd toolavailable by default on all Unix systemsprints shared library dependencies for a given binary. Most binaries are compiled to depend on shared libraries and will not work without them. Using ldd with the name of a binary (or another shared library) as the first parameter gives a list of files that must accompany the binary to work. Trying ldd on /bin/bash gives the following output:

# ldd /bin/bash         libtermcap.so.2 => /lib/libtermcap.so.2 (0x0088a000)         libdl.so.2 => /lib/libdl.so.2 (0x0060b000)         libc.so.6 => /lib/tls/libc.so.6 (0x004ac000)         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00494000)

Therefore, bash depends on four shared libraries. Create copies of these files in jail:

# mkdir /chroot/lib # cp /lib/libtermcap.so.2 /chroot/lib # cp /lib/libdl.so.2 /chroot/lib # cp /lib/tls/libc.so.6 /chroot/lib # cp /lib/ld-linux.so.2 /chroot/lib

The jailed execution of a bash shell will finally succeed:

# chroot /chroot /bin/bash bash-2.05b#

You are rewarded with a working shell prompt. You will not be able to do much from it though. Though the shell works, none of the binaries you would normally use are available inside (ls, for example). You can only use the built-in shell commands, as can be seen in this example:

bash-2.05b# pwd / bash-2.05b# echo /* /bin /lib bash-2.05b# echo /bin/* /bin/bash bash-2.05b# echo /lib/* /lib/ld-linux.so.2 /lib/libc.so.6 /lib/libdl.so.2 /lib/libtermcap.so.2

As the previous example demonstrates, from a jailed shell you can access a few files you explicitly copied into the jail and nothing else.

2.4.1.3 Using strace to see inside processes

The strace tool (truss on systems other than Linux) intercepts and records system calls that are made by a process. It gives much insight into how programs work, without access to the source code. Using chroot and ldd, you will be able to get programs to run inside jail, but you will need strace to figure out why they fail when they fail without an error message, or if the error message does not indicate the real cause of the problem. For that reason, you will often need strace inside the jail itself. (Remember to remove it afterwards.)

Using strace you will find that many innocent looking binaries do a lot of work before they start. If you want to experiment, I suggest you write a simple program such as this one:

#include <stdio.h> #include <stdarg.h>     int main(void) {     puts("Hello world!"); }

Compile it once with a shared system support and once without it:

# gcc helloworld.c -o helloworld.shared # gcc helloworld.c -o helloworld.static -static

Using strace on the static version gives the following output:

# strace ./helloworld.static execve("./helloworld.static", ["./helloworld.static"], [/* 22 vars */]) = 0 uname({sys="Linux", node="ben", ...})   = 0 brk(0)                                  = 0x958b000 brk(0x95ac000)                          = 0x95ac000 fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0 old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xbf51a000 write(1, "Hello world!\n", 13Hello world! )          = 13 munmap(0xbf51a000, 4096)                = 0 exit_group(13)

The strace output is ugly. Each line in the output represents a system call made from the process. It is not important at the moment what each line contains. Jailed binaries most often fail because they cannot open a file. If that happens, one of the lines near the end of the output will show the name of the file the binary attempted to access:

open("/usr/share/locale/locale.alias", O_RDONLY) = -1 ENOENT (No such file or directory)

As an exercise, use strace on the dynamically compiled version of the program and compare the two outputs. You will see how many shared libraries are accessed even from a small program such as this one.

2.4.2. Using chroot to Put Apache in Jail

Now that you know the basics of using chroot to put a process in jail and you are familiar with tools required to facilitate the process, we can take the steps required to put Apache in jail. Start by creating a new home for Apache and move the version installed (shown in Section 2.1.4) to the new location:

# mkdir -p /chroot/apache/usr/local # mv /usr/local/apache /chroot/apache/usr/local # ln -s /chroot/apache/usr/local/apache /usr/local/apache # mkdir -p /chroot/apache/var # mv /var/www /chroot/apache/var/ # ln -s /chroot/apache/var/www /var/www

The symbolic link from the old location to the new one allows the web server to be used with or without being jailed as needed and allows for easy web server upgrades.

Like other programs, Apache depends on many shared libraries. The ldd tool gives their names (this ldd output comes from an Apache that has all default modules built-in statically):

# ldd /chroot/apache/usr/local/apache/bin/httpd         libm.so.6 => /lib/tls/libm.so.6 (0x005e7000)         libcrypt.so.1 => /lib/libcrypt.so.1 (0x00623000)         libgdbm.so.2 => /usr/lib/libgdbm.so.2 (0x00902000)         libexpat.so.0 => /usr/lib/libexpat.so.0 (0x00930000)         libdl.so.2 => /lib/libdl.so.2 (0x0060b000)         libc.so.6 => /lib/tls/libc.so.6 (0x004ac000)         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00494000)

This is a long list; we make copies of these libraries in the jail:

# mkdir /chroot/apache/lib # cp /lib/tls/libm.so.6 /chroot/apache/lib # cp /lib/libcrypt.so.1 /chroot/apache/lib # cp /usr/lib/libgdbm.so.2 /chroot/apache/lib # cp /usr/lib/libexpat.so.0 /chroot/apache/lib # cp /lib/libdl.so.2 /chroot/apache/lib # cp /lib/tls/libc.so.6 /chroot/apache/lib # cp /lib/ld-linux.so.2 /chroot/apache/lib

2.4.2.1 Putting user, group, and name resolution files in jail

Though the httpd user exists on the system (you created it as part of the installation earlier); there is nothing about this user in the jail. The jail must contain the basic user authentication facilities:

# mkdir /chroot/apache/etc # cp /etc/nsswitch.conf /chroot/apache/etc/ # cp /lib/libnss_files.so.2 /chroot/apache/lib

The jail user database needs to contain at least one user and one group. Use the same name as before and use the identical user and group numbers inside and outside the jail. The filesystem stores user and group numbers to keep track of ownership. It is a job of the ls binary to get the usernames from the user list and show them on the screen. If there is one user list on the system and another in the jail with different user numbers, directory listings will not make much sense.

# echo "httpd:x:500:500:Apache:/:/sbin/nologin" > /chroot/apache/etc/passwd # echo "httpd:x:500:" > /chroot/apache/etc/group

At this point, Apache is almost ready to run and would run and serve pages happily. A few more files are needed to enable domain name resolution:

# cp /lib/libnss_dns.so.2 /chroot/apache/lib # cp /etc/hosts /chroot/apache/etc # cp /etc/resolv.conf /chroot/apache/etc

2.4.2.2 Finishing touches for Apache jail preparation

The walls of the jail are now up. Though the following files are not necessary, experience shows that many scripts require them. Add them now to avoid having to debug mysterious problems later.

Construct special devices after using ls to examine the existing /dev folder to learn what numbers should be used:

# mkdir /chroot/apache/dev # mknod -m 666 /chroot/apache/dev/null c 1 3 # mknod -m 666 /chroot/apache/dev/zero c 1 5 # mknod -m 644 /chroot/apache/dev/random c 1 8

Then, add a temporary folder:

# mkdir /chroot/apache/tmp # chmod +t /chroot/apache/tmp # chmod 777 /chroot/apache/tmp

Finally, configure the time zone and the locale (we could have copied the whole /usr/share/locale folder but we will not because of its size):

# cp /usr/share/zoneinfo/MET /chroot/apache/etc/localtime # mkdir -p /chroot/apache/usr/lib/locale # set | grep LANG LANG=en_US.UTF-8 LANGVAR=en_US.UTF-8 # cp -dpR /usr/lib/locale/en_US.utf8 /chroot/apache/usr/lib/locale

2.4.2.3 Preparing PHP to work in jail

To make PHP work in jail, you should install it as normal. Establish a list of shared libraries required and copy them into the jail:

# ldd /chroot/apache/usr/local/apache/libexec/libphp4.so         libcrypt.so.1 => /lib/libcrypt.so.1 (0x006ef000)         libresolv.so.2 => /lib/libresolv.so.2 (0x00b28000)         libm.so.6 => /lib/tls/libm.so.6 (0x00111000)         libdl.so.2 => /lib/libdl.so.2 (0x00472000)         libnsl.so.1 => /lib/libnsl.so.1 (0x00f67000)         libc.so.6 => /lib/tls/libc.so.6 (0x001df000)         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00494000)

Some of the libraries are already in the jail, so skip them and copy the remaining libraries (shown in bold in the previous output):

# cp /lib/libresolv.so.2 /chroot/apache/lib # cp /lib/libnsl.so.1 /chroot/apache/lib

One problem you may encounter with a jailed PHP is that scripts will not be able to send email because the sendmail binary is missing. To solve this, change the PHP configuration to make it send email using the SMTP protocol (to localhost or some other SMTP server). Place the following in the php.ini configuration file:

SMTP = localhost

2.4.2.4 Preparing Perl to work in jail

To make Perl work, copy the files into the jail:

# cp -dpR /usr/lib/perl5 /chroot/apache/usr/lib # mkdir /chroot/apache/bin # cp /usr/bin/perl /chroot/apache/bin

Determine the missing libraries:

# ldd /chroot/apache/bin/perl         libperl.so => /usr/lib/perl5/5.8.1/i386-linux-thread-multi /CORE/libperl.so (0x0067b000)         libnsl.so.1 => /lib/libnsl.so.1 (0x00664000)         libdl.so.2 => /lib/libdl.so.2 (0x0060b000)         libm.so.6 => /lib/tls/libm.so.6 (0x005e7000)         libcrypt.so.1 => /lib/libcrypt.so.1 (0x00623000)         libutil.so.1 => /lib/libutil.so.1 (0x00868000)         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00652000)         libc.so.6 => /lib/tls/libc.so.6 (0x004ac000)         /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00494000)

Then add them to the libraries that are inside:

# cp /lib/libutil.so.1 /chroot/apache/lib # cp /lib/tls/libpthread.so.0 /chroot/apache/lib

2.4.2.5 Taking care of small jail problems

Most CGI scripts send email using the sendmail binary. That will not work in our jail since the sendmail binary isn't there. Adding the complete sendmail installation to the jail would defy the very purpose of having a jail in the first place. If you encounter this problem, consider installing mini_sendmail (http://www.acme.com/software/mini_sendmail/), a sendmail replacement specifically designed for jails. Most programming languages come with libraries that allow email to be sent directly to an SMTP server. PHP can send email directly, and from Perl you can use the Mail::Sendmail library. Using these libraries reduces the number of packages that are installed in a jail.

You will probably encounter database connectivity problems when scripts in jail try to connect to a database engine running outside the jail. This happens if the program is using localhost as the host name of the database server. When a database client library sees localhost, it tries to connect to the database using a Unix domain socket. This socket is a special file usually located in /tmp, /var/run, or /var/lib, all outside the jail. One way to get around this is to use 127.0.0.1 as the host name and force the database client library to use TCP/IP. However, since a performance penalty is involved with that solution (Unix domain socket communication is much faster than communication over TCP/IP), a better way would be to have the socket file in the jail.

For PostgreSQL, find the file postgresql.conf (usually in /var/lib/pgsql/data) and change the line containing the unix_socket_directory directive to read:

unix_socket_directory = '/chroot/apache/tmp'

Create a symbolic link from the previous location to the new one:

# ln -s /chroot/apache/tmp/.s.PGSQL.5432 /tmp

MySQL keeps its configuration options in a file called my.cnf, usually located in /etc. In the same file, you can add a client section (if one is not there already) and tell clients where to look for a socket:

[mysqld] datadir=/var/lib/mysql socket=/chroot/apache/var/lib/mysql/mysql.sock     [client] socket=/chroot/apache/var/lib/mysql/mysql.sock

Or, just as you did with PostgreSQL, create a symbolic link:

# mkdir -p /chroot/apache/var/lib/mysql # chown mysql /chroot/apache/var/lib/mysql/ # ln -s /chroot/apache/var/lib/mysql/mysql.sock /var/lib/mysql

2.4.3. Using the chroot(2) Patch

Now that I have explained the manual chroot process, you are wondering if an easier way exists. The answer is, conditionally, yes.

The approach so far was to create the jail before the main process was started. For this approach to work, the jail must contain all shared libraries and files the process requires. This approach is also known as an external chroot.

With an internal chroot, the jail is established from within the process after the process initialization is completed. In the case of Apache, the jail must be created before request processing begins, at the latest. The process is born free and then jailed. Since the process has full access to the filesystem during the initialization phase, it is free to access any files it needs. Because of the way chrooting works, descriptors to the files opened before the call remain valid after. Therefore, we do not have to create a copy of the filesystem and we can have a "perfect" jail, the one that contains only files needed for web serving, the files in the web server tree.

Internal chroot can be dangerous. In external chroot approaches, the process is born in jail, so it has no opportunity to interact with the outside filesystem. With the internal chroot, however, the process has full access to the filesystem in the beginning and this allows it to open files outside the jail and continue to use them even after the jail is created. This opens up interesting opportunities, such as being able to keep the logs and the binaries outside jail, but is a potential problem. Some people are not comfortable with leaving open file descriptors outside jail. You can use the lsof utility to see which file descriptors Apache has open and determine whether any of them point outside jail. My recommendation is the following: If you can justify a high level of security for your installation, go for a proper external chroot approach. For installations of less importance, spending all that time is not feasible. In such cases, use the internal chroot approach.

It is obvious that internal chrooting is not a universal solution. It works only if the following is true:

The only functionality needed is that of Apache and its modules.
There will be no processes (such as CGI scripts) started at runtime. Alternatively, if CGI scripts are used, they will be statically compiled.
Access to files outside the web server root will be not be required at runtime. (For example, if you intend to use the piped logging mechanism, Apache must be able to access the logging binary at runtime to restart logging in case the original logging process dies for some reason. Piped logging is discussed in Chapter 8.)

Now that I have lured you into thinking you can get away from the hard labor of chrooting, I will have to disappoint you: Apache does not support internal chrooting natively. But the help comes from Arjan de Vet in the form of a chroot(2) patch. It is available for download from http://www.devet.org/apache/chroot/. After the patch is applied to the source code, Apache will support a new directive, ChrootDir. Chrooting Apache can be as easy as supplying the new root of the filesystem as the ChrootDir first parameter. The record of a successful chroot(2) call will be in the error log.

As a downside, you will have to apply the patch every time you install Apache. And there is the problem of finding the patch for the version of Apache you want to install. At the time of this writing only the patch for Apache 1.3.31 is available. But not everything is lost.

2.4.4. Using mod_security or mod_chroot

In a saga with more twists than a soap opera, I will describe a third way to jail Apache. Provided the limitations described in the previous section are acceptable to you, this method is the simplest: chrooting using mod_security (http://www.modsecurity.org) or mod_chroot (http://core.segfault.pl/~hobbit/mod_chroot/). Both modules use the same method to do their work (at the time of this writing) so I will cover them in this section together. Which module you will use depends on your circumstances. Use mod_security if you have a need for its other features. Otherwise, mod_chroot is likely to be a better choice because it only contains code to deal with this one feature and is, therefore, less likely to have a fault.

The method these two modules use to perform chrooting is a variation of the chroot(2) patch. Thus, the discussion about the usefulness of the chroot(2) patch applies to this case. The difference is that here the chroot(2) call is made from within the Apache module (mod_security or mod_chroot), avoiding a need to patch the Apache source code. And it works for 1.x and 2.x branches of the server. As in the previous case, there is only one new directive to learn: SecChrootDir for mod_security or ChrootDir for mod_chroot. Their syntaxes are the same, and they accept the name of the root directory as the only parameter:

SecChrootDir /chroot/apache

The drawback of working from inside the module is that it is not possible to control exactly when the chroot call will be executed. But, as it turns out, it is possible to successfully perform a chroot(2) call if the module is configured to initialize last.

2.4.4.1 Apache 1

For Apache 1, this means manually configuring the module loading order to make sure the chroot module initializes last. To find a list of compiled-in modules, execute the httpd binary with the -l switch:

# ./httpd -l Compiled-in modules:   http_core.c   mod_env.c   mod_log_config.c   mod_mime.c   mod_negotiation.c   mod_status.c   mod_include.c   mod_autoindex.c   mod_dir.c   mod_cgi.c   mod_asis.c   mod_imap.c   mod_actions.c   mod_userdir.c   mod_alias.c   mod_rewrite.c   mod_access.c   mod_auth.c   mod_so.c   mod_setenvif.c

To this list, add modules you want to load dynamically. The core module, http_core, should not appear on your list. Modules will be loaded in the reverse order from the one in which they are listed in the configuration file, so mod_security (or mod_chroot) should be the first on the list:

ClearModuleList AddModule mod_security.c AddModule ... AddModule ...

2.4.4.2 Apache 2

With Apache 2, there is no need to fiddle with the order of modules since the new API allows module programmers to choose module position in advance. However, the changes in the architecture are causing other potential problems to appear:

Unlike in Apache 1, in Apache 2 some of the initialization happens after the last module initializes. This causes problems if you attempt to create a jail in which the logs directory stays outside jail. The solution is to create another logs directory inside jail, which will be used to store the files Apache 2 needs (e.g., the pid file). Many of the modules that create temporary files have configuration directives that change the paths to those files, so you can use those directives to have temporary files created somewhere else (but still within the jail).
On some platforms, internal Apache 2 chroot does not work if the AcceptMutex directive is set to pthread. If you encounter a problem related to mutexes change the setting to something else (e.g., posixsem, fcntl, or flock).