Section 6.6 Hardening for Very High Security

---

6.6 Hardening for Very High Security

Here I cover some ideas for hardening a Linux system to the highest level of practical security. At present, Linux does not offer Orange Book C2 or B1 security, though a number of firms are working on this for the U.S. government; at least one plans to release its work under the standard GPL.

The first step in hardening a system is to turn off and remove all unnecessary services. Study your startup scripts under /etc/rc.d and disable and remove anything that is not needed. Especially disable sendmail, lpd, DNS (named), nfsd and mountd, popd, imapd, rwhod, Samba, AppleTalk, X, NIS (YP), linuxconf, swat, Netscape, and the Web server (such as Apache). Under Red Hat and its derivatives, checkconfig --del service will turn off a daemon and prevent its being restarted on reboot. rpm --uninstall package will remove the package from disk.

Some of these such as sendmail, lpd, DNS, nfsd and mountd, popd, imapd, and rwhod each have been broken into in a number of ways and so, statistically speaking, there might be more security holes lurking in them. Others, such as nfsd and mountd, are insecure due to their design. These latter two use UDP packets which can be spoofed easily. A few of these wrap themselves around so much of your system and are so complex that it is hard to configure them to be secure without making a mistake. These include Samba, Apache (the Web server), DOSemu, and X.

Because many people find it hard to live without X, some might consider leaving it on a high-security system if there is firewall software in place to block access to it (TCP ports 6000 6100 but usually not above 6009) from other systems. See also "X Marks the Hole" on page 117. Secure encrypted X over SSH is discussed in "Wrapping SSH Around X" on page 417. More than a few security experts would insist that X be removed from any high-security system. Some of the reasons are because of the X server running as root and controlling much of a system's hardware and having many paths to it, including TCP. The latter, at least, can be stopped by invoking it as follows:

 X -nolisten tcp :0 

Issue a ps command to list processes so that you know what daemons might be providing services that you do not want.

 ps axlww | fold | lpr 

Additionally, issue a command that shows which TCP and UDP ports are in use and what state each is in. Consider using the ports program discussed in "Turn Off Unneeded Services" on page 86, which provides the functionality of netstat but also flags suspicious ports. The ports program also knows the names of some common services that are not listed in the distributed /etc/services (though you certainly can add them to your /etc/services file). Either of the following will work. By printing the results you can make notes on the printout about which services are needed and why. Note that netstat's -p flag requires root access.

 ports | fold | lpr 

or, as root,

 netstat -atuvp | fold |lpr 

Disable each daemon not required. If you are not sure if a particular daemon is required, take the time to research it or experiment with disabling it.

You will want to disable any programs (which you do not need) that are set-UID to root or set-GID to any privileged group. This way if a cracker gets into your system as an ordinary user, she is unable to break these programs and elevate herself to root. You will want to do this "transitioning to a very secure system" in many small steps. This is because if you were to do it in a single step, if it did not work, it would be very hard to determine what went wrong.

Obviously, a chmod ugo-x will disable them while preserving the indication that they once were set-UID or set-GID in case you ever need to undo this for one of them. This invocation will list these programs for your consideration.

 find / -perm +6000 -perm +111 -ls 

Recently, a friend was trying to shrink a Linux system down to the bare essentials for both space and security for the appliance we built. He was not sure whether the sendmail daemon was necessary for sending mail out. I assured him that it was not necessary, but I encouraged him to try it. He tried it and then disabled it in the startup scripts (but left the executable so it could be invoked by Mail for outgoing mail).^[a]

^[a] If the destination system is temporarily unreachable, the e-mail will be enqueued. To cause sendmail to attempt retransmission of the queued mail you will need to invoke the following command periodically, possibly from cron:

 /usr/sbin/sendmail -q 

Alternatively, sendmail may be started as a daemon that checks the queue every 15 minutes, but does not listen on SMTP port 25:

 /usr/sbin/sendmail -q15 

In Red Hat and Mandrake, you can get this effect by editing the file /etc/sysconfig/sendmail to include

 DAEMON=yes QUEUE=15m 

This certainly should be done when the system is first configured and the output stored as should a full backup. An even better and automatic solution would be to have a look in "Tripwire" on page 649 and make use of this excellent utility that automatically will alert you if system files change in any way.

Open network ports should be checked regularly, typically hourly or daily, to make sure that there are no unusual ones open that could indicate a cracker's Trojan horse. The ports program is tuned for this purpose and will flag suspicious ports. The netstat program is the standard program for this. Your entire network may be checked by nmap or several other popular port scanners. In addition to looking for open ports and some analysis of versions of software you are running, some network analyzers such as Nessus and SARA will probe your systems for security vulnerabilities. The use of these is considered mandatory for very high security systems. URLs are listed in Appendix A.

Unless cron and at are needed, they should be disabled. This is because they allow a cracker to have his Trojan horse not be running most of the time so that your periodic invocations of ps to check for Trojans will not see it. The cracker simply has cron start it at known times. Unless you specifically check the contents of the root crontab (and that of other users) you would never detect this intrusion.

Many systems have been compromised when the SysAdmin had the root crontab invoke a script that lived in a directory which had world write access. For example, it might invoke /root/bin/nightly when /root was mode 755 but /root/bin was mode 777. No doubt, this root account defaulted to a umask of 0.

A popular solution to harden systems is to use some of mount's options, some having been created specifically to thwart crackers. These flags cause the kernel to not honor certain capabilities on mounted file systems. Certainly, one of the most popular is ro, used to mount a file system Read/Only. There are many possibilities here.

Some sites mount /httpd Read/Only and configure Apache to log under the /var/log/httpd directory. This makes Web page defacements, which are a rapidly growing problem, much harder. (Additionally, see "Scouting Out Apache (httpd) Problems" on page 275 and "Detecting Defaced Web Pages Automatically" on page 661.) Some sites offering Read/Only anonymous FTP access will have /home/ftp be a Read/Only mounted file system. Some sites will have /etc and /usr Read/Only.

The good news is Linux has been set up so that almost all programs generate logs, locks, and other file write operations under /var. This means that one can have / mounted Read/Only and just mount /var, /home, and /tmp as separate Read/Write file systems. The trick is that on boot-up, have / temporarily mounted Read/Write in order to mount the other file systems and then remount it Read/Only. This is done by editing /etc/rc.d/rc.sysinit in Red Hat derived distributions or /etc/rc.d/rc.S for Slackware. Find the line reading

 mount -n -o remount,rw / 

Add mount commands after it for mounting other file systems, such as /tmp, /var, and /home. On some systems, /tmp would be a good candidate for a RAMDISK to boost performance. Then add the following line to remount Read/Only:

 mount -n -o remount,ro / 

Sun pioneered diskless UNIX workstations back when disks were expensive. It was Sun's innovation to put all the changing files (other than the users' file system and /tmp) under /var and mount /usr or even / Read/Only and use a single copy on a NFS server for a whole group of workstations.

Some experimentation might be required to deal with some programs that try to write where they should not. The root home directory, /root, will be Read/Only, but this is not a bad thing. It prevents an interloper from altering root's shell startup scripts. If shell history is desired, a symbolic link to somewhere else such as /home/root (to be created) would be needed. Note that after mounting Read/Write, if any process holds a file open for a mode that allows writing, the kernel will not allow the file system to be remounted Read/Only.

Another popular option is nosuid. It causes the kernel to ignore the set-UID and the set-GID bits on any file on that file system. This protects against this vulnerability. Although applying this option to the root file system is pointless, it is an addition to the "Rings of Security" when applied to /httpd, /home/ftp, /home, /tmp, and /var/tmp. Similarly, the nodev option causes the kernel to ignore block and character device files on the specified file system. Thus, if a cracker manages an exploit that can create such a device, he cannot make use of it. This stops the use of mknod to break out of a chroot prison, but there are other ways out. (See "Defeating the chroot() Vulnerability" on page 319 for some ways and defenses against them.) Applying noexec and nodev to some of these is an excellent idea, too.

Yet another possibility is to put the system on some Read/Only medium, such as CD-ROM or DVD. Any Read/Write partitions can be mounted over directory stubs on the Read/Only medium. Such directories might be /var, /tmp, and /home (for user data). This technique is used by distributions like DemoLinux (www.demolinux.org) for showing how Linux can run on your hardware. If your distribution can be compressed to less than 100MB, it can run successfully off of a superdisk or zip drive, where the medium can be write protected. It takes a bit of effort to pull that trick off if you start with a full-blown distribution like Red Hat, Mandrake, or SuSE.

Most modern computers can boot off of an IDE or Small Computer System Interface (SCSI) CD-ROM. If you are so inclined, you could make the CD-ROM your boot medium. There are some interesting projects that provide software tools for building fast-recovery CDs that could be adapted to suit your particular needs. They are:

http://mkcdrec.ota.be/

www.microwerks.net/~hugo/

Just as you need tools to do your work, the crackers need tools to do their work, dastardly as it is. Their tools often require compilers, so be sure to remove yours from your high security systems. Because you can have a development system with the same version of Linux as your high-security production systems and you can arrange for the crackers to be unable to get to your development system (because of a firewall, separate unconnected network, etc.) this would not be a significant problem for you. It certainly would be for them.

However, this forces them to worry about a compatible version of Linux, including compatible dynamic libraries. This will not stop all of them, but it will cause some not to bother and will slow down the determined ones. All compilers and language tools should be removed from the system including the C and C++ compilers, ld, as, Java tools, Python, BASIC, and Perl (unless you need Perl). Realistically, under Linux this makes less of a difference than under other platforms because Linux is very popular among crackers too. It will block exploits where a cracker has only partial access to your system such that he must "hand key" the source to an exploit program and then compile and run it.

Unless you need them, consider removing tar, gzip, gunzip, zcat, ar, uudecode and uuencode, *news*, and inn* because these make it easier for crackers to move files into and out of your system. Remove the C compiler, Perl interpreter, and related programs to make it harder for crackers to build programs to further infiltrate your system. Be sure to remove the RPMs or *.tar.gz versions of them too. I truly hate hobbling a system like this but it is important to increase security. rpm --uninstall package will remove the package from disk on Red Hat and its descendents.

Login simulators are a severe problem where a cracker posing as a user (or a dishonest user) will run a program that simulates the login sequence to trick users into revealing their passwords. "Defeating Login Simulators" on page 325 covers a short explanation of this problem and a quick solution that is reasonably effective. Now solutions that are more effective are considered. They do require more effort to implement, one requiring minor kernel changes. Beyond updating the /etc/issue file (and educating users), the only further solution that would not require a lot of effort would be to create an effective auto-logout program. Although some shells have a primitive auto-logout feature, it can be defeated easily by an intruder turning the feature off or running her own shell.

However, the kernel conveniently updates a tty device's access time only when a keystroke from the keyboard is read. This fact can be used to write an effective auto-logout program.^[9] The auto-logout program would scan /proc/*/fd/* periodically to determine all open file descriptors that are tty devices. (It would not do simply to scan the utmp file of logged-in users because an unmodified Linux kernel does not kill background processes on logout that might be listening on a port. Although one of these processes would be fighting login for characters, it would get the password line half of the time.)

^[9] Even this could be defeated by someone leaving a weight on the keyboard that an unsuspecting user would remove and then provide the account name and password to the login simulator. This problem could be resolved by updating the access time only if the character typed is different from the previous character and accounting for function keys possibly generating multiple characters.

As part of the Digital VT* terminal emulation there might be support for a "Who are you?" feature whereby a certain escape sequence sent to the "terminal" will generate a response that looks like it was typed. This possibility would need to be analyzed to see if the cracker could use it to make it appear that keyboard input was received and thus defeat the auto-logout program.

If you have serial ports that are used for special purposes, a table of these devices could be created that the program would leave alone. You also might want to have the program exempt devices owned by root with the assumption that root knows better. It also would ignore any process whose /proc/PID/exe has the same inode and device as getty by using the stat() system call and comparing st_dev and st_ino. The program then would check the access time of each unique tty device for any that were more than X minutes old, indicating an idle tty. For any of these it would send a terminate signal to each process that has the tty open, wait perhaps 15 seconds, and then send a kill signal.

This approach requires some minor kernel modifications to prevent its defeat by crackers.

1. The utime() system call would need to return EPERM^[10] if someone other than root tried to change the access time of a character device. This will prevent the login simulator from updating its tty device's access time periodically.

   ^[10] The error number meaning Error, PERMission denied.

2. In the unmodified kernel the tty device's access time is updated, not when the character is typed, but when a program reads the character. This has the consequence that the login simulator could, after opening the tty device and putting it in raw mode, sleep a few seconds to allow the rogue to enter a specified large number of characters in the keyboard device's input buffer.

   The program can display a fake login prompt and then read one character every 21 seconds (25.5 seconds less a safety margin) to update the tty device's access time. The specific way to do this would be to to manipulate raw mode's c_cc[VMIN] value so that the minimum number of characters needed for the read() to return is one more than the number that the cracker has entered. The c_cc[VTIME] value for timeout would be set to 255 deciseconds (25.5 seconds).

Thus, if a hapless user starts typing her account name, the login simulator's read() returns immediately and normal harvesting of her account and password can commence. If instead there is nobody at the keyboard, the read() would be interrupted by an alarm() timeout whose value would be slightly less than 25.5 seconds (21 seconds).

This alarm is necessary because, otherwise, when the timeout of 25.5 seconds is reached the read() would return the characters that have been typed (by the cracker). This scheme will allow the login simulator to run for 255 ÷ 10 254 seconds or almost two hours plus the intended auto-logout value.

Although these features are documented in the manual pages, my article, "Unraveling the Mysteries of System V's Terminal Interface," in the November 1984 issue of UNIX Review Magazine gives additional details. A copy of this article is available at

www.verysecurelinux.com/84nov.html

The cracker could still get away with doubling the auto-logout threshold by using this algorithm with a single keyed character in the input buffer.^[11] I now present the change to the kernel code that is required to limit the cracker to extending the intended auto-logout timeout value to a factor of two (rather than a factor of 255). (I have not tested this code or verified the exploits.)

^[11] This doubling too could be eliminated by removing the setting of access time in read_aux() entirely and instead implementing it in put_queue() in drivers/char/keyboard.c. This probably would require using tty->device to search for the device's in-memory inode information and then updating i_atime. This might be too slow in this interrupt-level code.

In the read_aux() function in drivers/char/pc_keyb.c change

 if (count-i) {          file->f_dentry->d_inode->i_atime = CURRENT_TIME;          return count-i; } 

to

 if (count-i) {                    /*                     * Update atime only when                     * last char read.                     */          if (queue_empty())                   file->f_dentry->d_inode->i_atime                     = CURRENT_TIME;          return count-i; } 

There is a much simpler alternative that would detect a standard login simulator that harvests the password and then either exits or execs^[12] login to start the real login. This would be to create a simple program to scan the wtmp file and warn if one user logs in within a few seconds of another user logging out of the same tty. This would detect evidence of a login simulator after the fact. Of course by this time, if the cracker is really clever, he would have transmitted root's password to his own system (or to a compromised system that cannot be traced to him) and fed the password to a program that then logged in as root and inserted a Trojan and cleaned up the log files before saying Login incorrect. The slightly increased delay probably would not be noticed.

^[12] Hopefully, you closed the security hole that allows users to exec login via chmod 700 /bin/login as discussed in "Defeating Login Simulators" on page\~325.

You should pick the passwords and ensure that they are not easily cracked. (See "Avoiding Weak and Default Passwords" on page 42 for more on passwords.) Consider the relative risks of a network cracker modifying your kernel and forcing a reboot, versus a rogue with physical access to your system being able to insert a rogue floppy into the floppy drive and pressing reset.

If the risk of a successful network attack is greater than the risk of a physical attack, you might want to configure your CMOS to boot only from floppy (Drive A) and insert a read-only floppy with your kernel on it. Ditto for CD-ROMs. This will prevent a network intruder from adding a Trojan horse to your kernel and forcing a reboot from your hard disk.

A typical kernel Trojan horse would be to modify an unassuming system call to give its invoker root powers (by poking task_struct) if its invoker has set a flag somewhere, say by putting an unusual value in a register or passing an unusual value to the system call.

See "Confessions of a Berkeley System Mole" on page 373 for the entertaining and enlightening details.

Configure LILO to come up multiuser with no decisecond wait for the user to enter options to prevent someone from asking it to boot up with a single-user root shell. You can do this via

 lilo -d 0 

The LILO program has other features such as the

 password=password 

to specify a password and

 restricted 

that requires a password only if parameters are specified to LILO from the keyboard (such as single to specify booting to single-user mode which otherwise would allow gaining a root shell). You will want to change LILO's configuration file's permissions from the default 644 to 600 to prevent others from reading the clear text password with the following command:

 chmod 600 /etc/lilo.conf 

Of course, you already ensured that the various disk devices are accessible only by root. These would be the /dev/hd*, /dev/sd*, or /dev/md* devices for most systems.

Consider using separate physical networks for your secure systems and for your less secure systems. This would prevent an attack launched from a desktop system, say, one that a cracker broke into due to its unauthorized dial-up modem connected to an ISP.

The Weather Channel recognizes that their reputation and their business would be damaged if a cracker was able to plant false weather data, say, snowstorms in Miami or serious weather that causes businesses and schools to evacuate unnecessarily. If the false data blotted out a warning of actual hazardous weather, there could be loss of life.

To help thwart this, the Weather Channel has a network completely separate from their regular corporate network to receive their weather data feeds and route them to their weather computers.

Physically secure the system and anything connected to it, including monitors, keyboards, mice, and peripherals such as tape, floppy and CD drives, and printers. Keep backup tapes physically secure until erased. Any printouts containing confidential data should be kept locked up until securely shredded. Typically this means a server room. Some sites keep an activity log of all significant events, including backups and changes to hardware or software configuration.

At one well-run firm I consulted for we kept all backup tapes in the computer room, which was locked and alarmed at all times (and had only a few keys). Backup tapes destined for off-site storage were first loaded into locked boxes inside the computer room and signed for receipt by the storage company.

The boxes of tapes were stored in a vault until returned for recycling. On the rare cases where the courier did not show up, one of us brought the lock-box home for safe off-site storage for the week.

Encrypt any network communications with SSH or equivalent, if possible. Any cables and equipment carrying unencrypted data should be physically protected, typically carried inside steel pipe welded together to prevent opening except via a diamond-tipped saw. See also "Trust No One The Highest Security" on page 356 and "Very High GPG Security" on page 444.

---

Top