7.1 Prelude: What's Wrong with This Picture?
Before turning to the specifics of securing and monitoring Unix systems, let's take a brief look at three well-known historical Unix security problems (all of them were fixed years ago):
The Sendmail package used to include a debug mode designed to allow a system administrator to type in raw commands by hand and observe the effects. Unfortunately, because anyone can run the sendmail program, and because it runs as setuid root, a nefarious user could use sendmail to execute commands as root. This is an example of a security hole created by a back door in a program: an execution mode that bypasses the program's usual security mechanisms.
Traditionally, the passwd -f command enabled users to change the information in the GECOS field of their password-file entries. However, as originally implemented, the command simply added the new information to the user's GECOS field without examining it first for characters such as, for example, colons and new lines. This oversight meant that a treacherous user could use the command to add an entry to the password file. This is an example of a program's failure to validate its input. The program simply assumes that the input it receives is valid and harmless without checking that it is in the form and length that is expected.
Another variation of this problem is called a buffer overflow. A buffer overflow occurs when a program receives more input than the maximum amount that it is able to handle. When it later chokes on that input, there can beunexpected side effects, including the ability to run arbitrary commands in the user context of the program (often root). Modern programs are usually written to reject input that is too large, but we are still finding and fixing such bugs in programs written in previous years/decades.
The finger command displays various information about the user you specify as its argument: his full name and other password-file information, as well as the contents of the .plan and .project files in his home directory. finger is designed to make it easy to find out who is on the system and how to contact them. In the past, however, the command failed to check whether the .plan file in a user's home directory was readable by the user running finger before displaying its contents. This meant that an unscrupulous user could create a .plan in his own home directory as a link to any file on the system, then run finger on his own account and be able to view the contents of the target file, even when its file protection mode prevented his access. This is an example of a bug that arises from unconscious assumptions about the circumstances and context in which the program will be run.
What do these three items have in common? They all illustrate the fundamental Unix view that the system exists in a trustworthy environment of reasonable people. In all three cases, the programs failed to anticipate or check for unintended uses of their features. Seeing these problems merely as ancient bugs that have been long fixed misses the important point that such a view is inherent in the Unix operating system at a very deep level. This belief is evident even in the rhetoric of Unix commands as simple tools performing one task in a general and optimal way. You can do a lot more with a screwdriver than tightening and loosening screws.