13.9 Mitigating Process Manipulation Risks

There are a number of techniques that can be mitigate underlying security issues, so that even if your applications or network services are theoretically vulnerable to attack, they can't be practically exploited.

Here are the five main approaches:

• Nonexecutable stack and heap implementation

• Use of canary values in memory

• Running unusual server architecture

• Compiling applications from source

• Active system call monitoring

As with any bolt-on security mechanism, there are inherent positive and negative aspects. Here I discuss these approaches and their shortfalls in some environments.

13.9.1 Nonexecutable Stack and Heap Implementation

An increasing number of operating systems support nonexecutable stack and heap protection (including OpenBSD, Solaris, and a small number of Linux distributions). This approach prevents the instruction pointer from being overwritten to point at code on the stack or heap (where most exploits place their shellcode in user-supplied buffers).

To defeat this kind of protection, return-into-libc, or a similar attack executes in-built system library calls that can be used to compromise the system. These attacks require accurate details of loaded libraries and their locations, which can only practically be gained through having a degree of local system access in the first place.

From a network service protection perspective, implementing nonexecutable stack and heap elements can certainly prevent remote exploitation of most memory manipulation bugs.

13.9.2 Use of Canary Values in Memory

Windows 2003 Server, OpenBSD, and a number of other operating systems place canary values on the stack (and sometimes heap) to protect values that are critical to logical program flow (such as the saved frame and instruction pointers on the stack).

A canary value is a hashed word that is known by the system and checked during execution (e.g., before a function returns). If the canary value is modified, the process is killed, preventing practical exploitation.

Managing canary values can see a 5% to 10% slow-down in most system operations, which may be exaggerated in certain conditions (on database servers, for example). A number of academic researchers have analyzed the results of various stack and heap protection mechanisms that use canaries; their work is accessible from the following locations:

http://www.wirex.com/~crispin/opensource_security_survey.pdf

http://downloads.securityfocus.com/library/nbnfi-fe20011302.pdf

http://www.rsaconference.com/rsa2003/europe/tracks/pdfs/hackers_t14_szor.pdf

13.9.3 Running Unusual Server Architecture

Security through obscurity can certainly buy you a lot of time, and raise the bar to weed out all the script kiddies and opportunistic attackers who are attempting to compromise your servers.

One such method is to use a nonstandard operating system and underlying server architecture, such as NetBSD on a Sun SPARC system. A benefit of using a big endian architecture such as SPARC is that stack and heap off-by-one bugs aren't practically exploitable, and Intel x86 shellcode in prepackaged exploits (such as those found on Packet Storm, SecurityFocus, and other sites) won't be effective.

13.9.4 Compiling Applications From Source

As overflows become more complex to exploit and identify, they rely on more variables to remain constant on the target system in order to be exploited successfully. If you install precompiled server applications (such as OpenSSH, WU-FTP, Apache, etc.) from RPM or other packaged means, the GOT and PLT entries will be standard, and known to attackers.

However, if you compile your applications (server software in particular) from source yourself, the GOT and PLT entries will be non-standard, reducing the effectiveness of a number of exploits that expect function addresses to be standard in order to work.

13.9.5 Active System Call Monitoring

A small number of host-based IDS systems now perform active system call monitoring, to establish known logical execution paths for programs. If the program attempts to access a sensitive system call it usually doesn't, the proactive monitoring system kills the process. An example of this would be if an attacker attempts to remotely spawn a command shell, and calls to socket( ) are made when the policy defines that the process isn't allowed to make that system call.

Sana Security (http://www.sanasecurity.com) and Internet Security Systems (http://www.iss.net) produce active system call monitoring solutions, also known as Intrusion Prevention Systems (IPS), for Windows systems.

Systrace is an open source Unix-based alternative by Niels Provos. Systrace is part of NetBSD and OpenBSD, which provides active system call monitoring according to a predefined policy. It's also available for Linux and MacOS from these locations:^[1]

^[1] URLs for tools in this book are mirrored at the O'Reilly site, http://examples.oreilly.com/networksa/tools.

http://www.systrace.org/

http://www.citi.umich.edu/u/provos/systrace/