Software Penetration Testinga Better Approach

All is not lostsecurity penetration testing can be used effectively. The best approach bases penetration testing activities on security findings discovered and tracked from the beginning of the software lifecycle: during requirements analysis, architectural risk analysis, and so on. To do this, a penetration test must be structured according to perceived risk and offer some kind of metric relating the security posture of the software at the time of the test to risk measurement. Results are less likely to be misconstrued and used to declare pretend security victory if they are related to business impact through proper risk management. (See Chapter 2, which describes a risk management framework amenable to feeding security testing.)

Penetration testing is about testing a system in its final production environment. For this reason, penetration testing is best suited to probing configuration problems and other environmental factors that deeply impact software security. Driving tests that concentrate on these factors with some knowledge of risk analysis results is the most effective approach. Outsidein testing is great as long as it is not the only testing you do. The modern approach that I describe throughout the remainder of this chapter is much more closely aligned with risk-based security testing (see Chapter 7) than it is with application penetration testing as practiced by most consulting shops today. Be careful what you ask for!

Make Use of Tools

Tools (including the static analysis tools discussed in Chapter 4) should definitely be used in penetration testing. Tools are well suited to finding known security vulnerabilities with little effort. Static analysis tools can vet software code, either in source or binary form, in an attempt to identify common implementation-level bugs such as buffer overflows. Dynamic analysis tools can observe a system as it executes. These tools can submit malformed, malicious, and random data to a system's entry points in an attempt to uncover faultsa process commonly referred to as fuzzing [Miller et al. 1995]. Faults are then reported to the tester for further analysis. When possible, use of these tools should be guided by risk analysis results and attack patterns. (See the following box, Tools for Penetration Testing.)

Tool use carries two major benefits. First, when used effectively, tools can carry out a majority of the grunt work needed for basic software penetration testing (at the level of a fielded system). Of course, a tool-driven approach can't be used as a replacement for review by a skilled security analyst (especially since today's tools are by their nature not applicable at the design level), but a tool-based approach does help relieve the work burden of a reviewer and can thus drive down cost. Second, tool output lends itself readily to metrics. Software development teams can use these metrics to track progress over time as they move toward a security goal. Simple metrics in common use today do not offer a complete picture of the security posture of a system. Thus it is important to emphasize that a clean bill of health from an analysis tool does not mean that a system is defect free (recall the discussion of badness-ometers from Chapter 1). The value lies in relative comparison: If the current run of the tools reveals fewer defects than a previous run, progress has likely been made.

Tools for Penetration Testing

A number of tools purport utility for application security testing. Beware of those tools that claim to be a "hacker in a box." For a rant about (super-lame) application security testing tools like that, see the box Application Security Testing Tools: Good or Bad in Chapter 1. If we leave aside silly tools that do things like send 50 a's over port 80 in an attempt to cause a buffer overflow, there are a number of very useful tools left.

Fault Injection Tools

Software fault injection has an interesting future as a potential security technology. I coauthored the first book on this technique with Jeff Voas. If you're up for a tedious read, check out Software Fault Injection [Voas and McGraw 1998].

One of the most interesting modern fault injection engines for security is the Cenzic tool. This tool uses browser shunts to intercept transactions and allow an analyst to play around with them. Though this kind of testing activity is possible with perl and a command-line interface, results collection and automation make the second-generation tools worth considering. Cenzic began its life as a hacker tool called Hailstorm written by Greg Hoglund. Greg has since abandoned Cenzic, but the company carries on. For more, see <http://www.cenzic.com>.

James Whittaker and his merry band of security grad-students-turned-testers from Florida Tech created a company called Security Innovation. Whittaker takes malicious input to a new level. The Holodeck tool is useful for playback, security fault injection, and a number of other interesting tests. For more, see <http://www.sisecure.com>.

Other Tools

Also worth a look are application attack tools from SPI Dynamics <http://www.spidynamics.com> and Fortify Software <http://www.fortifysoftware.com>.

Dan Geer reports that Dave Aitel and company are having considerable success using Immunity's CANVAS tool <http://www.immunitysec.com/products-canvas.shtml>.

Another fun little tool is a shim that allows you to capture all of the HTTP traffic your browser sees. One such tool is here <http://www.ieinspector.com/httpanalyzer/>. Since these tools run in the browser, you don't have to jump through any hoops to see HTTPS traffic. Nice for messing around with Web interfaces (as a way of "making the client invisible") [Hoglund and McGraw 2004].

The main thing to remember about all of these tools is that they are most valuable in the hands of an expert. Whatever you do, don't set your five-year-old up with a power saw, or if you do, don't be too shocked by the messy results.

Getting Past Fault Injection: The Attacker's Toolkit

In Exploiting Software, Greg Hoglund and I go to great lengths to describe the attacker's toolkit. Tools commonly used by bad guys make great fodder for penetration testers. This is where your black hat gets its dark color. Any hacker (malicious or otherwise) worth his or her salt uses these tools.

• Disassemblers and decompilers: The notion that source code is required for security attacks is ridiculous. People interested in really understanding the ins and outs of software use disassemblers and decompilers all the time.

• Control flow and coverage tools: Though coverage tools were designed to help testers ascertain test effectiveness, they also make excellent attack tools. If you know that there is a vulnerability sitting deep in a target program, the next order of business is getting there. Control flow, data flow, and coverage analysis tools help.

• APISPY32: A tool particular to Win32, but well worth a mention. This tool allows an analyst to know when a program uses outside DLLs and/or other libraries. This kind of information is useful for interposition attacks, DLL substitution attacks, and a whole lot of other fun.

• Breakpoint setters and monitors: Debuggers are your friend. Breakpoints can even be set on target hardware to look for memory page access, device access, and the like. Understanding a program is much easier if you can watch it and freeze it at will.

• Buffer overflow: You know all about this one, right? See Building Secure Software if you want to know how a buffer overflow attack really works in painstaking detail [Viega and McGraw 2001]. Advanced attacks are covered in Exploiting Software [Hoglund and McGraw 2004]. Also see the new book The 19 Deadly Sins of Software Security [Howard, LeBlanc, and Viega 2005].

• Shell code: Payload anyone? The attacker's goal is to subvert control and run a program. This is the kind of program that gets run. See The Shellcoder's Handbook for more [Koziol et al. 2004].

• Rootkits: The ultimate weapon. At the apex of the attacker's toolkit is the rootkit. Rootkits hide away from standard system observers, employing hooks, trampolines, and patches to get their work done. Sophisticated rootkits run in such a way that other programs that usually monitor machine behavior can't easily detect them. A rootkit thus provides insider access only to people who know that it is running and available to accept commands. Kernel rootkits can hide files and running processes to provide a back door into the target machine. An excellent book, Rootkits: Subverting the Windows Kernelthe first book in the world on rootkitswas published in August 2005 [Hoglund and Butler 2005].

Other tools also worth a (much briefer) mention include the following:

• Debuggers (user-mode)

• Kernel debuggers

• SoftIce

• Fuzz

• Boron tagging

• The "depends" tool

• Grammar rewriters

Test More Than Once

As it stands today, automated review is best suited to identifying the most basic of implementation defects. Human review is necessary to reveal flaws in the design or more complicated implementation-level vulnerabilities (of the sort that attackers can and will exploit). However, review by an expert is costly and, for reasons just described, can be ineffective if the "expert" is not. By leveraging the seven software security touchpoints described in this book, software penetration tests can be structured in such a way as to be cost effective and give a reasonable estimation of the security posture of the system.

Penetration testing can benefit greatly from knowledge of the security risks built into a system. No design or implementation is perfect, and carrying risk is usually acceptable. Penetration testing can help you find out what this means to your fielded system. In fact, penetration testing in some sense collapses the "risk probability wave" into something much more tangible when testing clarifies ways that a risk can be exploited. That is, if you know what your likely risks are in the design, you can use penetration testing to figure out what impact this has on an actual fielded system.

As noted earlier, static and dynamic analysis tools should be uniformly applied; this holds true at the subsystem level too. In most cases, no customization of basic static analysis tools is necessary for component-level tests. However, dynamic analysis tools will likely need to be written or modified for the target component. Such tools often involve data-driven tests that operate at the API level. Any tool should include data sets known to cause problems, such as long strings, strange encodings, and control characters [Hoglund and McGraw 2004]. Furthermore, the design of the tool should reflect the security test's goalto misuse the component's assets, to violate intercomponent assumptions, or to probe risks. Customizations are almost always necessary.

Penetration testing should focus at the system level and should be directed at properties of the integrated software system. For efficiency's sake, testing should be structured in such a way as to avoid repeating unit-level testing (as described in Chapter 7), and should therefore be focused on aspects of the system that could not be probed during unit testing.

In order to be defined as penetration tests, system-level tests should analyze the system in its deployed environment. Such analysis may be targeted to ensure that suggested deployment practices are effective and reasonable, and that assumptions external to the system cannot be violated.