Incorporating Findings Back into Development

Perhaps the most common failure of the software penetration testing process is failure to identify lessons learned and propagate them back into the organization. As mentioned, it is tempting to view the results of a penetration test as a complete and final list of issues to be fixed rather than as a representative sample of faults in the system. Of course, even in this case, the existence of such a list does not do anything to actually fix the system. One of the major barriers to software security success is getting organizations to get around to fixing the problems found every day by security consultants. Don't for a minute believe that penetration testing results make you any more secure; only acting on them does.

Mitigation strategy is thus a critical aspect of any penetration test. Rather than simply fixing only those issues identified, developers should carry out a root-cause analysis of the identified vulnerabilities. For example, if a majority of vulnerabilities are buffer overflows, the development organization should determine just how these bugs made it into the code base. In such a scenario, lack of developer training, misapplication (or nonexistence of) standard coding practices, poor choice of languages and libraries, intense schedule pressure, failure to use a source code analysis tool, or any combination thereof may ultimately represent an important cause.

Once a root cause has been identified, mitigation strategies should be devised to address the identified vulnerabilities and any similar vulnerabilities in the software. Furthermore, best practices should be developed and implemented to address such vulnerabilities proactively in the future. (See Chapter 10 for a discussion of how this idea relates to a large-scale software security program.)

Going back to the buffer overflow example, an organization may decide to train its developers and eliminate the use of potentially dangerous functions, such as strcpy(), in favor of safer string-handling libraries such as those found in the C++ Standard Templates Library (STL). Perhaps a static analysis tool can be used to enforce this decision.

A good last step involves using test result information to measure progress against a goal. Where possible, tests for a mitigated vulnerability should be added to automated test suites (which can be used in regression testing). If the vulnerability resurfaces in the code base at some point in the future, any measures taken to prevent the vulnerability should be revisited and improved. As time passes, iterative penetration tests should reveal fewer and less severe defects in the system. If a penetration test reveals serious severe problems, the "representative sample" view of the results should give the development organization real reservations about deploying the system.