Aim for Good, Not Perfect

No individual touchpoint or tool can solve all of your software security problems. Static analysis tools are no different. For starters, static analysis tools look for a fixed set of patterns, or rules, in the code. Although more advanced tools allow new rules to be added over time, if a rule hasn't been written yet to find a particular problem, the tool will never find that problem. When it comes to security, what you don't know is pretty darn likely to hurt you, so beware of any tool that says something like, "Zero defects found, your program is now secure." The appropriate output is, "Sorry, couldn't find any more bugs."

A static analysis tool's output still requires human evaluation. There's no way for any tool to know automatically which problems are more or less important to you, so there's no way to avoid trawling through the output and making a judgment call about which issues should be fixed and which ones carry an acceptable level of risk. Plus, knowledgeable people still need to get a program's design right to avoid any flaws. Static analysis tools can find bugs in the nitty-gritty details, but they can't even begin to critique design. Don't expect any tool to tell you, "I see you're implementing a funds transfer application. You should tighten up the user password requirements."

Finally, there's computer science theory to contend with. Rice's theorem,^[2] which says (in essence) that any nontrivial question you care to ask about a program can be reduced to the halting problem,^[3] applies in spades to static analysis tools. In scientific terms, static analysis problems are undecidable in the worst case. The practical ramifications of Rice's theorem are that all static analysis tools are forced to make approximations and that these approximations lead to less-than-perfect output.

^[2] See <http://en.wikipedia.org/wiki/Rice's_theorem> if you need to understand more about Rice's theorem.

^[3] See <http://en.wikipedia.org/wiki/Halting_problem> if you're not a computer science theory junkie.

Static analysis tools suffer from false negatives (in which the program contains bugs that the tool doesn't report) and false positives (in which the tool reports bugs that the program doesn't really contain). False positives cause immediate grief to any analyst who has to sift through them, but false negatives are much more dangerous because they lead to a false sense of security.

A tool is sound if, for a given set of assumptions, it produces no false negatives. Unfortunately, the downside to always erring on the side of caution is a potentially debilitating number of false positives. The static analysis crowd jokes that too high a percentage of false positives leads to 100% false negatives because that's what you get when people stop using a tool. A tool is unsound if it tries to reduce false positives at the cost of sometimes letting a false negative slip by. Most commercial tools these days are unsound.