Chapter 19: Binary Auditing: Hacking Closed Source Software

Binary versus Source-Code Auditing: The Obvious Differences

Binary auditing can be likened to source-code auditing in that you're looking for the same bug classes and flaws in software; however, the method for looking for them has changed. If you are already familiar with source-code auditing, you probably won't have to change your thought process much. However, your methodology will change quite a bit.

First and foremost, you'll need an excellent understanding of the assembly language relevant to the platform on which your binary will run. If you're unclear on any important instructions, you will likely misinterpret much of the code you read and end up confused and frustrated. If you're not able to read and understand a disassembly, thoroughly learning the relevant assembly language is a good place to start.

Some binaries, especially p-code binaries such as Java classes or Visual Basic applications, can be fully decompiled to something that closely resembles their original source code. However, most binaries cannot be reliably decompiled with today's tools. This chapter will focus on auditing Intel x86 binaries, especially those compiled with the Microsoft Visual C++ compiler.

When auditing a binary, as when auditing source code, it's paramount to understand the code you're reading. However, what might be a very obvious security check in source can often translate into one or two instructions. Therefore, it's definitely necessary to remain aware of program execution at any point in a function. For example, it's often necessary to know what values are stored in what registers at a certain point in execution, and many values may be swapped in and out of a particular register in any given block of code.

Some vulnerabilities are just as easy or easier to spot in a binary than in source code; however, most bugs will be more subtle and harder to detect for someone attempting their first binary audit. As you become more familiar with code constructed by certain compilers, auditing binaries will become nearly as easy as auditing source code.