When You Find a Bug, You re Not Done

When You Find a Bug, You're Not Done!

When your tests find a defect, you can pat yourself on the back and start running your other test plans, right? No. You need go look for the bug's variations. The process of identifying variations can be rewarding when a bug variation is discovered. Here's a simple example. Let's assume the application processes text IP addresses, such as 172.100.84.22, and your test plans note that a buffer overrun occurs if the first octet is not a number but rather a long string, such as aaaaaaaaaaaaaaaaaaaa.100.84.22. If there's a bug in the code that processes the first octet, the chances are good the same bug exists when processing the other octets. Hence, when you discover this bug, you should update your test code to inject the following data:

aaaaaaaaaaaaaaaaaaaa.100.84.22 172.aaaaaaaaaaaaaaaaaaaa.84.22 172.100.aaaaaaaaaaaaaaaaaaaa.22 172.100.84.aaaaaaaaaaaaaaaaaaaa

A more methodical way to perform this kind of analysis is to follow these steps:

1. Reduce the attack.

   What aspects of a flaw are superficial and can be eliminated from the test plan? Eliminate them and create the base exploit. In the previous example, the IP address might be part of another data structure that has no effect on the defect. It's only the IP address we're interested in.

2. Identify fundamental exploit variables.

   Identify the individual parts of an exploit that can be modified to create new variations. The variables in an exploit are often easy to identify. In our example, the variables are the four IP octets.

3. Identify possible meaningfully distinct variable values.

   It's difficult to identify values that really are meaningfully distinct. The important variable values involved in a successful exploit are usually little-used or under-documented. A significant amount of time should be devoted to uncovering all the meaningfully distinct variable values. Ideally, this should involve analyzing existing documentation and source code or talking with the developers.

   No two defects will ever have the same set of exploit variables with the exact same set of values. However, many exploits will have some common variables. This means the set of values for those variables can be saved and reused for other tests. In the previous example, it appears that any long string in an octet causes the failure. You should refer to Figure 19-1 to help determine valid and invalid variable types.

4. Test the full matrix of variables and variable values.

   If you have properly identified all variables and all values, you now have all you need to construct a test of variations that isn't covered by an existing test case. This leads to more complete test code, but even with broad test plans the test must code must be good quality, and that's the next subject.