Incorporate Debug Tests into Unit Test Suites

Incorporate Debug Tests into Unit Test Suites

Because one small test will often lead us to discover the true source of a bug, it follows that effective debugging should involve lots of testing. There's simply no way to build a reliable system without thoroughly testing it.

Although many programmers may be unfamiliar with the XP style of coding (in which writing the tests is interleaved with writing the code), there is one form of testing that everyone does while coding: debugging . Some programmers write these tests using print statements, others write them to work with a standalone debugger, others with the debugger in their IDE of choice.

Quite often, you will see programmers discard these tests once they've gotten the program to run correctly. This is a waste of very good tests. Why not incorporate them into the unit test suite over a program? After all, if one of them were to exhibit an unexpected result, we'd want to know about it.

Incorporating debugging tests into the unit tests can be done with relatively little effort, but there are some adaptations that have to be made. Frequently, we'll write tests for debugging to display information about the internal state of the program, whereas we'll write unit tests to signal only if they fail. That's because debugging and unit testing serve different purposes.

When debugging, we are trying to form a more accurate model of the program's behavior, so as to diagnose a bug. But when testing, all we want to know is whether the code passed the tests. If instead we wrote the unit tests to print out a result and then manually checked that it was what we expected, we'd waste a lot of time (or, more likely, we would seldom run the unit tests). So, unit tests tend to be written such that the result is solely one of "pass" or "fail".

Debugging tests can still be used as unit tests with only slight modification. Consider that when the program is working correctly, the result of running a debugging test will match some expected result. It is straightforward to modify such a test so that, instead of simply printing out this result, it compares the result to what's expected. It can then be incorporated into a unit test suite quite easily.

Just as debugging helps in developing a large suite of unit tests over a program, unit tests can help significantly when debugging. When diagnosing a bug, if you can first run a suite of unit tests and verify that they pass, you can rule out a huge number of potential explanations for a bug. In this way, unit tests allow you to leverage your cognitive energy when modeling program behavior. This is yet another way that debugging is like performing a scientific experiment.

When a physicist forms an explanation of an experimental result, he automatically rules out all sorts of explanations that would defy a set of accepted principles about the way the world works. For example, he assumes that the results of his experiment will not change depending on the current weather conditions on Jupiter (well, unless he is performing an experiment on Jupiter), or depending on what he plans to eat for dinner. Unit tests enforce accepted principles of program behavior.

Bug Patterns Help Diagnose Bugs More Quickly

Sherlock Holmes, the world's greatest detective and master debugger, was so adept at his trade partly because he possessed an unparalleled degree of rationality. But debugging software, like thwarting crime, involves more than just an unswervingly rational mind. It also involves knowledge of a great many specific cases of bugs, their causes, and their remedies.

Expert ability in any field is not just the result of raw intelligence. It's also the result of a great deal of experience. That experience provides many specific examples that can be generalized into patterns.

Experts apply these patterns to new situations and reason about them more effectively. In particular, they can disregard irrelevant details more quickly and focus on what's important. Consider the following behavior of a variety of experts:

• Chess masters study numerous famous games and openings. The knowledge they acquire isn't simply rote memorization of the moves of those games. Instead, they extract patterns and principles and apply them to new games .

• Doctors study more than just anatomy and disease, but also numerous case studies, from which they extract patterns to apply to new cases.

• Astronaut crews spend countless hours running through drill after drill of systems failures on simulated missions. It's unlikely that the particular sequence of failures occurring on a simulated mission would occur during the real thing, but the astronauts extract patterns from this training that allow them to respond to a variety of problems more quickly.

• Historians study the political events of the past, extract patterns, and apply the knowledge when recommending actions to take in the future. Economists perform a similar function.

• Architects were the first to actually record various patterns of their discipline directly. New architects have been able to study these patterns and quickly learn from the years of experience of numerous past architects. As long as the rationale behind the patterns is taught with the patterns, this style of teaching can be an extremely efficient way to transfer knowledge to new students. (For more on teaching new programmers, see the section entitled "Learning in a Fast-Paced World" in Chapter 1 .)

Software engineers followed this behavior, too. They started copying the approach of the architects with the book Design Patterns: Elements of Reusable Object-Oriented Software (Gamma et. al., 1994). In that book, much of the acquired wisdom of the software community concerning how to design new systems was condensed into a set of patterns, with discussions as to when each pattern was applicable . This was followed by several books on anti patterns, providing negative examples to complement the positive. But all of these patterns are at the level of design. There are patterns involved with programming that occur below this level.

Sometimes, it's possible to identify a recurring bug pattern, complete with common symptoms, causes, and remedies. If you're aware of the bug pattern, you can then identify occurrences of that pattern more quickly and fix them.

In the case of simple typos, which occur in extraordinary variety, there aren't many common patterns of behavior that result. Fortunately, if you're programming with XP techniques, the occurrences of undetected typos should be extremely rare. That's good, because bugs resulting from typos can be just as difficult to diagnose as much more subtle bugs.

The more subtle bugs to which I am referring generally scuttle at the edge of coding and design. They involve more than just mistyping a name , but they aren't problems purely at the level of class diagrams; there's a lower-level problem with the code.

By examining these bug patterns, we can do more than just identify them; we can also consider what coding practices could help prevent recurrences of each pattern in the future. We will then have a much better appreciation of that coding practice than we would have if the methods were simply recommended for no reason at all.