8.3 Create a test case

Almost all works on debugging recommend creating a standalone test case from the original defective program. There are several benefits to creating such a test case.

First, if done correctly, your test case should take significantly less time to manifest the problem than the original program. This means that you will be able to complete experiments much more quickly than if you run the original program.

Second, if the test case can execute standalone and indicate whether it passed or failed to a script that executes it, it can easily be incorporated into a suite of regression tests. This makes it more likely that the problem won’t resurface in the future.

Third, if it turns out that the problem isn’t in code that you’re responsible for, you can turn the problem over to the people who are responsible. Having a good test case ready will enable the responsible party to turn a correction around more quickly.

A test case typically has two parts: the program and the input data. In this heuristic, we focus on the program. The next heuristic focuses on the input data to that program.

If you’re working on platform like UNIX™, on which programs can return status codes that can be inspected by a shell script, your test case should return a 0 (for no problem) if the test passes and return a positive integer if the test fails. This makes it possible to easily include your test case in automated suites. In addition, the test case should print a pass/fail message, so that a simple search of the output will show you the outcome.

8.3.1 Cutting down a simple application problem

If you’re diagnosing a problem with an application that works like a file-oriented filter, you can use the following procedure to cut down the program into a more manageable test case. A filter takes all of its input from the operating system standard input and sends all of its output to the operating system standard output.

Create an input file that contains all of the keyboard entries up to, but not including, the step that causes the problem to manifest. On a UNIX™ system, you can use the script command to capture these entries in a file as you enter them.
Create a driver that replays these actions to the application. This may be as simple as a shell script that redirects the standard input to the previously captured entries.
Identify the procedure that responds to the input that manifests the problem. Just before the call to this procedure, insert a call to a new procedure that captures the state of the program at entry to this procedure. The program state includes the values of all global variables, actual arguments to the procedure, open files, and so forth. The new procedure should write all of these values to a text file.
Execute the driver and capture the state of the program at entry to the procedure that manifests the problem. Terminate the application at that point.
Create a new main procedure that sets all of the program state without having to execute that part of the application that normally creates it. The driver should do the following:
- Open all files and set them to the status that was captured.
- Set all global variables to the values captured.
- Call the procedure that manifests the problem with the argument values that were captured.
Remove all procedures from the application that were executed only before the procedure that manifests the problem. Use a call graph program to identify these if the application is large or you’re unfamiliar with all of its parts.
Remove the normal main procedure and insert the new main procedure you have created. You should now be able to run the program and see the problem manifested almost immediately.

8.3.2 Cutting down a GUI application problem

Cutting down a problem with an application that uses a graphical user interface (GUI) is more involved. Capturing the sequence of inputs required to duplicate a problem means capturing the mouse movements, mouse clicks, and keyboard entries that result in a series of GUI objects being manipulated. Not only do these mouse actions and keyboard entries need to be captured they must also be replayed up to the point at which the problem occurred.

Different GUI environments provide varying levels of support for capturing mouse actions and keyboard entries. Java provides the java.awt.robot class for recording and playing back user actions. A number of software packages provide analogous functionality for X/Motif and the Windows™ environment. See our Website for software recommendations.

The first step in cutting down a GUI application problem is to create a log that contains all of the keyboard and mouse entries up to, but not including, the step that causes the problem to manifest. Use the appropriate system-dependent logging functionality as described previously.

Second, create a driver that replays these actions to the application. This means linking in additional libraries that will read the log file and short-circuit the normal entry of keyboard and mouse actions.

From this point, you can follow steps 3 through 7 for cutting down a command-line application problem.

8.3.3 Cutting down a compiler problem

If you’re working on a compiler, you can use some special techniques to build a minimal test case.

Identify the procedure in the user’s application that was being processed when the compiler defect manifested itself. This evidence can be a premature termination of the compiler, an assertion message, generation of wrong code for that particular procedure, and so forth.
Correlate internal compiler data structures back to the source position in the user’s application. If the compiler you’re working on doesn’t annotate its representation of the user program with the source file name, line number, and column number of the corresponding lexical element of the user’s program, you’re working on a toy, not a real compiler. Fix this problem now; the payback will be nearly immediate.
Remove as much of the user application code that hasn’t been processed yet as possible. It is better to remove both the calls and the definitions of procedures that haven’t been processed. If you can’t remove the calls to procedures that haven’t been compiled, at least replace the bodies with stubs.
Perform a series of binary searches to remove irrelevant loop nests, code blocks, and statements from the offending procedure. In each search, you can effectively remove half of the items under question either by commenting them out or by using a preprocessor directive. If, after you remove them from the compilation, the problem still occurs, cut the remaining code in half and omit it from the compilation. Using this process recursively, remove all loop nests that aren’t needed to manifest the problem. Then, remove all code blocks from those loop nests that aren’t needed to manifest the problem. Finally, remove all statements from those code blocks that aren’t needed to manifest the problem.

How does creating a test case suggest hypotheses? It focuses your attention on just those parts of the program that are actually causing problems. If a test program doesn’t include a particular part feature of the application, there is no point in forming a hypothesis that the handling of that feature is the cause of the defect. By the time you have cut down a test case program to the minimum size required to manifest the defect, you will have eliminated a whole host of potential hypotheses from further consideration.

Creating a standalone test case is one of the first things we do when diagnosing a bug. If done correctly, it can provide you with additional hypotheses and refined hypotheses from stabilization activities.