8.5 Categorize the problem

8.5.1 Correctness

Here is a list of questions useful in categorizing a problem with the correctness of the output of a program:

Is there any output at all?
Is any output missing (deletion)?
Is there extra output (insertion)?
Are the individual output values correct (substitution)?
Are the individual output values in the correct order (transposition)?
Are the individual output values close to correct, but not acceptable?

What hypotheses do these questions suggest? Are complete elements missing, or are individual elements (numbers, strings) only partially displayed? Are numerical values missing because they have been incorrectly aggregated with others? Is an entire sequence of values missing from either the beginning or the end of the output? Are individual random items missing, or are groups of items missing?

Are the extra values due to repetition of correct values? Is an entire sequence of values inserted at either the beginning or the end of the output? Are individual random items inserted, or are groups of adjacent items inserted?

Is the same value incorrectly substituted in many places, or are there different values substituted in each erroneous location? Are the values substituted a progression of related values?

Are the transposed values adjacent to their correct position, or are they some distance from where they belong? Is the distance from the correct position a constant for all values, or is it random? Are individual random items out of position, or are groups of adjacent items out of position, but in position with respect to each other?

Is the difference between the actual and expected values in the least significant digits? Are the actual values off from the expected values by a random or a constant amount? Are the actual values off from the expected values by a constant factor or a progression of factors?

Identifying one of these situations doesn’t guarantee that it’s the cause of correctness problems. If you find one, however, it’s a good candidate for an initial hypothesis.

8.5.2 Completion

Here is a list of questions useful in categorizing a problem with the completion of a program:

Did the program appear to be in an infinite loop?
Did the program stop prematurely?
Did the program shut itself down, or did it just abort?
Which procedure did the program abort in?
Which source statement did the program abort in?
Which machine instruction caused the program to abort?
Which type of exception was taken?
What was the program return code?

What hypotheses do these questions suggest? How do you know that the program is in an infinite loop? Did it stop producing output, or did it produce the same output many, many times? Could it just be running very slowly?

While the symptom of the defect may be showing up in a particular loop running without stopping, you must be careful not to assume that the problem must be in the loop itself. Any statement executed prior to the loop might be the actual cause. In the same manner, for all the questions regarding the place where the program stopped, the initial hypothesis must be that some previously executed statement caused the problem.

Identifying one of these situations doesn’t guarantee that it’s the cause of completion problems. If you find one, however, it’s a good candidate for an initial hypothesis.

8.5.3 Robustness

Here is a list of questions useful in categorizing a problem with the robustness of a program:

Did the program reject valid inputs?
Did the program give incorrect warnings about valid inputs?
Did the program accept invalid inputs?
Did the program fail to give warnings about invalid inputs?

What hypotheses do these questions suggest? Obviously, if a program isn’t checking its input correctly, a logical hypothesis is that the input checking code is at fault.

Look at the Boolean logic inside each of the input checks. Are the senses of the comparison operators correct? Do they include the proper boundary conditions? Are the proper Boolean operators used to connect the comparisons?

Look at the values being used to check the input. Are they arbitrary choices based on the implementation, or do they reflect the application context? Are reasonable tolerances used when testing floating-point values?

Identifying one of these situations doesn’t guarantee that it’s the cause of a robustness problem. If you find one, however, it’s a good candidate for an initial hypothesis.

8.5.4 Efficiency

Here is a list of questions useful in categorizing a problem with the efficiency of a program:

Does the program take too long to process the inputs?
Does the execution time grow linearly with the size of the input set?
Are there inputs sets of different sizes, which cause the program to run too long?

What hypotheses do these questions suggest? If the execution time of a program doesn’t grow linearly with the size of the input set, then the logical place to look for a problem is in those parts of the program that are most likely to have time complexity worse than linear time.

Look at all the places where data is being sorted in the program. Do any of the sorting algorithms have an average complexity worse than O(n log n)? (This is the complexity of the most commonly used high-performance sorting algorithms.) Are sorting algorithms being used that have a worst-case complexity different from the average case?

Look at all the places where data is being searched in the program. Do any of the searching algorithms have an average complexity worse than O(n)?

Look at all nested loops, even if they aren’t involved in sorting or searching. Is the number of iterations of each loop in the nest proportional to the size of the input?

Look at the input sets that cause unacceptable performance. Is the data ordered randomly? Is the data ordered differently than it’s processed? Are there aggregate structures or elemental data items that only occur in the input sets with unacceptable performance? Are there aggregate structures or elemental data items that will cause excessive memory allocation?

Identifying one of these situations doesn’t guarantee that it’s the cause of unacceptable performance. If you find one, however, it’s a good candidate for an initial hypothesis.

These questions aren’t an exhaustive list of ways you can categorize a software defect. They do illustrate the kind of questions you should be asking to generate relevant hypotheses.