Section I - Black Box Testing Techniques

List

Chapter 3: Equivalence Class Testing

Chapter 4: Boundary Value Testing

Chapter 5: Decision Table Testing

Chapter 6: Pairwise Testing

Chapter 7: State-Transition Testing

Chapter 8: Domain Analysis Testing

Chapter 9: Use Case Testing

Overview

Definition

Black box testing is a strategy in which testing is based solely on the requirements and specifications. Unlike its complement, white box testing, black box testing requires no knowledge of the internal paths, structure, or implementation of the software under test (SUT).

The general black box testing process is:

• The requirements or specifications are analyzed.
• Valid inputs are chosen based on the specification to determine that the SUT processes them correctly. Invalid inputs must also be chosen to verify that the SUT detects them and handles them properly.
• Expected outputs for those inputs are determined.
• Tests are constructed with the selected inputs.
• The tests are run.
• Actual outputs are compared with the expected outputs.
• A determination is made as to the proper functioning of the SUT.

Applicability

Black box testing can be applied at all levels of system development—unit, integration, system, and acceptance.

As we move up in size from module to subsystem to system the box gets larger, with more complex inputs and more complex outputs, but the approach remains the same. Also, as we move up in size, we are forced to the black box approach; there are simply too many paths through the SUT to perform white box testing.

Disadvantages

When using black box testing, the tester can never be sure of how much of the SUT has been tested. No matter how clever or diligent the tester, some execution paths may never be exercised. For example, what is the probability a tester would select a test case to discover this "feature"?

 if (name=="Lee" && employeeNumber=="1234" &&
 employmentStatus=="RecentlyTerminatedForCause") {
 send Lee a check for $1,000,000;
 }

Key Point

When using black box testing, the tester can never be sure of how much of the system under test has been tested.

To find every defect using black box testing, the tester would have to create every possible combination of input data, both valid and invalid. This exhaustive input testing is almost always impossible. We can only choose a subset (often a very small subset) of the input combinations.

In The Art of Software Testing, Glenford Myers provides an excellent example of the futility of exhaustive testing: How would you thoroughly test a compiler? By writing every possible valid and invalid program. The problem is substantially worse for systems that must remember what has happened before (i.e., that remember their state). In those systems, not only must we test every possible input, we must test every possible sequence of every possible input.

Key Point

Even though we can't test everything, formal black box testing directs the tester to choose subsets of tests that are both efficient and effective in finding defects.

Advantages

Even though we can't test everything, formal black box testing directs the tester to choose subsets of tests that are both efficient and effective in finding defects. As such, these subsets will find more defects than a randomly created equivalent number of tests. Black box testing helps maximize the return on our testing investment.

References

Myers, Glenford J. (1979). The Art of Software Testing. John Wiley & Sons.