Chapter 10: Combinatorial Testing

Like Goldilocks, testers and project managers are continually struggling with the issue of how much testing is too little, too much, or just right. Game quality has to be good enough for consumers but testing can't go on forever if the game is going to hit its release date. Trying to test every possible combination of game event, configuration, function, and options is neither practical nor economical under these circumstances. Taking shortcuts or skipping some testing is risky business.

Pairwise combinatorial testing is a way to find defects and gain confidence in the game software while keeping the test sets small relative to the amount of functionality they cover. "Pairwise" combination means that each value you use for testing needs to be combined at least once with each other value of the remaining parameters.

Parameters

Parameters are the individual elements of the game that you want to include in your combinatorial tests. You can find test parameters by looking at various types of game elements, functions, and choices such as:

• Game events

• Game settings

• Gameplay options

• Hardware configurations

• Character attributes

• Customization choices

The test you create can be homogenous ‚ designed to test combinations of parameters of the same type, or heterogeneous ‚ designed to test more than one type of parameter in the same table.

For example, testing choices from a Game Options screen for their effect on gameplay is done with a homogenous combinatorial table. If you go through various menus to select different characters , equipment, and options to use for a particular mission, then that results in a heterogeneous table.