Combinatorial Economics

The examples used in this chapter have produced tables with significant efficiency, covering hundreds of potential combinations in no more than a few dozen tests. As it turns out, these are very modest examples. Some configurations can yield reductions of more than 100:1, 1000:1, and even beyond 1,000,000:1. It all depends on how many parameters you use and how many test values you specify for each parameter. But do you always want to do less testing?

Some game features are so important that they deserve more thorough testing than others. One way to use pairwise combinatorial tests for your game is to do full combinatorial testing for critical features, and pairwise for the rest. Suppose you identify 10% of your game features as "critical" and that each of these features has an average of 100 tests associated with them (approximately a 4x4x3x2 matrix). It is reasonable to expect that the remaining 90% of the features could be tested using pairwise combinatorial tables, and only cost 20 tests per feature. The cost of full combinatorial testing of all features is 100*N, where N is the total number of features to be tested . The cost of pairwise combinatorial testing 90% of those features is 100*0.1*N + 20*0.9*N = 10*N+18*N = 28*N. This provides a 72% savings by using pairwise for the non-critical 90%.

Another way to use combinatorial tests in your overall strategy is to create some tables to use as "sanity" tests. The number of tests you run early in the project will stay low, and then you can rely on other ways of doing "traditional" or "full" testing once the game can pass the sanity tests. Knowing which combinations work properly can also help you select which scenarios to feature in pre-release videos , walkthroughs, or public demos.