Combinatorial Tools

At some point, you will find it difficult to construct and verify large parameter and value counts. Fortunately, James Bach has made a tool available to the public at www.satisfice.com/tools.shtml that handles this for you. It is also provided on the CD-ROM that comes with this book. The Allpairs tool uses a tab-delimited text file as input and produces an output file that includes a pairwise combinatorial table as well as a report on how many times each pair was satisfied in the table.

To use Allpairs, start by creating a file that contains tab-delimited columns of parameter names with the test values in the following table. Here is an example based on match settings from the fighting game Dead or Alive 3 (DOA3):

Difficulty	MatchPoint	LifeGauge	RoundTime
Normal	1	Smallest	NoLimit
Easy	2	Small	30
Hard	3	Normal	40
VeryHard	4	Large	50
‚	5	Largest	60
‚	‚	‚	99

Remember, this is not an attempt at a combinatorial table ‚ the tool will provide that. This is a description of the parameters you want to test: Difficulty level, Match Points needed to win a match, Life Gauge size displayed on the screen, and Round Time in seconds. Even though there are only four parameters, the fact that they have 4, 5, 5, and 6 values to each to test would make this difficult to construct and validate by hand. That also means there are 600 (4*5*5*6) values if you try to test all four-wise combinations. You should expect a much smaller test set from a pairwise combinatorial test of these options ‚ somewhere in the 30 ‚ 40 range ‚ based on the dimensions of the two largest parameters (6*5).

Now open a DOS (yes ‚ DOS!) window and enter allpairs input.txt > output.txt where input.txt is the name of your tab-delimited parameter list file, and output.txt is the name of the file where you want to store the generated combinatorial table. Make sure you are in the directory where the files are located, or provide the full path .

For this DOA3 table, the command might be allpairs doaparams.txt > doapairs.txt . Here's what the test case portion of the output looks like:

 

TEST CASES

Case	Difficulty	MatchPoint	LifeGauge	RoundTime	Pairings
1	Normal	1	Smallest	NoLimit	6
2	Easy	2	Small	NoLimit	6
3	Hard	3	Normal	NoLimit	6
4	VeryHard	4	Large	NoLimit	6
5	Hard	1	Small	30	6
6	VeryHard	2	Smallest	30	6
7	Normal	3	Large	30	6
8	Easy	4	Normal	30	6
9	VeryHard	1	Normal	40	6
10	Hard	2	Large	40	6
11	Easy	3	Smallest	40	6
12	Normal	4	Small	40	6
13	Easy	1	Large	50	6
14	Normal	2	Normal	50	6
15	VeryHard	3	Small	50	6
16	Hard	4	Smallest	50	6
17	Normal	5	Largest	60	6
18	Easy	1	Largest	60	4
19	Hard	2	Largest	60	4
20	VeryHard	3	Largest	60	4
21	Easy	5	Smallest	99	5
22	Normal	4	Largest	99	4
23	Hard	5	Small	99	4
24	VeryHard	5	Normal	99	4
25	~Normal	5	Large	NoLimit	2
26	~Easy	5	Largest	30	2
27	~Hard	5	Largest	40	2
28	~VeryHard	5	Largest	50	2
29	~Hard	4	Smallest	60	2
30	~Hard	1	Large	99	2
31	~VeryHard	~1	Largest	NoLimit	1
32	~Normal	~1	Small	60	1
33	~Easy	~2	Normal	60	1
34	~Easy	~3	Large	60	1
35	~Normal	2	~Smallest	99	1
36	~Easy	3	~Small	99	1

Aren't you glad you didn't have to do that by hand! The "case" and "pairings" columns are added to the output by the Allpairs tool. "Case" is a sequential number uniquely identifying each test case. The "pairings" number indicates how many necessary parameter pairs are represented by the set of values in each row. For example, the "pairings" value in row 18 is 4. You can check for yourself that row 18 produces four new pairs: Easy-Largest, Easy-60, 1-Largest, and 1-60. The Largest-60 pair was satisfied earlier in the table at row 17, and the Easy-1 pair first appears in row 13.

Values that begin with the "~" symbol are wildcards. That is, any value of that parameter could be placed there without removing one of the necessary pairings to complete the table. The tool arbitrarily chooses but you, the knowledgeable tester, can replace those with more common or notorious values, such as defaults or values that have caused defects in the past.

The output from Allpairs also produces a Pairing Details list, which is an exhaustive list of each necessary pair and all of the rows that include that pair. One of the pairings listed for the DOA3 table is

MatchPoint

Difficulty

1

Easy

2

13, 18

which means that the pair Match Point = 1 and Difficulty = Easy occurs 2 times ‚ in rows 13 and 18 of the table.

In the same list, the entry

RoundTime

LifeGauge

60

Largest

4

17, 18, 19, 20

traces the RoundTime = 60 and LifeGauge = Largest pair to rows 17 ‚ 20 of the combinatorial table. This kind of information is especially useful if you want to limit your testing to all the instances of a particular pair. One reason for doing that would be to limit verification testing of a release that fixed a bug caused by one specific pair.

Another use for the Pairing Details information is to quickly narrow down the possible cause of a new defect by immediately testing the other entries in the table that had the same pairs as the test that just failed. For example, if the test in row 13 fails, search the Pairing Details list for other pairs that were included in row 13. Then run the tests on any rows listed in addition to row 13. Here are the pairs that are satisfied by row 13:

RoundTime	MatchPoint	50	1	1	13
RoundTime	LifeGauge	50	Large	1	13
RoundTime	Difficulty	50	Easy	1	13
MatchPoint	LifeGauge	1	Large	2	13, 30
MatchPoint	Difficulty	1	Easy	2	13, 18
LifeGauge	Difficulty	Large	Easy	2	13, 34

From this information, tests 18, 30, and 34 could be run next to help identify the pair that causes the defect. If none of those tests fail, then the cause is narrowed down to the first three pairs, which are only found in row 13: 50-1, 50-Large, or 50-Easy. If test 18 fails, then look for the 1-Easy pair to be the cause of the problem. Likewise, if test 30 fails then suspect the 1-Large combination. If test 34 fails you can suggest Large-Easy as the cause of the problem in your defect report.

The Allpairs output file is tab delimited so you can paste it right into Microsoft Excel or any other program supporting that format. The Allpairs tool files and the examples from this chapter, including the complete output file, are provided on the book's CD-ROM.