Constructing Tables | Game Testing All in One (Game Development Series)

To see how a combinatorial table is constructed , start with a simple table using parameters that have only two possible values. Games are full of these kinds of parameters, providing choices such as On or Off, Male or Female, Mario or Luigi, or Night or Day. This test combines character attributes for a Jedi character in a Star Wars game to test their effects on combat animations and damage calculations. The three test parameters are character Gender (Male or Female), whether the character uses a one-handed (1H) or two-handed (2H) Light Saber, and whether the character follows the Light side or the Dark side of the Force.

The table starts with the first two parameters arranged in the first two columns so that they cover all four possible combinations, as shown in Figure 10.1.

Figure 10.1: First two columns of Jedi combat test.

To construct a full combinatorial table, repeat each of the Gender and Light Saber pairs, and then combine each with the two possible Force values. When the Light and Dark "Force" choices are added in this way, the size of the table ‚ determined by the number of rows ‚ doubles, as shown in Figure 10.2.

Figure 10.2: Complete three-way combinatorial table for Jedi combat test.

For a pairwise combinatorial table, it's only necessary to combine each value of every parameter with each value of every other parameter at least once somewhere in the table. A pair that is represented in the table is said to be "satisfied," while a pair not represented in the table is "unsatisfied." The following twelve pairings must be satisfied for the Jedi combat table:

Male Gender paired with each Light Saber choice (1H, 2H)
Female Gender paired with each Light Saber choice (1H, 2H)
Male Gender paired with each Force choice (Light, Dark)
Female Gender paired with each Force choice (Light, Dark)
One-Handed (1H) Light Saber paired with each Force choice (Light, Dark)
Two-Handed (2H) Light Saber paired with each Force choice (Light, Dark)

To make the pairwise version, rebuild from the table in Figure 10.1 by adding a column for the Force values. Next , enter the Light and Dark choices for the Male character, as shown in Figure 10.3.

Figure 10.3: Adding Force choices for the Male rows.

Adding the "Dark" value to the first Female row will complete the pairwise criteria for the "1H" Light Saber value, as illustrated in Figure 10.4.

Figure 10.4: Adding the first Force choice for the Female character tests.

Finally, fill in the Light value in the second Female row to get the table in Figure 10.5 that satisfies the pairwise criteria for all parameters. In this case, it ends up being the same size as the two-parameter table. Including the Force parameter in these tests is free in terms of the resulting number of test cases. In many cases, pairwise combinatorial tables let you add complexity and coverage without increasing the number of tests you will need to run. This will not always be true; sometimes you will need a few more tests as you continue to add parameters to the table. However, the growth of the pairwise table will be much slower than full combinatorial tables for the same set of parameters and their values.

Figure 10.5: Completed pairwise combinatorial table for three Jedi combat parameters.

In this simple example, the pairwise technique has cut the number of required tests in half as compared to creating every mathematically possible combination of all of the parameters of interest. This technique and its benefits are not limited to tables with two-value parameters. Parameters with three or more choices can be efficiently combined with other parameters of any dimension. When it makes sense, incorporate more parameters to make your tables more efficient.

The number of choices (values) tested for a given parameter is referred to as its dimension . Tables are characterized by the dimensions of each parameter. They can be written in descending order, with a superscript to indicate the number of parameters of each dimension. In this way, the Jedi combat table completed in Figure 10.5 is described as a 2 ³ table. A table with one parameter of three values, two parameters of four values, and three parameters of two values is described as a 4 ² 3 ¹ 2 ³ . Another way to describe the characteristics of the table is to list the parameter dimensions individually in descending order with a dash in between each value. Using this notation, the Jedi combat table is a 2-2-2 table, and the second example mentioned above is described as 4-4-3-2-2-2. You can see how the second notation takes up a lot of space when there are a lot of parameters. Use whichever one works best for you.

Create pairwise tables of any size for your game tests using the following short and simple process. These steps may not always produce the optimum (smallest possible) size table, but you will still get an efficient table.

Choose the parameter with the highest dimension.
Create the first column by listing each test value for the first parameter N times, where N is the dimension of the next-highest dimension parameter.
Start populating the next column by listing the test values for the next parameter.
For each remaining row in the table, enter the parameter value in the new column that provides the greatest number of new pairs with respect to all of the preceding parameters entered in the table. If no such value can be found, alter one of the values previously entered for this column and resume this step.
If there are unsatisfied pairs in the table, create new rows and fill in the values necessary to create one of the required pairs. If all pairs are satisfied, then go back to step 3.
Add more unsatisfied pairs using empty spots in the table to create the most new pairs. Go back to step 5.
Fill in empty cells with any one of the values for the corresponding column (parameter).

The next example is a little more complicated than the previous one. Use the preceding process to complete a pairwise combinatorial table for some of the NFL 2K5 Game Options, as shown in Figure 10.6.

Figure 10.6: ESPN NFL 2K5Game Options screen.

Quarter Length is selectable from 1 to 15 minutes, with 5 as the default. You will use 1, 5, and 15 in your table. Play Calling choices are By Package, By Formation, and Let Coach Pick. Game Speed choices are Slow, Normal, and Fast. Three other options ‚ Challenges, Coach Mode, and Multiple Hot Routes ‚ all have the choice of being either On or Off. The Performance EQ option is not included in this example. As a result, you will create a 3 ³ 2 ³ table consisting of three parameters with three choices: Quarter Length, Play Calling, Game Speed, and three parameters with two choices: Challenges, Coach Mode, and Multiple Hot Routes.

If you aren't familiar with the game or the detailed rules of football, that doesn't matter right now. You just need to be able to understand and follow the seven steps previously listed.

Begin the process with steps 1 and 2 and list the Quarter Length values three times in the first column of the table. This is because Quarter Length is one of the parameters with the highest dimension (3). One of the parameters with the next highest dimension is Play Calling, which also has a dimension of three.

Next, apply step 3 and put each of the three Play Calling values in the first three rows of column 2. Figure 10.7 shows what the table looks like at this point.

Figure 10.7: Starting the NFL 2K5 Game Options test table.

Apply step 4 to continue filling in the next column. Starting with the fourth row, enter a Play Calling parameter that creates the most number of new pairs. Because this is only the second column, you can only create one new pair. "Package" Play Calling has already been paired with "1 min" Quarter Length, so you can put "Formation" in row 4 to create a new pair. Likewise, "Coach" and "Package" can go in rows 5 and 6 to create new pairs with "5 min" and "15 min," respectively. Figure 10.8 shows the resulting table at this point in the process.

Figure 10.8: Adding the second set of Play Calling values.

Continue with step 4 to complete the Play Calling column. At the seventh row, enter a Play Calling parameter that creates a new pair with the "1 min" Quarter Length value. "Package" (row 1) and "Formation" (row 4) have already been paired, so "Coach" is the correct value for this row. By the same process, "Package" goes in row 8 and "Formation" in row 9. Figure 10.9 shows the first two columns completed in this manner.

Figure 10.9: Completing the Play Calling column.

Applying step 5, check that all of the pairs required for the first three columns are satisfied:

Quarter Length = "1 min" is paired with "Package" (row 1),"Formation" (row 4), and "Coach" (row 7).

Quarter Length = "5 min" is paired with "Package" (row 8),"Formation" (row 2), and "Coach" (row 5).

Quarter Length = "15 min" is paired with "Package" (row 6),"Formation" (row 9), and "Coach" (row 3).

Since all of the pairs required for the first two columns are represented in the table, step 5 sends us back to step 3 to continue the process with the Game Speed option and its three test values. Applying step 3, list the "Slow," "Normal," and "Fast" Game Speed values at the top of the third column, as shown in Figure 10.10.

Figure 10.10: Starting the Game Speed column.

Proceed with step 4 to add the Game Speed value that creates the most pairs for row 4 ("1 min" and "Formation"). "Slow" is already paired with "1 min" in row 1 and "Normal" is already paired with "Formation" in row 2, so "Fast" is the correct entry for this row. In the same manner, "Slow" creates two new pairs in row 5, and "Normal" creates two new pairs in row 6. Figure 10.11 shows what the test table looks like at this point.

Figure 10.11: Adding the second set of Game Speed values.

Again, continue with step 4 to complete the Game Speed column. "Normal" produces two new pairs in row 7, "Fast" in row 8, and "Slow in row 9. Figure 10.12 shows the completed Game Speed column.

Figure 10.12: Completing the Game Speed column.

It's time again to check that all the required pairs are satisfied. Because the first two columns have been previously verified , there's no need to check them again. Check the new Game Speed column against all of its predecessors, as follows:

Quarter Length = "1 min" is paired with "Slow" (row 1), "Normal" (row 7), and "Fast" (row 4).
Quarter Length = "5 min" is paired with "Slow" (row 5), "Normal" (row 2), and "Fast" (row 8).
Quarter Length = "15 min" is paired with "Slow" (row 9),"Normal" (row 6), and "Fast" (row 3).
Play Calling = "Package" is paired with "Slow" (row 1), "Normal" (row 6), and "Fast" (row 8).
Play Calling = "Formation" is paired with "Slow" (row 9),"Normal" (row 2), and "Fast" (row 4).
Play Calling = "Coach" is paired with "Slow" (row 5), "Normal" (row 7), and "Fast" (row 3).

With all of the required pairs satisfied at this point, step 5 sends you back to step 3 to add the Challenges parameter. Figure 10.13 shows the two Challenges test values added to the top of the fourth column.

Figure 10.13: Starting the Challenges column.

Apply step 4 and add the Challenges value in row 3 ("15 min", "Coach," and "Fast") that creates the most pairs with column 4. Either "Yes" or "No" will create a new pair with all three of the other values in this row. For this exercise, choose "Yes" for row 3. Continue from there and add the right values for rows 4 through 6. A "Yes" in row 4 would only create one new pair with "Formation" so "No" is the right value to put here, creating pairs with "1 min" and "Fast." Rows 5 and 6 are populated with "No" to create two new pairs in each of these rows as well. Figure 10.14 shows the table with "Yes" chosen for row 3, and the subsequent values for rows 4, 5, and 6.

Figure 10.14: Adding the second set of Challenges values.

Now choose the right values for the remaining rows. A "No" in row 7 does not create any new pairs, since "1 min" is already paired with "No" in row 4 and "Coach" is already paired with "No" in row 6. "Yes" in this row does create a new pair with "Normal," so it is the only correct choice. Rows 8 and 9 must also be populated with "Yes" to create new pairs with "5 min" and "Formation," respectively. Figure 10.15 shows the completed Challenges column.

Figure 10.15: Completing the Challenges column.

Now check that all the required pairs for the Challenges column are satisfied:

Quarter Length = "1 min" is paired with "Yes"(rows 1,7) and "No"(row 4).

Quarter Length = "5 min" is paired with "Yes"(row 8) and "No"(rows 2, 5).

Quarter Length = "15 min" is paired with "Yes"(rows 3,9) and "No"(row 6).

Play Calling = "Package" is paired with "Yes" (rows 1, 8) and "No" (row 6).

Play Calling = "Formation" is paired with "Yes"(row 9) and "No"(rows 2,4).

Play Calling = "Coach" is paired with "Yes" (rows 3, 7) and "No" (row 5).

Game Speed = "Slow" is paired with "Yes" (rows 1, 9) and "No" (row 5).

Game Speed = "Normal" is paired with "Yes" (row 7) and "No" (rows 2, 6).

Game Speed = "Fast" is paired with "Yes" (rows 3, 8) and "No" (row 4).

So far, so good! Having satisfied all of the pairs required by the Challenges column, go back again to step 3 to continue with the Coach Mode option. Add the "Yes" and "No" Coach Mode values to the top of the fifth column, as shown in Figure 10.16.

Figure 10.16: Starting the Coach Mode column.

Step 4 requires the value in column 5 that creates the most for row 3. Only "No" creates a new pair with the four other values in this row ("15 min," "Coach," "Fast," and "Yes"). Repeat for row 4 and choose "Yes," which creates three new pairs ("Formation," "Fast," and "No"), while "No" only provides one new pair with "1 min." Populating rows 5 and 6 with "Yes" creates two new pairs in each of these rows, while "No" would only add one new pair in each case. Figure 10.17 shows the table with "No" chosen for Coach Mode in row 3, and "Yes" for rows 4, 5, and 6.

Figure 10.17: Adding to the Coach Mode column.

A "No" in the remaining rows produces a new pair for each: "1 min," "Package," and "Slow." Figure 10.18 shows the completed Coach Mode column.

Figure 10.18: Completing the Coach Mode column.

It's time again to check that all the required pairs for the new column are satisfied:

Quarter Length = "1 min" is paired with "Yes" (rows 1, 4) and "No" (row 7).
Quarter Length = "5 min" is paired with "Yes" (row 5) and "No" (rows 2, 8).
Quarter Length = "15 min" is paired with "Yes" (row 6) and "No" (rows 3, 9).
Play Calling = "Package" is paired with "Yes" (rows 1, 6) and "No" (row 8).
Play Calling = "Formation" is paired with "Yes" (row 4) and "No" (rows 2, 9).
Play Calling = "Coach" is paired with "Yes" (row 5) and "No" (rows 3, 7).
Game Speed = "Slow" is paired with "Yes" (rows 1, 5) and "No" (row 9).
Game Speed = "Normal" is paired with "Yes" (row 6) and "No" (rows 2, 7).
Game Speed = "Fast" is paired with "Yes" (row 4) and "No" (rows 3, 8).
Challenges = "Yes" is paired with "Yes" (row 1) and "No" (rows 3, 7, 8, 9).
Challenges = "No" is paired with "Yes" (rows 4, 5, 6) and "No" (row 2).

This confirms that the pairs required for the Coach Mode column are all satisfied. The process sends you back to step 3 to pair the Multiple Hot Routes values, which will be placed in the table's" Multiple Routes" column. Add the "Yes" and "No" values to the top of the this column, as shown in Figure 10.19.

Figure 10.19: Starting the Multiple Routes column.

As you proceed from here, something new happens. Either of the Multiple Routes values added to row 3 creates four new pairs, so neither value can be selected. A "Yes" creates new pairs with "15 min," "Coach," "Fast," and "No" (Coach Mode), while a "No" creates new pairs with "15 min," "Coach," "Fast," and "Yes" (Challenges). As you go through the table, you will find that no preferred value can be found for any of the remaining rows. Don't trust me on this (remember Rule 2?) ‚ check for yourself! According to step 4, one of the Multiple Routes values in the first two rows should be changed. Figure 10.20 shows the updated table with the second Multiple Routes value changed to "Yes."

Figure 10.20: Restarting the Multiple Routes column.

Continue to step 4 from this point and see that there are now clear choices for the remaining rows. A "No" in row 3 provides new pairs with all of the first five columns, versus only four new pairs that would be provided by a "Yes." Another "No" in row 4 provides four new pairs versus three from using "No," and rows 5 and 6 get two new pairs from a "No" versus only one from a "Yes." Figure 10.21 shows how the table looks with these values filled in.

Figure 10.21: Adding to the Multiple Routes column.

Complete the final three values for the table in the same manner. "Yes" is the only value that produces a new pair in each of these rows. The completed table is shown in Figure 10.22.

Figure 10.22: The completed Game Options test table.

Now, for perhaps the last time, check that all the required pairs for the Multiple Routes column are satisfied:

Quarter Length = "1 min" is paired with "Yes" (rows 1, 7) and "No" (row 4).
Quarter Length = "5 min" is paired with "Yes" (row 2) and "No" (rows 5, 8).
Quarter Length = "15 min" is paired with "Yes" (row 9) and "No" (rows 3, 6).
Play Calling = "Package" is paired with "Yes" (rows 1, 8) and "No" (row 6).
Play Calling = "Formation" is paired with "Yes" (rows 2, 9) and "No" (row 4).
Play Calling = "Coach" is paired with "Yes" (row 7) and "No" (rows 3, 5).
Game Speed = "Slow" is paired with "Yes" (rows 1, 9) and "No" (row 5).
Game Speed = "Normal" is paired with "Yes" (rows 2, 7) and "No" (row 6).
Game Speed = "Fast" is paired with "Yes" (row 8) and "No" (rows 3, 4).
Challenges = "Yes" is paired with "Yes" (rows 1, 7, 8, 9) and "No" (row 3).
Challenges = "No" is paired with "Yes" (row 2) and "No" (rows 4, 5, 6).
Coach Mode = "Yes" is paired with "Yes" (row 1) and "No" (rows 4, 5, 6).
Coach Mode = "No" is paired with "Yes" (rows 2, 7, 8, 9) and "No" (row 3).

Well done! By creating a pairwise combinatorial table, you developed nine tests that can test this set of parameters and values comprising 216 possible mathematical combinations (3*3*3*2*2*2). It was certainly worth the effort to create the table in order to save 207 test cases! Also note that for this table you didn't have to resort to steps 6 and 7. That won't be true in every case, so don't rule it out for the future.

Now you are ready to test the game using the combinations in the table and check for any irregularities or discrepancies with what you expect to happen. Create test tables as early as possible, for example by using information provided in the design document way before any working code is produced. Check any available documentation to see if there is a clear definition of what should happen for each of your combinations. That will equip you to raise questions about the game that may not have been considered . This is an easy way to prevent bugs and improve gameplay.

A second approach is to ask people involved with code or requirements "What happens if ‚ " and read your combinations. You may be surprised how many times you will get an answer like "I don't know" or "I'll have to check and get back with you." This is a much more economic alternative to finding surprises late in the project. It is also much more likely that your issues will be fixed, or at least considered, by the time the code is written.

Don't just check for immediate or near- term effects of your combinatorial tests. It's important to make sure that a menu selection is available or a button performs its function when pressed, but mid-term and far-term effects can lock up or spoil the game down the road. Here are some of these effects to consider:

q ‚ ‚	Does my game or session end properly?
q ‚ ‚	Do achievements get recorded properly?
q ‚ ‚	Can I progress to appropriate parts of the game or story?
q ‚ ‚	Did actions taken in the game get properly counted toward season /career accomplishments and records?
q ‚ ‚	Can I properly start and play a new session?
q ‚ ‚	Can I store and retrieve sessions or files?

Here are some of the observations you might make from running the NFL 2K5 Game Options tests you completed in Figure 10.22:

Playing in Coach Mode prevents the user from selecting or controlling any of the players during the game. With that mode on, what is the function of the Multiple Hot Routes choice, since the player is prevented from assigning any Hot Routes? Maybe this relationship could be made clear to the user by "graying out" the Multiple Hot Routes choices if Coach Mode is set to "Yes."
Normally, the A button (on Xbox) is used to change player selection and skip over in-game events such as timeouts. In Coach Mode, some of the A button actions, like active player selection, are blocked. As a side effect, Coach Mode prevents the user from interrupting timeouts. You end up waiting approximately 90 seconds for the timeout to expire before gameplay resumes.
According to NFL rules and the game's own documentation, coaches should not be allowed to challenge plays in the last two minutes of the half. If this is literally applied, then no coaches' challenges should be possible at all in a game where each quarter is only one minute long. However, NFL 2K5 does permit coaches to challenge plays during the 1st and 3rd quarters of the game when Quarter Length = 1 minute.
One of the features of NFL 2K5 is to provide Crib Points for various accomplishments during the game and one of the rewards is 10 points for every hour of gameplay. When playing with Quarter Length = 15 minutes, the actual gameplay time (four quarters) takes more than 1 hour . During the game you will get pop-up notifications of a 10-point crib reward. This can happen two or three times depending on the pace you play. Once the game ends, the summary of Crib Points earned during that game shows that the 10-point game time award was only counted once.