Defect Triggers and Test Designs

Each element in the test design represents one of the Defect Triggers. Using one or more test design technique will not by itself guarantee that all of the Defect Triggers will be sufficiently represented in your tests. It takes an explicit effort to identify appropriate triggers and incorporate them into whatever type of test designs you produce. All of the Defect Triggers should be included either in a single test design or a set of test designs related to a specific game feature, function, or capability. If you have data from previous projects, see which triggers have been effective at finding defects and include those in your designs as well as any others you can think of.

The effectiveness of each trigger can be measured in terms of defects per test. You can also think of this as the sensitivity of the game code to each trigger. A large defects/test number relative to other triggers tells you how to find bugs economically and may also hint at an underlying flaw in the game platform design or implementation. If you only have time or resources to run a given number of tests, running the tests for the most effective triggers will yield more defects than the trigger that produces the fewest defects per test (usually the Normal trigger). As you continue to create and run more tests for the most effective triggers, you will saturate them and no longer be able to find new bugs . Repeat this process to establish saturation for all of the triggers.

Combinatorial Design Trigger Examples

Let's go back to the HALO Advanced Controls combinatorial design, shown in Chapter 10 in Figure 10.26, to see if any triggers need to be added. Look Sensitivity is tested for its default, minimum, and maximum values. The minimum and maximum values could be considered Stress values since the game is supposed to respond ("process") as slowly or as quickly as it can to the movement of the joystick. The remaining parameters have values that determine whether a capability is either on or off. None of these address a particular configuration or a situation that would apply to Startup, Restart, Exception, or Stress conditions. As a result, the majority of test values represent Normal behavior. For this test to be more effective, incorporate the missing triggers as well as any other possible Stress values.

Start by identifying Configuration resources related to the Advanced Controls options. All of these parameters affect the use of the game controller. There are different types of game controllers, such as those with or without vibration and those that are wireless or wired. The controller can be inserted into any of the slots provided on the game console. It is also possible to remove the controller during the process of selecting the options, and subsequently re-insert it in the same position or a different one. These possibilities create new parameters and values to add to the combinatorial table.

The updated table is shown in Figure 15.5. Because of the added complexity introduced by the new parameters and values, the Allpairs tool was used to generate this table.

Figure 15.5: Advanced Controls combinatorial table with Configuration triggers.

As an alternative, create a separate table to cover the configuration-related parameters and value pairs. This kind of approach enables you to use the mostly "Normal" table as a sanity test and then proceed with the tables for the other triggers once the game passes the sanity tests. The Advanced Options game controller configuration table is shown in Figure 15.6.

Figure 15.6: Advanced Option game controller configuration table.

As it turns out, no matter which slot the controller is removed from, the game prompts you to "Please reconnect #1 controller" as shown in Figure 15.7. Additionally, your ability to modify the settings is restored no matter which slot you replace the controller in. Another side effect or removing the controller is that the changes you made prior to its removal are lost ‚ the settings revert to their last stored values. This might not be so bad on a screen where you only have a few values to keep track of, but if you are updating dozens of values to customize an RPG character, this loss of data can be frustrating. In contrast, Tony Hawk's Underground identifies the specific controller that was removed, requires the controller to be replaced into the same slot in order to continue, and preserves the changed option values.

Figure 15.7: HALO " game controller removed" attention notice.

The next step is to seek out Exception trigger opportunities. Since the option values are selected by scrolling, there is no opportunity to enter a "wrong" value. However, it may be possible to disrupt the selection mechanism itself. The A and B buttons are used for accepting the options or going back to the previous screen. Try holding down X, Y, the Left Trigger ("L"), or Right Trigger ("R") during the selection of the test values.

Again, one of your options is to add a column for these values plus the "None" choice into a single table, as shown in Figure 15.8. Although the table has grown again, these 28 cases represent pairwise coverage of 15,360 total possible combinations of these values!

Figure 15.8: Advanced game options with Configuration and Exception triggers.

A potential danger in doing this is that most of your test cases will result in an exception behavior that may prevent you from observing the effects of the other test values. In Figure 15.8, only six tests ‚ 1, 2, 13, 14, 25, and 26 ‚ avoid an input exception. A way around this is to create a separate table to isolate the exception effects, as shown in Figure 15.9. The "NONE" value for the Simultaneous Key parameter is not included because it is not an Exception trigger and it is already implicitly represented in the non-exception table for this feature.

Figure 15.9: Original Advanced Controls table with only Exception triggers added.

The extreme Look Sensitivity values were identified as Stress triggers, but is there any other "stressful" operation that can be done during option selection? For this particular game, both the left analog stick and the D-Pad on the game controller can be used to scroll through the options (vertically) and choices (horizontally). Operating them simultaneously could produce interesting results. Add this to the test table by defining the Scroll Control parameter with the values of LEFT STICK, D-PAD, and BOTH. Follow the same rationale as for the previous triggers when deciding whether to add these parameters and values to a single table for this screen versus creating a separate table for this trigger.

All that's left now is to identify Startup and Restart triggers for your Advanced Controls tests. These particular settings are tied to individual player profiles. This presents the opportunity to test the settings for a brand new profile versus one that has already been in use. The new profile behavior is your Startup trigger. Add this to the tests as a "Profile" parameter with NEW (Startup) or EXISTING (Normal) choices.

The advanced controls selection process can be restarted in a variety of ways: go back to the previous screen without saving, eject the game disk from the console, or turn the console off. Follow up these operations by going back into the Advanced Controls screen to check for any abnormalities. Since these settings can be stored in either internal or removable memory, another way to do a "restart" is to load information previously saved to external memory back on top of your internally saved modified values. Represent these possibilities in your table with a "Re-Enter" parameter that has a possible value of NONE for the Normal trigger and BACK, EJECT, OFF, and LOAD EXTERNAL for the Restart trigger.

TFD Trigger Examples

TFD triggers are located along the flows. The Ammo TFD template, provided in Appendix D, will be used to illustrate how to incorporate all of the Defect Triggers into a TFD. It has a few more flows than the TFD you constructed in Chapter 11, but is it "complete" in terms of triggers? Use it in a specific context ‚ Unreal Tournament 2004 for the PC ‚ and see what you can dig up.

To begin with, the template includes plenty of Normal trigger flows such as GetGun and GetAmmo when you have neither ( NoGunNoAmmo ). However, the same event can represent different triggers depending on its context with respect to the states it's leaving and entering. For example, GetAmmo when you already have the maximum amount is a case of performing a function when a resource (ammo) is at its maximum. This qualifies as a Stress trigger. Shooting a gun with no ammo falls on the other end of the spectrum where the ammo resource is at a minimum (0). Figure 15.10 shows the Ammo TFD template with these Stress triggers highlighted.

Figure 15.10: Ammo TFD template with Stress flows highlighted.

Now how about adding a Startup trigger? The TFD Enter flow jumps right to the point where the player is active in the match. In reality, there is a "pre-game" period where the player can run around the arena before hitting the "fire" button (usually the left mouse button) to initiate the match. This is relevant to the purpose of the test because a player who runs over weapons or ammo during this time should not accumulate any items as a result.

Represent this startup process on the TFD by performing "mitosis" on the NoGunNoAmmo state. That is, split it into two connected states. One state retains the original name and connections (except for the Enter flow) while the other captures the dry run and countdown behavior. Figure 15.11 shows the process of splitting this portion of the TFD.

Figure 15.11: NoGunNoAmmo state mitosis.

The new PreMatch state can be introduced to the TFD. Start by disconnecting the Enter flow from NoGunNoAmmo and attaching it to PreMatch . Then add flows to attempt GetAmmo and GetGun during the PreMatch period. These flows are Startup triggers. Figure 15.12 shows how this looks.

Figure 15.12: PreMatch state and Startup trigger flows added to Ammo TFD.

Next add the Restart trigger to the diagram. It's possible to change your status to Spectator in the middle of a match and then Join back in as a participant. Spectator mode takes your character out of the game and lets you follow players in the game while you control the camera angle. Any guns or ammo picked up prior to entering Spectator mode should be lost when you Join to return to the same match in progress. Rejoining the game from Spectator mode is done instantly without the countdown timer that you get when you start the match for the very first time. Suspending and rejoining can be done at any time during the match after the initial countdown timer has expired . Add a SpectateAndJoin flow from each of the in-game states on the TFD and tie it back to NoGunNoAmmo . Don't forget the loop flow from NoGunNoAmmo back to itself. A TFD with these updates is shown in Figure 15.13.

Figure 15.13: Ammo TFD with SpectateAndJoin Restart flows highlighted.

Note how more "pressure" is being put on the NoGunNoAmmo state with all of the flows entering and exiting. It's like a busy intersection; they tend to be much more dangerous than the ones that aren't so busy. This reflects the importance of this state to the well-being of the feature and its potential sensitivity to changes.

The TFD is getting cozy but there are a still a few more triggers to consider. For the Exception trigger, there is an operation available to use a weapon's alternate fire mode. Typically, the left mouse button is used for normal firing and the right mouse button for alternate firing. Some weapons, such as the Grenade Launcher, do not have an alternate firing mode. They should not fire nor decrement their ammo count when the user attempts alternate firing.

This is something you can use as an Exception trigger. Since this "Unsupported AltFire" operation will not change the ammo status of the weapon, add it as a loop on the TFD states where there is both a gun and ammo. Your result should look like Figure 15.14.

Figure 15.14: Ammo TFD with AltFire Exception flows highlighted.

Finally, the Configuration must be included. One of the weapon settings in the game allows the player to select an older style rendering of certain weapons. While this is appealing to players who owned earlier titles in this series, it also creates an additional test responsibility. You can check that changing the weapon rendering while the game is in progress does not affect the amount of ammo loaded in that weapon or produce unwanted artifacts such as unexpected audio or "shadow" (duplicate) weapons. Add a ToggleWeaponSkin flow at all of the states where the player has the weapon. Since this should not affect the ammo, these flows will loop back into the states from which they originated. Figure 15.15 shows the TFD with these Configuration flows.

Figure 15.15: Ammo TFD with weapon skin Configuration flows.

Now the TFD is getting really crowded! Just remember that the same option that was presented for combinatorial tests is applicable to TFDs, test tree designs, or any other formal or informal means you use to come up with test cases. You can incorporate the triggers into a single design or create companion test designs that work together as a suite of tests to provide the trigger coverage you need.

You may also find it useful to document the intended triggers for each test element. One easy way is to provide a letter code in parentheses after the event name on each TFD flow, parameter value for combinatorial tests, or branch node for test tree designs. You can count the number of times each letter appears to see how many times each trigger is being used. It also helps you classify defects you find when running the tests. Just be aware that this carries a maintenance burden to re-evaluate the trigger designation whenever you move or add new test elements to the design.