Chapter 12: Cleanroom Testing
Cleanroom testing is a technique extracted from a software development practice known as Cleanroom Software Engineering. The original purpose of Cleanroom testing was to exercise software in order to make mean time to failure (MTTF) measurements over the course of the project. In this chapter, Cleanroom testing is applied to the problem of why customers find problems in games after they have been through thousands of hours of testing before being released. If one measure of a game's success is that the users (players) will not find any bugs , then the game team's test strategy should include a way to detect and remove the defects that are most likely to be found by users.
So how do users find defects? Users find defects in software by using it the way users use it. That's a little bit of a tongue twister, but it points to a testing approach that exercises the game according to the way the players are going to use it. That's what Cleanroom test development does; it produces tests that play the game the way players will play it.
Usage Probabilities
Usage probabilities, also referred to as usage frequencies, tell testers how often game functions should be used in order to realistically mimic the way customers will use the game. They can be based on actual data you might have from studies of game players or based on your own expectations about how the game will be played . Also take into account the possible evolution of a user 's play during the life of the game. A player's patterns would be different just after running the tutorial than they would be by the time the player reaches the boss on the final level. Initially, the player would utilize fundamental operations and have few if any special items unlocked. Clicking dashboard icons would occur more frequently than key commands and user-defined macros. Matches or races might take longer at the end of the game due to the higher difficulty and closer matching of the player's skill to the in-game opponent (s). Usage information can be defined and utilized in three different ways:
-
Mode-based usage
-
Player-type usage
-
Real-life usage
Mode-Based Usage
Game usage can change based on which mode the player is using such as single player, campaign, multiplayer, or online.
Single-player mode may involve one or only a few confrontations or missions. The action usually starts right away so the player is less likely to perform "build-up" operations such as building advanced units and spending money or points on expensive skill- boosting items. Some features may not be available at all to the single player, such as certain characters , weapons, or vehicles. The single player's character may also have limited race, clan, and mission options.
Campaigns tend to start the player with basic equipment and opponents and then introduce more and more complex elements as the campaign progresses. For sports games, Franchise or Season modes provide unique options and experiences that aren't available when playing a single game, such as draft picks, training camp, trading players, and negotiating salaries. RPG games will provide more powerful spells, armor , weapons, and opponents as your characters level up. Racing games may provide more powerful vehicles, add-ons, and powerups , as well as more challenging tracks.
Multiplayer gaming can take place on the same machine ‚usually for 2 ‚4 players, across two interconnected consoles or over the Internet for more massive multiplayer experiences. Headset accessories are used for team confrontations, but aren't something you're likely to use by yourself unless the game has voice commands. Text chatting also is used in multiplayer games, giving the text keyboard a workout. Game controls can also be assigned to smack talk phrases and gestures, which you will also use to taunt your inferior opponents. The time spent on a multiplayer session can be much greater than what a single player will spend , sometimes extending into the wee hours of the morning. This also brings up the fact that multiplayer games may involve players from different time zones and geographical regions , bringing together a variety of game clocks and language settings.
Player-Type Usage
Another factor that influences game usage is the classification of four multi-user player categories described by Richard A. Bartle in "Hearts, Clubs, Diamonds, Spades: Players Who Suit MUDs." He describes players by their tendencies to emphasize either Achievement, Exploration, Socializing, or Killing when they participate in multiplayer games.
The Achiever wants to complete game goals, missions, and quests. He will gain satisfaction in advancing his character's level, point, and money totals in the most efficient way possible. Achievers may replay the game at a higher level of difficulty or under difficult circumstances such as using a last-place team or going into combat armed only with a knife . They will also be interested in reaching bonus goals and completing bonus missions.
Explorers are interested in finding out what the game has to offer. They will travel around to find obscure places and the edges of the map; unmapped territory will draw their attention. The Explorer will look around and appreciate the art and special beauty in the game such as a particularly nice moonrise or light shining through a stained glass cathedral window. She is also likely to attempt interesting features, animations, combos , and physics effects. Expect the Explorer to try to open every door and check the inventory at all of the stores. The Explorer wants to figure out how things work. Think of the phrase "I wonder what would happen if ‚ ?"
The goal of the Socializer is to use the game as a means to role play and get to know other players. Chat and messaging facilities are important to him, as well as joining social groups within the game such as clans , guilds , and so on. He may host meetings or tournaments, or bring many players together in one place for announcements, trading, or even an occasional wedding . Socializers will use special game features once they find out about them from other players.
Killers enjoy getting the best of other players. They engage in player versus player and real versus realm battles . Killers know where the taunt keys are and how to customize and activate an end zone celebration . Headsets, chats, and private messages are also tools that the Killer uses to bait and humiliate his opponents.
Finally, here are some other gamer "types" to consider when you go to test your game the same way players play the game:
-
Casual gamer : Sticks mostly to functions described in the tutorial, user manual, and on-screen user interface.
-
Hard- core gamer : Uses function keys, macros, turbo buttons , and special input devices such as joysticks and steering wheels. Checks the Internet for tricks and tips. May also have game hardware juiced up to run the highest graphics resolution and frame rate.
-
Button Masher : Values speed and repetition over caution and defense. Wears out the A button on the controller to run faster, jump higher, or strike first. Will run out of ammo. The advent of the Nintendo DS may also breed the Button Masher's cousin; the Stylus Scratcher .
-
Customizer : Uses all of the game's customization features and plays the game with custom elements. Will also incorporate unlocked items, decals, jerseys, teams , and so on.
-
Exploiter : Always looking for a shortcut. Will use cheat codes, look for cracks in zone walls, and pick off opponents from a secret or unreachable spot. Creates bots to craft items, earn points, and level up. Uses infinite card combos against AI and human Collectable Card Game (CCG) opponents.
Real-Life Usage
Some games now have a built-in mechanism for capturing your preferences and making the information visible to you. This mechanism can also be used for capturing or downloading your friends ' tendencies when they play on your console, allowing you to practice against a "virtual" friend when they are away. Likewise, the tendencies of coaches and celebrity game players can be stored and delivered with the game so you can try out your own strategies against their unique styles of play. ESPN NFL 2K5 has such a feature known as the VIP Profile. Figures 12.1 and 12.2 show graphics compiled from the game that chart two players' tendencies. You can compare Haiken's tendencies on the left to coach Danny's play patterns on the right.
Figure 12.1: NFL 2K5 offensive tendencies for players Haiken and Danny.
Figure 12.2: NFL 2K5 defensive tendencies for players Haiken and Danny.
On defense there are also differences between the two players, as shown in Figure 12.2. Haiken will tap the A button to speed up the active defender about 95% of the time while Danny taps A about 67% (two thirds ) of the time on defense and holds down A to charge about 33% of the time (one third). Danny also shows about an 80% preference for Wrapping the ball carrier when tackling, where Haiken favors Wrapping only 60% of the time.
Why is it important to account for these tendencies? Testing based entirely on balanced use of the game features would not reveal defects such as a memory overflow caused by tapping the A button repeatedly on every play over the course of a maximum length game.
Even the VIP feature itself is not immune to usage bugs. In the VIP Viewer, Haiken's Passing Chart shows 999/999 for Middle Medium pass completions/attempts. The actual numbers of 1383 completions and 2278 attempts are properly shown in the Statbook view. It seems the Passing Chart doesn't account for pass-happy players running up large numbers by playing five seasons of Franchise mode with some 60-minute games thrown in.
EAN: 2147483647
Pages: 205