Automated Testing

When I described functional test checklists I mentioned that it was unreasonable to expect a tester to be physically and mentally capable of a 100% test of every game feature and permutations of those features in combination. Most games are extremely complicated and tend to be peppered with random events to make the game experience interesting. Wise developers and test teams create their game with special instrumentation, test tools, and event playback and recording systems to help the test team run through an immense test suite in a fraction of the time.

An excellent example of this is the problem of verifying the payout and animations of a video poker machine. One card combination in particular has about a 1:40,000 chance of appearing (the royal flush in five-card draw). This isn't even an extreme case—some slot machines have a very small chance, 1:500,000, of hitting the jackpot. This situation is far from unique because almost every game is based on the outcome of combinations of random events and the input of a player, which is essentially random.

The first set of test tools we created to help the Microsoft test team were simple dialog boxes to preset the outcome of any roll of the dice or shuffle of the deck. The testers would use the tools (or cheats, as they are called if they aren't removed from the release build), to be able to run through their functional tests without relying on random events. To be honest this made testing these games possible, not just convenient. The test team wanted more in the next version, and after talking with them we understood that what they were asking for was an interface to our game systems that basically ran the functional tests automatically.

We created a playback/record system in our game based on user input events, and the testers at Microsoft wrote a Visual Basic tool to run the recorded scripts in batches and compare the output to known good builds. Take a look at a small piece of the test script for blackjack:

 // // Automation RECORD file // // This script file was generated via // the automated record/playback feature // // Product:        Casino 2.0 // Build:          02.02.07.0052 // Filename:       d:\automation tool\scripts\bj.dbg.txt // Date:           04/11/2003 // Time:           10:25:22 // // All comments are proceeded by '//' // Commands end with ';' // 0000 : RANDOM_SEED("1018545922"); 0000 : SET_START_BALANCE("999900.00"); 0000 : LEAN("1"); 0000 : ADD_SCREEN("SCREEN_BLACKJACK"); 0000 : RANDOM_SEED("1018545930"); [TCBegin] Verify changing table limits working correctly [ExReS] [GAME] Table Limits Set: min=100.00 max=5000.00 0023 : MOUSE_MOVE( "0","368","278" ); 0036 : MOUSE_MOVE( "0","106","73" ); 0009 : MOUSE_LBUTTON_DOWN( "1","106","73" ); 0002 : MOUSE_LBUTTON_UP( "0","106","73" ); 0000 : MOUSE_MOVE( "0" , "106" , "73" ); 0039 : MOUSE_MOVE( "0","170","142" ); 0015 : KEYBOARD_KEY_DOWN( "88","1","45","X" ); 0004 : KEYBOARD_KEY_UP( "88","1","49197","X" ); 0023 : KEYBOARD_KEY_DOWN( "88","1","45","X" ); 0006 : KEYBOARD_KEY_UP( "88","1", "49197", "X" ); [TCEnd] 0023 : KEYBOARD_SYSKEY_DOWN("18","1","8248"); 0005 : MOUSE_MOVE("0","170","142"); 0005 : KEYBOARD_SYSKEY_UP( "9","1","40975" ); 0006 : KEYBOARD_KEY_UP( "18","1","49208","VK_MENU" ); 0000 : MOUSE_MOVE( "0","482","312" ); 0000 : KEYBOARD_KEY_UP("9","1","49167","VK_TAB"); 0003 : MOUSE_MOVE("0","482","312"); 0018 : KEYBOARD_KEY_DOWN("38","1","328","VK_UP"); 0002 : KEYBOARD_KEY_UP("38","1","49480","VK_UP"); 0014 : MOUSE_MOVE("0","482","313"); 0002 : MOUSE_MOVE("0","482","313"); 0006 : KEYBOARD_KEY_DOWN("69","1","18","E"); 0000 : SET_CARDS("11","1","0","1","0","4","2","1","1","11","0","1","0",                  "1","0","13","3","1","0","8","0","10","1"); 0001 : MOUSE_MOVE("0","590","413"); 0006 : MOUSE_MOVE("0","590","413"); 0016 : KEYBOARD_KEY_DOWN("90","1","44","Z"); 0006 : KEYBOARD_KEY_UP("90","1","49196","Z"); 0010 : KEYBOARD_KEY_DOWN("90","1","44","Z"); 0007 : KEYBOARD_KEY_UP("90","1","49196","Z"); [TCBegin] Verify payout on loss (dealer has blackjack) [ExRes] [GAME_STATUS] Player 2 (1) Won=0.00 0010 : KEYBOARD_KEY_DOWN("68","1","32","D"); 0004 : KEYBOARD_KEY_UP("68","1","49184","D"); 0019 : ADD_SCREEN("COMMON_DIALOG"); 0000 : MOUSE_MOVE("0","580","417"); 0036 : KEYBOARD_KEY_DOWN("27","1","1","VK_ESCAPE"); 0001 : REMOVE_SCREEN("COMMON_DIALOG"); 0001 : MOUSE_MOVE("0","580","417"); 0001 : KEYBOARD_KEY_UP("27","1","49153","VK_ESCAPE"); 0001 : MOUSE_MOVE("0","580","417"); 0044 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); 0002 : KEYBOARD_KEY_UP("13","1","49436","VK_RETURN"); 0036 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0005 : KEYBOARD_KEY_UP("88","1","49197","X"); 0006 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0006 : KEYBOARD_KEY_UP("88","1","49197","X"); [TCEnd] 0014 : KEYBOARD_SYSKEY_DOWN("18","1","8248"); 0013 : KEYBOARD_SYSKEY_DOWN("115","1","8254"); 0000 : SYSCOMMAND("61536","0"); 0000 : ADD_SCREEN("COMMON_DIALOG"); 0000 : MOUSE_MOVE("0","580","417"); 0002 : KEYBOARD_SYSKEY_UP("115","1","57406"); 0001 : KEYBOARD_KEY_UP("18","1","49208","VK_MENU"); 0009 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); 0001 : REMOVE_SCREEN("COMMON_DIALOG"); 0000 : MOUSE_MOVE("0","580","417"); 0000 : REMOVE_SCREEN("SCREEN_BLACKJACK"); 0000 : ADD_SCREEN("SCREEN_DEMOEXIT"); 0000 : KEYBOARD_KEY_UP("13","1","49436","VK_RETURN"); 0000 : MOUSE_MOVE("0","580","417"); 0002 : MOUSE_MOVE("0","580","417"); 0009 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); 0001 : REMOVE_SCREEN("SCREEN_DEMOEXIT"); // Date:           04/11/2003 // Time:           10:26:49 // End of file 

Our game could be run with a command switch that told it to ignore mouse and keyboard input and grab events through the test script instead. Needless to say, we had to modify our core game engine significantly to make this possible.

When the script was executed, the output of the game was logged into a text file that would be compared with similar output from a known good run. If the two outputs were different, it identified a change in the system that could be a bug. Take a look at the output for the above blackjack script:

 // // Automation OUTPUT file // // Generated from script 'd:\automationtool\scripts\bj.dbg.txt' // // This script file was generated via // the automated record/playback feature // // Product:        Casino 2.0 // Build:          02.00.00.0056 // Filename:       d:\automationtool\results\batchrun\results[apr_16_2003].txt // Date:           04/16/2003 // Time:           17:21:00 // // All comments are proceeded by '//' // Commands end with ';' // // // Automation RECORD file // // This script file was generated via // the automated record/playback feature // // Product:        Casino 2.0 // Build:          .00.00.0056 // Filename:       d:\automation tool\scripts\bj.dbg.txt // Date:           04/11/2003 // Time:           10:25:22 // // All comments are proceeded by '//' // Commands end with ';' // 0000 : RANDOM_SEED("1018545922"); [ENVIRONMENT] Player 'Wanda Jean' balance set: 526.00 0000 : SET_START_BALANCE("999900.00"); 0000 : LEAN("1"); [UI] Activating control 'Blackjack' 0001 : REMOVE_SCREEN("SCREEN_LOBBY"); 0000 : ADD_SCREEN("SCREEN_BLACKJACK"); 0000 : RANDOM_SEED("1018545930"); 0000 : MOUSE_MOVE("0","374","342"); 0000 : KEYBOARD_KEY_UP("74","1","49188","J"); 0000 : RANDOM_SEED("1018545930"); [GAME] Table Limits Set: min=25.00 max=1000.00 0000 : MOUSE_MOVE("0","374","342"); 0000 : RANDOM_SEED("89009"); [GAME_STATUS] Beginning of Round [GAME_STATUS] Checking Player Balance (w): 5,000.00 [GAME_STATE] Current state is now BLACKJACK_STATE_TAKE_BETS. 0047 : KEYBOARD_SYSKEY_DOWN("18","1","8248"); 0008 : KEYBOARD_SYSKEY_UP("9","1","40975"); 0020 : KEYBOARD_KEY_UP("18","1","49208","VK_MENU"); 0000 : MOUSE_MOVE("0","371","278"); 0000 : KEYBOARD_KEY_UP("9","1","49167"."VK_TAB"); 0003 : MOUSE_MOVE("0","371","278"); 0016 : KEYBOARD_KEY_DOWN("90","1","44","Z"); [UI] Activating control 'Cycle Avatars' 0006 : KEYBOARD_KEY_UP("90","1","49196","Z"); 0048 : KEYBOARD_KEY_DOWN("90","1","44","Z"); [UI] Activating control 'Cycle Avatars' 0005 : KEYBOARD_KEY_UP("90","1","49196","Z"); [TCBegin] Verify changing table limits working correctly [ExRes] [GAME] Table Limits Set: min=100.00 max=5000.00 0023 : MOUSE_MOVE("0","368","278"); [UI] Mouse over control 'Raise Limit' 0036 : MOUSE_MOVE("0","106","73"); 0009 : MOUSE_LBUTTON_DOWN("1","106","73"); 0002 : MOUSE_LBUTTON_UP("0","106","73"); 0000 : MOUSE_MOVE("0","106","73"); [UI] Activating control 'Raise Limit' [GAME] Table Limits Set: min-100.00 max=5000.00 [GAME_STATUS] Checking Player Balance (w): 5,000.00 [GAME_STATE] Current state is now BLACKJACK_STATE_GAME_OVER. [GAME_STATUS] Beginning of Round [GAME_STATUS] Checking Player Balance (w): 5,000.00 [GAME_STATE] Current state is now BLACKJACK_STATE_TAKE_BETS. 0039 : MOUSE_MOVE("0","170","142"); 0015 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0004 : KEYBOARD_KEY_UP("88","1","49197","X"); 0023 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0006 : KEYBOARD_KEY_UP("88","1","49197","X"); [TCEnd] 0023 : KEYBOARD_SYSKEY_DOWN("18","1","8248"); 0005 : MOUSE_MOVE("0","170","142"); 0005 : KEYBOARD_SYSKEY_UP("9","1","40975"); 0006 : KEYBOARD_KEY_UP("18","1","49208","VK_MENU"); 0000 : MOUSE_MOVE("0","482","312"); 0000 : KEYBOARD_KEY_UP("9","1","49167","VK_TAB"); 0003 : MOUSE_MOVE("0","482","312"); 0018 : KEYBOARD_KEY_DOWN("38","1","328","VK_UP"); [UI] Activating control 'Bet Area' [GAME_STATUS] Player 2 (w) placed bet: 100.00 0002 : KEYBOARD_KEY_UP("38","1","49480","VK_UP"); 0014 : MOUSE_MOVE("0","482","313"); 0002 : MOUSE_MOVE("0","482","313"); 0006 : KEYBOARD_KEY_DOWN("69","1","18","E"); [UI] Activating control 'Cheat Field' 0000 : SET_CARDS("11","1","0","1","0","4","2","1","1","11","0","1",                  "0","1","0","13","3","1","0","8","0","10","1"); [GAME] Deck stacked with 1h 1h 4c 1d 11h 1h 1h 13s 1h 8h 10d 0001 : MOUSE_MOVE("0","590","413"); 0006 : MOUSE_MOVE("0","590","413"); 0016 : KEYBOARD_KEY_DOWN("90","1","44","Z"); [UI] Activating control 'Deal Cards' [GAME_STATE] Current state is now BLACKJACK_STATE_DEAL. 0003 : KEYBOARD_KEY_UP("68","1","49184","D"); [GAME_STATUS] Player 2 (w) (hand 0) got card: 4 of CLUBS [GAME_STATUS] Player 2 (w) (hand 0) has evaluated to: 4 [GAME_STATUS] Dealer  got card: JACK of HEARTS [GAME_STATUS] Dealer  has evaluated to: 10 [GAME_STATUS] Player 2 (w) (hand 0) got card: KING of SPADES [GAME_STATUS] Player 2 (w) (hand 0) has evaluated to: 14 [GAME_STATUS] Dealer  got card: 8 of HEARTS [GAME_STATE] Current state is now BLACKJACK_STATE_OFFER_INSURANCE. [GAME_STATE] Current state is now BLACKJACK_STATE_CHECK_10_FOR_BLACKJACK. [GAME_STATE] Current state is now BLACKJACK_STATE_PLAYER_HAND. [GAME_STATUS] Player 2 (w) chose to stand 0033 : KEYBOARD_KEY_DOWN("83","1","31","S"); [UI] Activating control 'Stand' [GAME_STATE] Current state is now BLACKJACK_STATE_DEALER_HAND. [GAME_STATUS] Dealer has evaluated to: 18 [GAME_STATE] Current state is now BLACKJACK_STATE_PAY_WINNERS. 0004 : KEYBOARD_KEY_UP("83","1","49183","S"); [GAME_STATUS] Player 2 (w) LOST [GAME_STATUS] Player 2 (w) Bet-100.00 [GAME_STATUS] Player 2 (w) Won=0.00 [GAME_STATUS] Player 2 (w) Net=-100.00 [GAME_STATE] Current state is now BLACKJACK_STATE_ACKNOWLEDGE_RESULTS. [GAME_STATUS] End of Round 0031 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); [UI] Activating control 'Clear Table' [GAME_STATE] Current state is now BLACKJACK_STATE_CLEAR_TABLE. 0002 : KEYBOARD_KEY_UP("13","1","49436","VK_RETURN"); [GAME_STATE] Current state is now BLACKJACK_STATE_CLEAR_DEALER. [GAME_STATE] Current state is now BLACKJACK_STATE_GAME_OVER. 0037 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); [UI] Activating control 'Clear Table' [GAME_STATE] Current state is now BLACKJACK_STATE_CLEAR_TABLE. 0002 : KEYBOARD_KEY_UP("13","1","49436","VK_RETURN"); [GAME_STATE] Current state is now BLACKJACK_STATE_CLEAR_DEALER. [GAME_STATE] Current state is now BLACKJACK_STATE_GAME_OVER. [GAME_STATUS] Beginning of Round [GAME_STATUS] Checking Player Balance (w): 4,900.00 [GAME_STATE] Current state is now BLACKJACK_STATE_TAKE_BETS. 0036 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0005 : KEYBOARD_KEY_UP("88","1","49197","X"); 0006 : KEYBOARD_KEY_DOWN("88","1","45","X"); 0006 : KEYBOARD_KEY_UP("88","1","49197","X"); [TCEnd] 0014 : KEYBOARD_SYSKEY_DOWN("18","1","8248"); 0013 : KEYBOARD_SYSKEY_DOWN("115","1","8254"); 0000 : SYSCOMMAND("61536","0"); 0000 : ADD_SCREEN("COMMON_DIALOG"); 0000 : MOUSE_MOVE("0","580","417"); 0002 : KEYBOARD_SYSKEY_UP("115","1","57406"); 0001 : KEYBOARD_KEY_UP("18","1","49208","VK_MENU"); 0009 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); [UI] Activating control 'Yes' 0001 : REMOVE_SCREEN("COMMON_DIALOG"); 0000 : MOUSE_MOVE("0","580","417"); 0000 : REMOVE_SCREEN("SCREEN_BLACKJACK"); 0000 : ADD_SCREEN("SCREEN_DEMOEXIT"); 0000 : KEYBOARD_KEY_UP("13","1","49436","VK_RETURN"); [UI] Mouse over control '25' 0000 : MOUSE_MOVE("0","580","417"); 0002 : MOUSE_MOVE("0","580","417"); 0009 : KEYBOARD_KEY_DOWN("13","1","284","VK_RETURN"); [UI] Activating control '23' 0001 : REMOVE_SCREEN("SCREEN_DEMOEXIT"); // Date:           04/11/2003 // Time:           10:26:49 // End of file // Date:           04/16/2003 // Time:           17:21:50 // End of file 

It turned out that test scripts were fairly sensitive to changes in the code. Anytime a button moved around on the screen the events would fail to press it, and the test script would report an error. A better design for automation would have insulated the test scripts from simple changes in the interface such as a button moving around, but they'll still break if a button is removed. Still, this kind of automation is excellent when it is used to quickly verify stable parts of a game as other development continues. It is also fairly inexpensive to implement in any game that uses a message pump to deliver user input to the game's user interface code.

Gotcha

Any user interface system that uses polling instead of a message pump is extremely difficult, if not impossible to automate.

Our Microsoft Card product used a much better solution for automation, knowing full well the shortcomings of the design on the Casino product. Instead of relying on low level user input, it recorded high level game events, such as moving a card from one position to another. This design was much more robust and worked well in the automation tool.

If possible, you should design your game so that it can play itself. Our card game was written by describing rules in C++ code written to a particular interface. One of the methods in the interface calculated the set of legal moves for that game state, and another method evaluated a move for its desirability. It was then completely trivial to write a piece of code that allowed the game to play itself. Programmers would run their games all night long and find 75% of the bugs before the test team even saw the game.

All of these test tools could be run from the command line, which enabled significant functional testing as a part of the build process. Our development team had a turnkey build process that automatically ran on a build machine, leaving the programmers doing the build plenty of time for some rousing games of Robotron and Joust in our lounge. After a successful build, the automation tool developed by the Microsoft test team would run a few of the most stable test scripts. As a result we never sent the Microsoft team a bad build, saving everyone's weekends.

For those skeptics out there who say that their game is too complicated for automatic testing tools I'll say you aren't thinking hard enough. We had automatic record / playback scripts in Ultima VIII, a multithreaded real time RPG. If Ultima VIII can have it, any project can have it.

Best Practice

Automated testing can also be invaluable in ad-hoc testing. Imagine a tester that is having a hard time coming up with the exact steps to reproduce a bug. If she is recording each and every play session and somehow gets the bug to reproduce, all she has to do is attach the recorded events to the bug report. A programmer will most likely be able to play back her session, see the bug, and fix it.

The bottom line is that every application can and should have as much automation built into it as a part of the overall design. It may be impossible to retrofit an existing title for serious automation, as we found in the Casino project. The results are worth the effort. The team works more efficiently and eliminates monkey work. The best part is you'll only have to make the investment once, because you can install the automation tools in your core libraries and every game you make will have automation.