Types of Games to Automate

Clearly some games are better suited to full-scale test automation than others, although most would benefit from at least some degree of automation.

Massively Multiplayer Games

One key genre that has had reported success with test automation is that of massively multiplayer (MMP) games like The Sims Online and World of Warcraft . It is precisely these massively multiplayer, persistent state world games or sizable role-playing games that have been such a challenge to code and test prior to retail launch. The sheer scale of these systems and their distributed nature makes them inherently difficult to test. Yet, the stability of these worlds at launch is clearly desirable, given the negative impact that a lack of stability will have on customer service calls, player experience, and your company's reputation. Here, then, is an ideal opportunity for automatic testing to shine . Even with MMPs, automated testing is not for all parts of the game, but generally more for those elements that

Have a high number of repetitive actions
Must simulate a very large number of players connected at once
Are synchronized among a large number of servers and clients simultaneously

Simulating server load, for instance, has been a key use of automated scripts. Simply having large, open Beta tests cannot always consistently or reliably give you the stress/load data you are seeking ‚ at least not in a repeatable and consistent fashion that is ideally sought after if you want to launch a stable product.

Another key use for automatic testing scripts in massively multiplayer games is that of repetitive behaviors and their effects on the game code and lag issues. Complete testing of such games can include picking up and dropping any given object thousands (if not tens of thousands) of times, or entering, leaving, and re-entering rooms thousands of times. For human testers to undertake this form of testing is not only extremely tedious , but also unreliable and unscalable. Surprisingly, as many obvious advantages as this approach seems to have, many companies building MMP games have not written such scripts and have relied on Beta testers to return the data they seek. If they do create such scripts, they often limit them to certain classes or races of characters instead of creating them for the entire game. This undoubtedly reflects the lack of upper-level management commitment to test automation in today's games industry. But if these tools are to be created, the testing management needs to ensure that they are not only easy to integrate into current game code, but also flexible enough to be reused in future games.

Clearly, automated techniques are going to be of significantly less use for MMPs where code is highly complex and often changed, or where there has historically been a low incidence of errors. Similarly, as described earlier, it is probably best left to manual testing to discover small bugs in interfaces, individual game loop logic, characters getting stuck in the environment, minor graphical clipping issues, and so on.

Ideally with an MMP, whether you are testing highly repetitive tasks or seeking to emulate server load, you will want to create test scripts that mimic actual players as closely as possible. For this reason you will likely want to have the actual main game code run the scripts and, where possible, use prerecorded actions of real players as templates upon which to build your automated scripts (another reason that even with test automation, human testing is unlikely to entirely disappear!). Furthermore, where possible you will want to have test clients speaking to the same server that live players speak to, so that the main game server cannot tell whether it is receiving input from live players or your automated script-based client. To achieve this, you can create a simple scripting language that sends common commands to the game server, such as pickup_object , buy_object , open_door , and so on. Ideally, you will also provide a simple interface so testers can easily set up scripts to automate a series of repetitive actions by indicating how often a given loop must be repeated, for example.

Other Types of Games

Many other types of games can benefit from automated testing, too. Level-based games such as first-person shooters could have sequences of play tested by well-written scripts that imitate a player repeatedly shooting in a certain pattern. Additional tests could be automated to randomly test paths through a level. Such tests would seek out instances where clipping occurs, check if the game hangs , or find where players can pass through walls and floors. Although automated tests are unlikely to be of much use on level balancing or the subjective elements of playing a level (how much "fun" it is), any aspect of the design that can be tested by repeated action is optimal for automated testing. For instance, it can be highly tedious to have to test the entire GUI of a game over and over again with every new build. This is traditionally an area where human testers are ‚ or can be ‚ inefficient. Simple scripts that automatically test each combination of button press or menu item selection can significantly increase the early detection of bugs in the game code and lessen the chance that any bug will persist to retail release of the game.

On the face of it, simple puzzle games might seem to be the least likely candidates for automatic testing. In fact, testing can be extremely useful for such titles. Many early EDGE games produced in the 1980s were puzzle games in which the clear danger faced by testers was that they knew the solution to the game and the players did not. Thus, automated scripts were used in even simple puzzle games to specifically test what would happen if players made moves the programmers had not expected them to make. These were moves that the human testers tended to miss . The result of that approach was that not a single error of this kind was reported in any of the puzzle games released during that period. Needless to say, this is a good way to make your manager a fan of game test automation!

Mobile phone games are another genre potentially well-suited to test automation. A developer can be required to have any given game run on many tens or hundreds of mobile phone handsets over the life of the title. Most handsets differ only slightly from each other, but some differ substantially in annoyingly disparate ways. Automating the testing of game screen displays, key press actions, and menu selections for each phone is advantageous over purely manual testing. Very modular test code is essential to cost-effective automation of these mobile phone games.