The Future of Test Automation

There are increasing trends in game design toward procedurally written code and algorithmic approaches to game design. This suggests that in moving forward, it may become easier and easier to integrate automated testing into game code because the code will often be generated by algorithms that may be modified to perform self-detection of errors and self-healing of errors identified. For instance, if the on-screen experience ( path -finding, collision detection, clipping, and so on) is being generated algorithmically, then logically the same algorithmic substructure can be used to test the outcomes and self-cure any errors that are detected .

Beyond the advantages that game design trends may bring, the game industry is likely to remain one where test automation, while desirable, is going to remain difficult to deploy on a wide-scale basis simply because so much game code is custom written for each game. There is likely to remain a fundamental problem that large-scale test automation will be only justifiable if a company can envision ways in which automation will eventually pay off because the test system is reusable for all games they produce over the course of many years . This, however, is unlikely to happen unless the idea of testing the code is built into the game design practices of the company at a very deep level and deployed across all games that the company works on.

Finally, once the practice of designing tests becomes more prevalent , they can be integrated into an automated test generation process. As features evolve during the course of the project, you would only need to update the design and regenerate a new set of automated tests. Here's a small SimScript test that checks that when a player with 0 money bashes a pi ƒ ±ata his money total will go up:

 set_data    avatar   skill    10 set_data    avatar   money    0 use_object  piata   bash wait_until  piata   broken log_message Money should go up after breaking the piata log_dat     avatar   money 

This test case might appear in a combinatorial test design, as shown in Figure 16.2. The test parameters are starting skill, money, and what kind of item to bash. The values for this test are 0, 10, and pi ƒ ±ata, respectively.

Figure 16.2: Bash testing combinatorial table excerpt.

Each of the test values has a small piece of test code associated with it.

Starting Money = 0 is coded as

 set_data     avatar   money   0 

Skill = 10 is coded as

 set_data     avatar   skill   10 

Item to Bash = Pi ƒ ±ata is coded with two lines:

 use_object   piata   bash wait_until   piata   broken log_message  Money should go up after breaking the piata 

Finally, the code that checks the result is

 log_dat     avatar   money 

Using basic test design principles, you can imagine what might be in the rest of the test table. Besides a skill of 10, what about 0 or the maximum possible skill? What if the player's money is maximum before breaking the pi ƒ ±ata ‚ is the money lost or does it lie on the ground for other players to pick up? What other items besides the pi ƒ ±ata can be broken to provide money to the player? These questions may already be answered in the game's design or they might not. All it takes is a few extra pieces of code to incorporate additional test values. For example, the code for 0 skill would not be much different from the code for skill = 10 :

 set_data      avatar    skill   0 

You can generate code for each row in your combinatorial table by substituting the test code for each value that appears in the row. If a new "bashable" item is introduced in an expansion set, just add it to the test table, define new code for it, and regenerate the automated tests.

The same test code generation approach can be applied to test flow diagrams. In this case, test code is associated with each state, event, and action. Once the paths are generated, substitute the test code for each primitive to compose the equivalent automated test. A TFD excerpt for the pi ƒ ±ata bashing test might look like Figure 16.3.

Figure 16.3: TFD excerpt for pi ƒ ±ata bashing test.

The TFD design principles raise some different questions than the previous combinatorial test. Once the pi ƒ ±ata is bashed, what happens if you try to bash it again, creating a "loop" flow at the Pi ƒ ±ataBroken_MoneyUp state? Is there a way to repair the pi ƒ ±ata so it is no longer broken, creating a flow back to the Skill10_Money0 state? As with combinatorial test code generation, you just need to update the design, add new test code for added or changed elements, and regenerate the test paths. Moving or removing existing flows will not cost you anything in terms of new coding. In fact, it's also "free" to add new flows to the diagram that use primitives that already have their test code defined.