Chapter 16: Game Test Automation

Test automation is a hotly debated topic in the game industry. At one extreme are those who argue that all aspects of game testing should be automated. At the other, many remain unconvinced that test automation in the game industry is viable other than in certain, specific instances. That said, a surprisingly large number of game developers and publishers have made little or no use of test automation nor have they explored its uncharted potential.

Can Automated Testing Work for Me?

The potential advantages of automating game testing are

• Improved game reliability

• Greater tester and machine efficiency

• Greater consistency of results

• Repetition, repetition, repetition

• Faster testing

• The ability to simulate tremendously large numbers of concurrent players for stress and load testing without having to use large pools of human testers

In order to enjoy these benefits, you need to deal with a number of factors that can get in the way of successful game test automation if they aren't handled properly.

Production Costs

The cost of hiring new developers to write additional lines of code into a game to support automation (as well as writing scripts and other test utilities, interfaces, and so on) can be more than it would cost to pay human testers. Furthermore, the reality is that code written for the purposes of automating testing is like any other code. It may not work right and comes with its own set of bugs . It's very possible for automated testing to add significantly to both the cost and development time of a game. Cost overruns and missed deadlines are common in the game industry. The threat of higher costs and further delays without any guarantee (some would argue) of clear benefits makes it hard for many managers to approve a large-scale automated testing program. To have a fighting chance, your test automation strategy must address the areas of the game that can provide the best return on the automation labor and code investment.

Reusability

Reusability of automated test code from one game to another is also a factor that has deterred many managers from approving automated testing programs in their company. Unlike other software businesses, the game industry is famous for creating new code with each new title. Even when the "same" engine or core code is reused for a new game, that engine or code is often modified, repaired, and improved upon. Those changes could prevent some or all of the automated test code from working with the new title. Maximize your automated test reusability by providing functions that hide the game code details from the automated tests. For each new or revised game, only a few functions should have to change in order to re-enable the use of your existing automated test library.

The Human Factor

It is extremely unlikely that even the most cleverly written test automation code will ever replace the need for actual human testers playing your game. For instance, it would be an impressive algorithm indeed that could automatically determine whether a given part of a game was sufficiently "fun" to play. Similarly, human focus group testing is going to remain crucial to gaining key feedback about a game's playability and balance. Even aspects of game mechanics, personal opinions about the design of interfaces, and so on are all likely to be determined by human testers for some time to come. In addition, human testers are likely to continue to out-perform even the most cleverly written test scripts when it comes to the creative "let's see if I can find a way to break this" mentality . The knack or instinct for finding new ways to break the game sets good testers apart from the merely average ones. It is a uniquely human characteristic to think of new and ingenious ways to press a combination of keys and buttons in a way the designer and programmers of the game never thought of to find that the code can crash in the most spectacular of ways and on the most unpredictable of occasions! However, once this has been accomplished, the operations that lead to these crashes can themselves be automated, freeing up time for humans to find more of the playability, balance, and "fun" issues in the game.

Scale Matters

A fundamental decision you need to make is whether your automation is going to be done on a large or small scale. If your vision is to automate virtually all aspects of testing in your company, you will have to take a great many factors into account.

First, large-scale automation will involve a fundamental shift in the way the entire company operates. And for some, this factor alone will mean large-scale test automation is not feasible .

Second, large-scale automation is a long- term investment that may not pay off in the first few games you create once the system is in place. Very long game projects can be an exception, as well a single titles that must be ported to multiple platforms. All key people in the company will need to understand that the payoff may be far off in the future.

Third, test automation is a form of software development. Although this is widely agreed upon, it is also surprisingly frequently overlooked when companies discuss the topic. Automation is not going to be a simple matter of buying an off-the-shelf product or tool that is simply thrown into the development mix and "magically" reduces the time it takes to identify and fix bugs, or to improve playability.

If you intend to go large-scale, you are clearly going to need somebody in your company who can act as a key evangelist for test automation. Without someone at a sufficiently high level who can act as the visionary for the process and who can get behind it and make sure that interest in the idea doesn't wane when there are no immediate "upfront" advantages, the automatic testing program is likely to fail and fall into disuse.

If you are considering a small-scale roll-out of automatic testing tools and programs, your risks are considerably smaller. Start the process by selecting test automations that are likely to give fairly immediate rewards, thereby buoying up support for automation within the company, while only incrementally increasing cost or development time.

Tip ‚  

If you are not committed to a long-term investment, large-scale test automation is probably not right for your company.

Great Expectations

Adding automatic testing tools creates a heightened expectation in upper management. Suddenly, the fact that automatic testing tools are being used can lead managers to have unrealistic expectations about lower production costs, reduced development times, and so on. Indeed, this exact point is a key hurdle facing any test automation advocate in a company. How do you introduce something to management that in the near term might significantly add to cost and production time, but that holds out the promise of potential gains down the road in titles that may not see the light of day for some seasons to come? If there is sufficient stability and commonality in your typical game code or the types of games you produce, then perhaps you will have a clear argument. If your automated test code, scripts, and tools need to be extensively rewritten for each new game, your argument about the long-term payoff will be greatly diminished.

It is vital that you set management expectations accurately and reasonably when proposing to automate even a small part of your testing process. In doing so, you may find you are up against ingrained beliefs about automated testing, the result of media hype, misconceptions of many kinds, and more. Here are some of the false expectations you may face:

• Everything will be automated. For the vast majority of companies it just will not make sense to automate everything.

• Automation will lead to lower costs and greater efficiency. While greater efficiency is hopefully gained where automation is used properly, the fact is that, in the near term at least, costs will rise as extra lines of code and scripts need to be written, management of code increases , greater volume of data is generated that has to be managed and considered , and so on.

• All manual tests will be automated. Much of what is being done manually will not be suitable for automation.

• Tools will be purchased off the shelf. In reality, if your company uses a lot of proprietary game code, your test code and test scripts will probably be custom written too.

• No ramp-up time will be needed. For some reason, automation seems to carry with it the connotation for some of no ramp-up , but the opposite is the case.

• Financial benefits will be immediate. There will be an expectation of instant payback, when in fact it may be several titles down the road before the full positive effects of automation are realized.

• Automated defect reporting will remove the need for human staff. Without human involvement, teams soon discover that automated systems can generate a staggering amount of data. Much of that data should not be sent to the team, because it consists of false positives, false negatives , or merely duplicate data.

"We currently have five machines constantly building and validating data; one for each platform, including the PC, and the "monkey" machine, which runs a test script that loads each portion of the game while mashing controller buttons. The "monkey" machine so far has not been as effective as I hoped; usually somewhere there's a crash bug in the game that takes many hours to solve and the monkey is sitting idle during those hours.

Getting these autobuild scripts built isn't free. It takes a few weeks of coder time, and then requires regular maintenance. Frequently our autobuild machines will fail for one reason or another; a human has to check on them on a semi-regular basis (daily isn't often enough; hourly is too often) to make sure they're still churning away."

Jamie Fristrom, programmer on Spider-Man2

http://www. gamasutra .com/features/20031212/fristrom_01.shtml#

 

Infrastructure

If you are intent on bringing automated testing to your company, a key consideration is to first build a solid infrastructure onto which the automatic testing programs can be placed, selected, and tracked. This will be essential if the test department hopes to keep up with changes that must be made to the test code whenever there is any significant change to a game build, a game's functionality, or its feature set.

Putting this infrastructure in place is software development involving the introduction of extensive database-driven systems to implement and track the test program. Time, resources, and funding need to be allocated to ensure the system is well designed and bulletproof before the necessary step of integrating it with game development. Here is a very important point: The way that all your company's games are written may have to change significantly to incorporate the test code, the hooks, the harnesses, and the frameworks that will enable test automation to be integrated into your company's game-creation process. This is especially true if you decide to go for a large-scale deployment of test automation.

Team Work

Game test automation involves a combination of software development and database management skills. Automating your testing doesn't require a whole new set of skills or a significant change in fundamental company procedures, other than the obvious fact that automation must now be inherent in all steps of the process of creating a title.

Still, you are likely to need a new game team structure that can focus on the needs of creating and rolling out your test automation program. You will need to have key positions on the test team for programmers who fully understand game code. They need to be able to write test code and scripts that fully integrate with the game code or they need to be able to convey clearly and succinctly their needs to the game code developers who can add the necessary hooks and lines of code that automatic testing requires.

Any effort to introduce test automation into a project needs the new test automators to be staffed in addition to the existing game developers and the balance of your testing department. It will be vital for the "automators" to remain uninvolved in the day-to-day issues of either the game code development or manual testing. This sub-team will need to be a well-managed one that works harmoniously and cooperates within itself, as well as being capable of working smoothly with the development team and the rest of the testing department. If you use staff that is already assigned to other testing tasks , you run the risk of significant conflicts between the needs of the "regular" testers and those doing automation. When the demands of manual testing get overloaded (and depending on where in the development cycle your title is, this is likely), the manager, testers, or programmers will find it harder and harder to justify keeping the automatic testing up to date. Potentially , automatic testing can get so far out of synch that it will get put aside in favor of using only human testers. Once the automation is up and running, manual testers can gradually shift their efforts to executing, maintaining, and creating new automated tests. Ideally, over time, every tester should be capable of contributing to both manual and automated testing. This transition period may span many releases or even multiple game projects.

Maturity Test

Is your test department a mature, well-developed organism comprised of well-trained individuals with a clear testing system? Or, do you still tend to undertake game testing on a rather ad hoc basis? Do you use entirely external testers? These factors will determine whether it is advisable for you to introduce automated testing, let alone seek to go entirely automatic. As a rule, adding automation may either be a significant boon to your company or a major disaster depending on the skill set of your team and the existing systems that you have (or do not have) in place. If your projects already tend to go substantially past due and over budget, you may decide that you are an ideal candidate for test automation, believing that it will shorten your development times and lower your costs by increasing your efficiency. Sadly, this is almost the opposite of what usually takes place. If your organization is already running at lower than peak efficiency, adding automation to your testing department could, and probably will, add significant new headaches . It can substantially increase the code that your team has to deal with (potentially hundreds of test scripts that have to be constantly maintained and updated, tracked, and so on) and appreciably increase the amount of test data that your team has to process, store, and analyze. Imagine you are already inundated with thousands of bug reports coming in from your internal and external testers. How will you, your team, and your system cope if an automated program generates tens of thousands of lines of data, much of which can be ignored but none of which can you assume is unimportant?

Introducing test automation in the middle of a project will not help a poorly run test effort. As Figure 16.1 suggests, it's a bad time to automate when your test department is overwhelmed and having trouble keeping up with the demands of development and marketing to get the game completed and bug free. Yet this is exactly the mistake that often occurs. Testers at the end of their wits recommend automation as a way to lessen their workload, not realizing it is (at least in the near term) likely to substantially increase the workload of both testers and programmers.

Figure 16.1: Should you automate?