Five Habits of Effective Automated Testing

Establishing the five practices described in the following sections will positively impact the effectiveness and success of your test automation initiatives.

Filter Automated Test Results

If you decide to move ahead with some degree of automated testing, a key element to put in place early on is some form of filter or prescreening process for the data your test scripts generate. This process should include both human and automated filters to ensure that the same bug is not being reported over and over, or that trivial bugs are being passed on through the system and accorded the same level of priority as important bugs. Indeed, you would do well to implement a tiered system of bug importance, because it will relate specifically to the potentially immense amount of data your test scripts may produce. A high incidence of a particular bug can be an indicator of how important it is to fix it, so your system cannot merely assume that frequently reported bugs are trivial. You will also want to introduce a feedback loop in your internal system to ensure that erroneous "bugs" your scripts turn up are not continuously generated and flooding the system with unnecessary data. Also, see to it that the test scripts are rewritten as necessary to enhance their hit rate on significant bugs ‚ bugs worth noting and eliminating. It is common for the introduction of automated testing to go hand-in-hand with excessive numbers of error email reports that get widely disseminated throughout the company. These take precious time from team members who have to wade through trying to decide which bug report they should attend to and which they can safely ignore. Designating a specific member of the team ‚ an ombudsman or ombugsman? ‚ whose job it is to act as a filter can save tens upon tens of wasted person-hours as you go automatic.

What Happens in Testing Stays in Testing

For very practical reasons, you definitely want to ensure that all hooks and additional code you've added to your game code for testing purposes are removed prior to launch.

Keep the game code clean for launch right from the early stages of development by ensuring that any comments placed in the code or any functions that are introduced for testing (infinite life, invulnerability, hooks for automatic test scripts, and so on) can be easily and reliably removed. Nothing can be more embarrassing than to let code ship with comments like "remember to remove this prior to shipping!" These comments suggest that the code was not in a true release state when it went to retail, which is hardly the image most companies want to project. Where possible, consider whether such code or hooks can be placed in separate DLL files. Clearly label test files separately from the release version of the same files to reduce the risk that the commercial code will enable people to cheat or flood the game servers with simulated players just as you did in your stress test phase of automated testing. Although all of this may sound obvious, a book could be written on code and comments that have been left in games when they were never meant to be there at retail!

Manage Builds

Test procedure automation will be substantially impeded if your company does not have good build management procedures in place. Once you set up scripts to test certain parts of the game code with each new build, any builds that slip through with those parts of the code omitted will throw your automated system into disarray. Your tests will generate a sizable amount of false data in the belief they have found errors. As mentioned earlier, if you have a good filter in place, this need not disrupt your entire project. Nonetheless, it will introduce unnecessary headaches for the person responsible for filtering the automated test output.

Poor build management can also lead to immense confusion with automated scripts if the changes made in a given build should have been accompanied by revised test hooks written into the new game code or revised test programs and tools written to accommodate the new changes in the code. Continue implementing your best configuration management practices in conjunction with automated testing and also consider how they might be impacted by the introduction of automation.

Test Game Code Prior to Committing

In the rush to get a title completed, programmers can be tempted to commit code that is not tested . This can bring the entire project to a grinding halt. Here is one area where test automation can flourish and become integral to the build-management system upon which the rest of the test automation program relies. Building into your system the requirement that every programmer must submit his or her new code for testing before it can be committed is just good sense. This is also something that can be highly automated. By setting up a procedure whereby the route to committing new code is via a test machine, and by automating the tests that the machine carries out before clearing new code for a commit, you can trap just about all common errors well before they have any chance of tainting your main game code. The test scripts you run on this test machine can look for common errors and automate an email notice program that notifies all key personnel of the new code, its new features, and so on ‚ using the notes that each programmer is required to attach to new code they submit to the test machine. Another advantage of automating this system is that you can track the mistakes that are being made and take steps to prevent them going forward.

"One rule you have to have when you allow sloppy submits is this: don't submit if you're going to leave work soon. It's very tempting when you get to the end of the day, and you've put the finishing touch on that feature you were implementing, to submit it and go home. But then you might break the build and someone else is going to have to fix it. Either be willing to wait until the auto-build machines have validated your submit, or wait until the next morning to submit."

Jamie Fristrom, programmer on Spider-Man2

http://www. gamasutra .com/features/20031212/fristrom_01.shtml#

 

Integrate Testing with Building

The practice of ensuring that all new code gets submitted for testing before a coder can make a commit is a significant step in the right direction. It's also essential to a well-ordered system that you ensure your team is working on the same build version and that at any given time the build version is as stable and bug-free as possible. You can achieve this by doing the following:

• At least once a day, automate the creation of a build that also coordinates all new and existing assets and executables. Automated testing can be used to ensure that new assets do not break the existing code and to test new code snippets, audio, and so on.

• Make sure the system embeds version numbers and any other meta-data that might prove useful in the smooth running of the development cycle, such as asset and code source data and so on.

• Make sure the system runs automated regression tests.

• Once a day, have someone run the latest version of the game to check for obvious errors the automated system may have missed.

The potential negative effects of adding test automation to your procedures was mentioned earlier in this chapter. Nevertheless, basic automation of source version control, running automated tests on daily builds, running daily regression tests, and insisting that all new code at least have basic tests run on it before it is committed are just plain good-sense practices. You can save many person-hours simply by keeping obviously buggy code out of the system and by ensuring everyone is working on the same version. Limit the basic test scripts in the automated system by testing just for obvious crashes, omissions, and so on. This will not introduce any significant new test code to the overall system that itself then would need constant debugging and management. Given that it is usually easy to find stable change points in your code, you can write a script that ensures that everyone is automatically synchronized to the latest known stable release that your automatic test system has validated.

"What finally did help is the end-to-end autobuild, or, as we call it, "churning". As soon as one build completes and is successful, the testing machine gets the next change. If there is no new changelist yet, the testing machine idles until another change is submitted. Then it syncs to the new change, rebuilds the data, and emails the team if there are any failures. We've accepted that people are going to make mistakes ‚ giving up on Steve Maguire's claim that it's possible to keep bugs out of the master sources ‚ what we have now is a system that catches those mistakes within minutes of when they're made."

Jamie Fristrom, programmer on Spider-Man 2

http://www.gamasutra.com/features/20031212/fristrom_01.shtml