Trust No One

On the surface this sounds like a cynical approach, but the very fact that testing is built into the project means that something can't be trusted. You'll read more about this in Chapter 3, "Why Testing Is Important," and in Chapter 5. The very existence of testers on a game project is a result of trust issues, such as the following:

• The publisher doesn't trust that your game will release on time and with the promised features, so they write contracts that pay your team incrementally based on demonstrations and milestones.

• The press and public don't trust that your game will be as good and fun and exciting as you promise, so they demand to see screen shots and demos, write critiques, and discuss your work in progress on bulletin boards .

• Project managers don't trust that the game code can be developed without defects, so testing is planned, funded , and staffed. This can include testers from a third-party QA house and/or the team's own internal test department.

• The publisher can't trust the development house testers to find every defect, so they may employ their own testers or issue a beta release for the public to try it out and report the defects they find.

Don't take it personally . It's a matter of business, technology, and competition. Lots of money is on the line and investors don't want to lose it on your project. The technologies required to produce the game may not even have been available at the time development started, giving your team the opportunity to create the kind of game no one has ever done before. By trying to break the game, and failing, you establish confidence that it will work. Games that don't come out right fall victim to rants and complaints posted on the Internet. Don't let this happen to you!

Balancing Act

Evaluate the basis of your testing plans and decisions. Hearsay, opinions , and emotions are elements that can distract you from what you should really be doing. Using test methods and documenting both your work and results will contribute to an objective game testing environment.

Measuring and analyzing test results ‚ even from past games ‚ gives you data about your game's strengths and weaknesses. The parts that you trust the least ‚ the weak ones ‚ will need the most attention in terms of testing, retesting, and analysis. This relationship is illustrated in Figure 1.4.

Figure 1.4: Low trust means more testing.

The parts you can trust the most ‚ the strong ones ‚ will require the least attention from you, as illustrated in Figure 1.5. These should still be retested from time to time to re-establish your trust. Chapter 5 helps you determine what kind of testing to do, and when to do it. Chapter 8 gives you specific strategies and criteria for planning, choosing, and revisiting testing.

Figure 1.5: More trust leads to less testing.

Note ‚  

Chapter 6, "Software Quality," introduces you to some basic principles for evaluating the trust-worthiness of your game code. Chapter 9, "Testing by the Numbers," describes measurements that you can compile from the test data you normally collect and explains how to analyze those measurements to zoom in on specific problem areas.

Word Games

It's useful to be wary of advice you get from outside the test team. Well-meaning people will suggest shortcuts so the game development can make better progress, but you won't remove bugs from the game by simply not finding them. Don't trust what these people are telling you. At the same time, don't cross the boundary from being distrustful to turning hostile . The whole team is working to deliver the best game it can, even when it doesn't seem that way to a tester.

A general form of statements to watch out for is "X happened , so (only/don't) do Y." Here are some examples:

• "Only a few lines of code have changed, so don't inspect any other lines."

• "The new audio subsystem works the same as the old one, so you only need to run your old tests."

• "We added foreign language strings for the dialogs, so just check a few of them in one of the languages and the rest should be okay too."

And some variants:

• "We only made small changes so don't worry about testing <insert feature name here>."

• "You can just run one or two tests on this and let me know if it works."

• "We've gotta get this out today so just ‚ ."

Tip ‚  

You'll be surprised how many bugs you will find by behaving opposite from the advice you get from other people about what should and should not be tested .

Don't equate a " trust no one" attitude with a "don't do anything you're asked to do" attitude. If a test lead or the project manager needs you to meet goals for certain kinds of testing to be done, be sure you fulfill your obligation to them before going off and working on the stuff you don't trust. The difference is between being a hero ("I finished the tests you wanted, and also managed to start looking at the tournament mode and found some problems there. We should do more testing on that next time around") or a zero ("I didn't have time to do the tests you wanted because I was getting some new tests to work for the tournament mode").

Note ‚  

In Chapter 4, "The Game Team," you learn something about what the other people on the game project are doing, and how your testing affects their work and impacts the progress of the game.

Last Chance

Examine your own tests and look for ways you can improve so you gain more trust in your own skills in finding defects. Just never let that trust turn into arrogance or the belief that you are perfect. Leave room to mistrust yourself just a little bit. Remain open to suggestions from managers, developers, other testers, and yourself. For example, if you're in the middle of running a test and you're not sure you are testing the right version ‚ check it! You may have to go back and start over, but that's better than reporting the wrong results and wasting other people's time too.

As game development progresses, management and developers want to feel comfortable about the quality of the game and its readiness for the next milestone and, ultimately, final release. As a tester, you should not be lulled into complacency. I often re-energize my team by instructing them to "Treat this release like it's our last chance to find problems." Conflicts will arise about whether or not to introduce new tests, and you'll hear complaints about why important problems are found so late in the project. There are many reasons for late defects showing up that have nothing to do with incompetent testing. Here are some you will run into:

• The defects were introduced late, just before you found them.

• Bugs from earlier rounds of testing kept you from getting to the part of the game where the late defect was hiding.

• As you spend more time testing the game you become more familiar with where the defects are coming from, so it is perfectly natural that especially subtle problems might not be found until late in the project.

In any case, even if the bugs were there from the very first release, they were not put there by the testers. Somewhere there is an imaginary world where game testers get to hear "Thank you for finding this important defect right before we shipped it!" but don't count on that happening in our world (refer to Figure 1.6).

Figure 1.6: Our world (left) and the tester loving world (right).

Note ‚  

Chapter 12, "Cleanroom Testing," and Chapter 14, "Play Testing and Ad Hoc Testing," give you methods to use for testing the game based on your testing intuition and insight.

Trust Fund

You can get a head start on knowing what not to trust in a couple of ways. Sometimes the developers will tell you, if you just ask ‚

• Tester: "Hey Bill, is there anything you're particularly concerned about that I should focus on in my testing?"

• Bill: "Well we just redid the logic for the Fuzzy Sword quest, so we definitely want that looked at."

You can get more clues by mentioning parts of the system and seeing how people react . Rolling eyes and pauses in response are giveaways that there is some doubt as to how good that new weapon will work or if multiplayer will work as well as it did before the latest changes.

Note ‚  

In Chapter 3, you find out why testing is important to the health of the game. It covers many factors that contribute to making things go wrong and how you, the game tester, should deal with them.