Chapter 1: Two Rules of Game Testing

Whenever I start a new test team or bring a new tester into the group , I give them these two rules:

Rule 1: Don't Panic

Rule 2: Trust No One

Don't Panic

In a game project, panic is a bad thing. The person panicking did not choose to panic, and may not realize it is happening. It is an irrational reaction to a set of circumstances, and it can lead a tester to cause harm to the project. When I sense that a tester is reacting inappropriately to some unreasonable request, I will indirectly remind him not to panic by asking "What's rule one?"

Scuba divers put themselves in a situation similar to what game testers might face: limited resources (the equipment you bring with you), time constraints (air supply), rules to follow (rate of descent/ascent), and other surprises (unexpected sea visitors ). According to Dr. William Morgan, episodes of panic or near-panic may explain many recreational diving accidents and deaths. The panic attack was often spurred by something that a non- diver would deem serious ‚entanglement, an equipment malfunction, or the sight of a shark. But the attacks don't make things better, Morgan says ‚they can lead to irrational and dangerous behavior. ^[1] Even scuba divers with many years of experience sometimes experience panic for no apparent reason. ^[2]

Testing the wrong build, failing to notice an important defect, or sending developers on a wild goose chase after a non-existent bug shouldn't end up getting you physically hurt, but there will be a price to pay in extra time, extra money spent, and/or loss of sales and reputation.

Game project panic happens when you are

• Unfamiliar

• Unprepared

• Under pressure

• Unrested

• Nearsighted

Unfamiliar

As a member of a game team, you might be asked to do something you've never had to do before. You might be given someone else's tests to run, be thrown into the middle of a different game project, or told to take someone else's place at the last minute to do a customer demo. In situations like these, rely on what you know, stick to basics, and pick up any new or different ways of doing things by watching the people who have already been doing it.

You may even be asked to accomplish something you've never done before, such as achieve 100% automation of the installation tests, or write a tool to verify the foreign language text in the game. Maybe no one has ever done this before. Don't make a commitment right away, don't make stuff up, and don't try to be a hero. If you are unfamiliar with a situation, you act based on your best judgment, but it still may not be right. This requires good "radar" on your part to know when to get help, and also a dose of humility so you don't feel like you have to take on everything yourself or say "yes" to every request. You don't need to lose any authority or "street cred." Find someone who's "been there, done that" and can steer you toward some working solutions. Stay away from responses that are known to fail. You can even search the Internet to see if anyone else has been through it and lived to tell about it.

Note ‚ 

Chapter 8, "The Test Process," shows you how to define and follow a set of activities that will give you consistent test throughput and results, even when you're in unfamiliar territory.

Unprepared

Nobody expects the Spanish Inquisition, and a lot of unexpected things will happen on your project. Expect the unexpected! Many parts of the game need to be tested at various points in the game's life cycle. Behind the scenes, many different technologies are at work ‚3D graphics, audio, user interfaces, multithreading, and file systems to name a few. If you are not ready for a variety of test assignments and don't have the skills needed to perform them successfully, then you will stumble rather than star.

Study, practice, and experience are ingredients for good preparation. During the course of the project, try to get to know more about the game code. Keep up with the industry so you are also aware of what the next generation of games and technologies will be like. Become an expert in the requirements and designs for the parts of the game you are responsible for testing, and then get familiar with the ones you aren't responsible for. When you least expect it, you may need to take on a different position, fill in for another tester, or grow into more responsibility. Be ready when it happens.

Note ‚ 

The information in Chapter 5, "The Game Production Cycle," gives you a heads up on preparing yourself to succeed as a game tester, as well as covers what kinds of environments, projects, roles, and jobs you might find yourself in someday.

Under Pressure

Pressure can come from any of three directions:

• Schedule (calendar time to complete the project)

• Budget (money to spend on the project)

• Headcount (the quantity and types of people assigned to work on the game)

There's nothing to prevent one or more of these resources from shrinking at any time during the project. As a tester, these factors won't be under your control. Usually they are determined by business conditions or project managers. In any case, you will be impacted. Figure 1.1 shows the resources in balance with the scope of the project.

Figure 1.1: Resources balanced with project scope.

Moving in any one of these points on the triangle squeezes the project, creating pressure. Sometimes a game project starts out with one of these factors being too small, or they can get smaller anytime after the project has launched. For example, money can be diverted to another game, developers might leave to start their own company, or the schedule gets pulled in to release ahead of a newly announced game that competes with yours. Figure 1.2 shows how a budget reduction can cause pressure on the game project's schedule and headcount.

Figure 1.2: Budget reduction causes pressure.

Another way to cause pressure within this triangle is to try to stuff more into it than was originally planned for. This demand could be internally driven, such as adding more levels or characters , or scrapping the old graphics engine for a new one to take advantage of some newly announced hardware. Other unplanned changes might be made to support more game platforms than originally planned or to keep up with newly announced games in terms of number of levels, characters, online players supported, and so on. Figure 1.3 illustrates how increasing the scope of a project can put pressure on the budget and headcount if they are not increased.

Figure 1.3: Budget and headcount pressure caused by project scope increase.

When there is pressure on the project, you can expect it to get passed on. Someone demands something from you, and uses phrases like the following:

• I/we need ‚ immediately

• I don't care ‚

• That was then, this is now

• Figure out how to do it

• Make it happen

• Deal with it

• We can't afford to ‚

• Nothing else matters but ‚

It's likely that you will get more than one demand at a time, and from different people to boot. Examine the schedule, budget, and headcount available to you. Achieve the request by then scaling down what you would normally do so that it fits in your new triangle. Do the things that will have the most impact on meeting the request to the greatest extent possible. Then in the next release, take care of the stuff that didn't fit this time around.

Note ‚ 

Chapter 2, "Being a Game Tester," introduces you to what's expected of you in your role as a tester, and how to make an impact on the quality of the game.

Chapter 5, "The Game Production Cycle," describes how goals and expectations change as the game progresses from being a concept to hitting the shelves , and how this affects the tester's job along the way.

Unrested

Running long tests after staying up for 30 hours straight or working 100+ hours a week is not the best approach to finding defects. It is, however, a good way to introduce them! When developers do this, they keep testers in business, but it doesn't help get the game released. It's just as bad for the project when testers make mistakes.

Reporting a problem that doesn't really exist (for example, tested the wrong build, didn't do the setup or install properly, and so on) will send the developers on a wild goose chase and waste precious time. If you absolutely have to do testing late at night or at the end of a long week, make a checklist to use before and after the testing. If there's another tester around, have her check your stuff and you can check hers when she does her testing. Also, by writing down some of the information as you go along, you won't be prone to mistakes later on if you have to rely on your tired memory. It's kind of like a pre-launch checklist for the space shuttle. If something is wrong, stop the countdown. Go back and make it right ‚like it says in the test instructions. After testing is done, record pertinent results and facts. The next page includes an example checklist that you can start with and expand on to fit your own game projects.

In addition to putting practices into place for checking mistakes, look for ways to prevent them in the first place. Depending on your game platform and test environment, automation may be a viable option to make your work repeatable at any time of the day. Automated tasks can even go to work while you're at home resting.

Note ‚ 

Chapter 16, "Game Test Automation," and Chapter 17 "Capture/Playback Testing," describe techniques and tools you can use to make the best use of your testing and resting time.

Nearsighted

Panic symptoms can include too much focus on the near term . Many game projects take months, so make that a factor in deciding what to work on today and how to do it. A question I will ask a tester to put him back in the right frame of mind is "Will this be our last chance to test this?" If the answer is "no," then we discuss how to approach the present situation in the context of an overall strategy of repeated testing, feedback from test results, budgeting resources, and so on.

Successful sports teams know how to avoid panic. When they are losing, they're confident that they can come back from behind and win the game because they are a) familiar with the situation, b) prepared to deal with it from practice, film study, and in-game experience, c) rested, and d) don't feel pressure to make up the deficit immediately. Teams that have a losing record often lack one or more of these ingredients.

Late Night Testing Checklist

Pre-Test

Do you have the right version of the test?

Test version: ________________

Are you using the right version of the build?

Build version: ________________

Are you using the right hardware configuration/settings?

Describe: __________________________________

Are you using the right game controller and settings?

Describe: __________________________________

Which installation options did you use (if any)?

Describe: __________________________________

Is the game in the right initial state before running the test case?

Describe: __________________________________

Post-Test

Did you complete all of the test steps in order?

Did you document the completion of the tests and the test results?

Did you record all of the problems you found?

If you reported a problem, did you fill in all of the required fields?

 

Note ‚ 

Chapter 7, "Test Phases," shows you what kinds of testing should be done along the way as the game code matures. This helps you test appropriately for a particular situation and also know that you can rely on the additional testing you will do later in the game.

^[1] "I'm Afraid We Must Talk About ‚ Panic Underwater." The Why Files <http://whyfiles.org/sports/scuba> (January 21, 2005)

^[2] "More questions and answers about Panic Underwater ‚ " The Why Files <http://whyfiles.org/sports/scubaq4.html> (January 21, 2005)