Chapter 16: Testing

Overview

Through the entire development phase, testing is performed by a variety of skilled team members. Programmers are constantly checking and testing their code for accuracy, for speed, for efficiency, and against the designer’s specifications. QA (quality assurance) testers perform various tests throughout the development process that varies as the game development cycle progresses.

In the beginning of a game’s development, the testing should check basic design issues such as: Are the graphic images displayed correctly? Are they in the correct location on the screen? Do the buttons work in various modes like mouse-over, clicked-on, and normal mode? Are the correct sounds and sound effects being played? Are the sounds audible? Does the volume need adjusting?

The next phase should test the user interface, the input and user decision paths (the numerous possible actions a player may take), and the basic gameplay.

As the game becomes more playable and stable (fewer glitches and bugs), various hardware configurations should be tested, such as numerous PC brands, CPU speeds, operating systems (Windows 3.1, 98, 2000, Me, and XP are examples), 3D cards, graphic cards (various supported graphic modes), and printers. Actual gameplay from an expert’s POV (knowledgeable, genre fanatic) to a complete idiot’s POV (beginner or novice player) should be tested, and problems that were previously reported and now labeled as “fixed” (problem was resolved) should also be verified. If the game supports multiplayer or Internet options, these components should be tested throughout the day and night for the throughput “test” play speed (latency) and to verify the aspects in the game that should work in multiplayer mode.

A good practice is to have the programmers include in the game an “audit trail” that writes the algorithms that determine all gameplay (a list of each algorithm’s variables and conditions set). This way, when a tester finds an unexpected response or action, the programmer can check the equations and conditions that caused this event. Also, many gaming companies have the QA testers videotape their sessions. Since most developers can never find the problems that QA has uncovered, the tape is a good source for proof and a record of current bugs and problems that need to be addressed. This is especially needed if the testers and developers do not work in the same area.

Testers need to create a common database (records) of a product’s ongoing testing problems and bugs. This “bug report” can be as simple as a text document or a spreadsheet. Better still are databases like Microsoft’s Works, Microsoft’s Access, and Oracle. The report should be in a common area that all testers have access to and provide the producer (the person who is responsible for the project) a uniform report in which to communicate with the developers.

Here is an example of a bug report that a tester could use (feel free to expand upon this):

*Video SMPTE is the Society of Motion Picture and Television Engineers. There is an SMPTE time code standard (hr:min:sec:frame) used to identify video frames.

In the bug report, we want to know everything about the problem (what the tester saw and thinks is a problem). There can be multiple pages based on the same problem if the developers have not resolved it to the tester’s satisfaction. The producer’s report should be listed in “severity” order and status, flagging the number of times this problem has been submitted to the developers.

Upon the bug being fixed, the tester and the senior tester must sign off on the report, and the producer must give the testers a Producer ID number to validate that everyone is in agreement that the bug has been fixed (the testers, the producer, and the development team).

Why does testing matter to the game designer?

The design document should have everything needed to program and create the artwork and audio that the team needs to implement your vision. Missing graphics, empty areas with no ambient sounds or sound effects, and programmers guessing how to code are the result of poor and incomplete design instructions (the design document).

The testers should be able to read your design document and lay out a testing plan from the basic question “does the game follow the design?” (graphics, sound effects, audio ambience, player control, and NPC AI) to all paths that a player can select from the start to termination of the game (successfully completing the game and the numerous ways to end the game early or unsuccessfully). The testers should be able to plan the testing from single player to multiplayer via a network or the Internet.

Once a test plan has been developed, the producer and the testers should have a conference that includes the game designer to discuss the testing plan, to modify any areas not included in the plan, and to sign off on the plan (agree to the strategy of testing the game).