Pre-Production

Depending on your role on the team and when you were brought into the project, you may think that testing begins sometime after a good portion of the game is developed. In reality, testing begins when the project begins. There may not be people called "testers" involved at the beginning, but project scope, design, and assets are being produced from the start that need to be evaluated, critiqued, and corrected.

Much of what happens at the early stages of the project will set the tone for how well testing will go later on. This means both how good the software stands up to testing, and how well the tests themselves are organized and executed. The bottom line is that both the development and test teams can go home earlier at night if more effort and skill is applied to testing activities at the beginning of the project rather than trying to fix things later on by throwing more testers (and more overtime work) at it.

You can't test quality into a game. The quality of the game is established by the code, graphics, and sounds that are produced and compiled into the game code. All testing can do is tell the development team what is wrong with the code. Testing better earlier can get problems fixed sooner and cheaper.

If you received a coupon in the mail at the beginning of your project that said "send in this coupon to save 20% or more on your project," would you send it in? When you save testing for the end of the project it's like having that coupon but not sending it in because you didn't want to pay for the postage to mail it.

Planning Tasks

Almost as soon as a project is conceived, planning for test begins. Test planning includes the tasks outlined in the following sections.

Determine the Scope of Testing the Project Will Require

The design document, TDD, and project schedule are reviewed by the test manager in order to formulate a "scope of test" document that outlines how much testing resources ‚ that is, time, people, and money ‚ he or she will need to get the game tested thoroughly for release (see the following sidebar, "Expansion Plans").

Expansion Plans

The following is brief scope-of-test memo written at a small publisher planning to develop an expansion pack to an RTS released earlier that same year.

MEMORANDUM

To:	Executive Producer
From:	Manager of Quality Assurance
RE:	RTS EXPANSION TEST PLAN SUMMARY

Summary

It will take 1,760 hours to test this expansion pack, based on the following assumptions:

• 50-day production schedule,

• Four-person test team,

• 10% allowance for overtime, and

• No post-release patch testing.

Single Player (900 hours)

A significant amount of QA time will be spent testing the new campaign. Because the story mode of these missions will be highly script-dependent, testers will be tasked with breaking those scripts to ensure the user will have a seamless, immersive gameplay experience.

Because the developer has not designed cheats in the game, and because our experience with the original game was such that saved games could not reliably be brought forward from prior builds, campaign mode will take up the majority of test time.

Multiplayer (650 hours)

The thrust of multiplayer testing will be to:

1. Ensure correct implementation of new units and the new tile set,

2. Debug new maps,

3. Debug "interface streamlining" (new functionality described in design doc),

4. Stress test game size ,

5. Stress test army size,

6. Stress test game length, and (as time permits )

7. Balance testing.

Because the expansion pack introduces 12 new units, we will be concerned only with high-level balance testing ‚ if one of the new units gives its clan an overwhelming advantage (or disadvantage ), we would bug it out. We do not have the resources available to re-evaluate each of the more than 50 existing units against the new units. We will count on the developer's design team (and user feedback compiled since the release of the original game) to fine-tune the balance of the expansion pack.

Test Matrices (210 hours)

Because this a product for the PC and not a console, there will not be first-party TRC component to the testing. However, we will provide a similar standards-based level of final release testing based on a number of PC standards developed from our own experience as well as standards used at other PC game publishers.

We will run the following standard matrices on the game:

1. Install/Uninstall matrix (with an emphasis on interoperability with the previous product)

2. Windows "gotchas" matrix

3. Publisher Standards matrix

4. Multiplayer connectivity matrix

We will also produce and run the unit matrix developed while testing the original game on each new unit in the expansion pack.

Compatibility (0 hours)

Because the minimum system requirements will not change from the original game, we do not anticipate needing the services of a third-party hardware compatibility lab for compatibility testing. If any machine-specific bugs on the varied hardware in our internal lab crop up during the normal course of testing, we will evaluate at that point whether a full compatibility sweep is warranted.

Overtime (tbd)

Because this product has only modest upside for the company, QA will work with Production to make best efforts to contain overtime costs. At this point we anticipate working overtime only on such occasions that failure to do so will make the product late.

 

Assign a Lead Tester

This is no trivial matter. The lead tester's experience, temperament, and skill set will have a tremendous influence over the conduct of the testing cycle. This may be the single most important decision the test manager makes on the project. A lead tester must be

• A leader ‚ able to motivate the test team and keep them focused and productive

• A team player ‚ able to recognize the role test plays as part of the larger project team

• A communicator ‚ able to gather and to present information clearly and concisely

• A diplomat ‚ able to manage conflicts as they arise (and they will arise)

The test manager, or the lead tester, should then appoint a "vice lead tester," often called a primary tester . On very large teams (for example, more than 30 testers), it's not uncommon to have more than one primary tester, each leading specific sub-teams (for example, multiplayer, career mode, and so on).

Determine Phase Acceptance Criteria

In an ideal world, you will be working from a contract, design spec, or product plan that defines very specific criteria for each phase of testing. But it's seldom an ideal world.

The lead tester should take whatever materials are available and write a specification for the Alpha, Beta, and Gold (release) versions of the game. By establishing clear and unambiguous entry acceptance criteria for each phase of testing, you can avoid conflicts later in the project when you may feel pressure from various parts of the organization to begin, say, Beta testing on a build that isn't truly Beta. Once the test manager has approved these criteria, they should be disseminated to all senior members of the project team.

Three elements are required in the certification planning for each test phase:

• Entry criteria: The set of tests that a build must pass before entering a given test phase. The game won't be considered "at Alpha" until the code passes the Alpha Entry test, for example.

• Exit criteria: The set of tests that a build must pass before completing a test phase.

• Target date: The date both the development and test teams are working toward for a specific phase to launch.

Participate in Game Design Reviews

As mentioned in earlier chapters, all stakeholders benefit from test playing an active role from the beginning of a project. The lead tester or primary tester should participate regularly in design reviews. Their role is not to design the game, but rather to stay abreast of the latest design changes, as well as to advise the project manager of any technical challenges or testing complications that may arise from any anticipated feature revision. Changes in the scope and design of the game will dictate changes in the scope and flow of the testing; forewarned is forearmed.

Set up the Defect Tracking Database

This is a critical step, in that a poorly designed database can waste precious seconds every time someone uses it, and those seconds add up quickly to man-hours toward the end a project ‚ man-hours you will wish you had back! Figure 7.2 shows a typical entry in a bug database ‚ note that the bug type "Unexpected Result" is too general. Aren't all bugs unexpected?

Figure 7.2: Typical entry in a bug database.

The lead tester and project manager should mutually agree on appropriate permissions ‚ that is, which team members have edit rights to which fields. The lead tester should also ask the project manager for a list of development team members to whom bugs will be assigned. The "assigned to" field allows the lead tester, project manager, or anyone else so entrusted to review new bugs and assign them to the right member of the development team to be investigated and fixed. Programmers and artists then search the database for bugs assigned to them. Once they've resolved the bug, they can assign the bug back to the lead tester so that the fix can be verified on the next build.

Whether the bug database is going to sit on an internal server or be accessible over the Internet, it's a good idea at this point to populate the bug database with a few dummy records and double-check all passwords and permissions, both locally and remotely. Every person who will have access to the bug base should be assigned an individual password, and the lead tester can allow or block edit rights to individual fields based on the role that person will play on the project team (see the following sidebar, "Too Many Cooks?").

Draft Test Plan and Design Tests

Having current and detailed knowledge of the game design is critical as the lead tester begins to draft the test documents. Begin drafting an overall test plan that defines what types of tests will be done and what the individual suites and matrices will look like (see Chapter 8, "The Test Process"). This is the point in the project where you can put the methods described in Part IV of this book to good use. Remember: Prior planning prevents poor performance.

Test Plan

A test plan acts as the playbook for the test team. It identifies the test team's goals along with the resources (staff, time, tools, and equipment) and methods that are necessary to achieve them. Test goals are typically defined in terms of time and scope. They may also be tied to dependencies on other groups. The testing timeline often includes intermediate goals for one or more milestones that occur prior to the final release of the game. Any risks that could affect the ability to meet the test goals are identified in the test plan along with information about how to manage those risks if they occur. The scope of a test plan can be limited to a single subsystem of the game or it can span many game features and releases. If the game is being developed at multiple sites, the test plan helps define what test responsibilities are allocated to each team. Appendix C contains a basic test plan outline and the book's CD provides a link to a test plan template document you can fill in for your own projects.

Too Many Cooks?

A defect tracking database (or bug database or "bug base") that is editable by only the lead tester is not very useful ‚ these tend to be very static and incapable of conveying up-to-the minute information about the state of the project. Neither is a bug base in which every member of the team can edit every field ‚ these are chaotic and ultimately useless.

In designing the bug base, the lead tester must balance the need for members of the project team to communicate with each other about a particular defect with the equally important need to control the flow of information to all members of the project team. Programmers need to be able to comment on or ask questions about a defect in the Developer Comments or Notes field, but they can't be allowed to close the defect arbitrarily by changing the Status field to "closed." Testers need to be able to describe the bug in the Brief Description and Full Description fields, but they may not be qualified to judge who should own the bug in the Assigned To field.

Here are some recommendations:

• Status should be editable by the lead tester only. The default value for this field should be "New," so that as testers enter bugs, they can be reviewed and refined by the lead tester before the status is changed to "Open" and is assigned to a member of the development team.

• Severity should be editable by the lead tester or primary testers. Remember that the severity of a defect is not the same as its "fix priority." Testers tend, rightly, to be passionate about the defects they find. It is the job of the test team leaders to check against this and assign a severity in an objective manner.

• Category Fields should be input by the testers and editable by the lead or primary tester. These fields include such specifics as Game Type, Level, Bug Type, Reproduction Rate, and any other field that includes specific information about the bug.

• Brief/Full Description should be input by the testers and editable by the lead or primary tester.

• Assigned To is a field that should be editable by the lead tester and any member of the development team. The lead tester will typically assign new bugs to the project manager, who will then review the bug and assign it to a specific programmer or artist to be fixed. Once the bug is fixed, that person can either assign it back to the project manager for further review, or back to the lead tester so that the fix can be verified in the next build and the bug can be closed.

• Developer Comments should be editable by the project manager and any member of the development team.

• Priority should be editable by the project manager and senior members of the development team. This field is primarily a tool to help the project manager prioritize the flow of work to members of the development team.

 

Test Case

A test case describes an individual test that is to be performed by a tester or testers. Each test case has a distinct objective, which is part of the test case description. A test case also describes what operations to perform in order to meet its objective. Each individual operation within a test case is a test step . The level of detail in the test case can vary based on the standards of a particular test organization. Test cases are conceived and documented by each tester who is assigned a set of responsibilities in the test plan. The total set of test cases produced by a tester should fully cover his or her assigned responsibilities.

Test Suite

A test suite is a collection of related test cases that are described in further detail. The test suite gives step-by-step instructions about what operations to perform on the game and what details to check for as a result of each step. These instructions should be sufficient for manual execution of the test or for writing code to automate the test. Depending on how the detailed tests are written, they may or may not depend on the steps that were taken in a previous test. Ideally, each test in the suite can be individually identified and executed independently of the other tests in the suite. Think of the test cases as individual chapters in a table of contents, while the test suite is a book that puts the test cases together into a detailed, cohesive story.

Testing before Testing Begins

You may soon begin to get proto-builds in bits and pieces, with requests from the development team to do very directed testing of certain narrow features in order to help them build the code. This sub-phase is sometimes called modular testing , because you're testing individual "modules" of code, not a complete build.

At this stage of development, it is entirely likely that as code becomes functional and modules are tested, the design of the game may be revised significantly "on the fly." Patience is required as you make several iterative revisions to your test documents.

During modular testing, it is premature to begin writing bugs beyond the narrow scope of the module's test case. True defect testing won't begin until the game is accepted for Alpha testing.

Finally, the lead tester should begin to recruit or hire additional team members as necessary according to his or her resource plan. Once the team is in place, test kickoffs can begin.