Beta

By the end of Alpha, the development team should have very clear idea of the game they're creating. The development team has, for the most part, stopped creating new code and new artwork, and will now shift their focus to perfecting what they've already created. It's time to identify and fix the remaining bugs .

Although the term "Beta testing" frequently refers to any outside testing, it is only at the early stages of the Beta phase that final gameplay testing should take place with people outside the design team. The majority of testing done by outside Beta testers during true Beta is bug reporting and load testing. Gameplay feedback and suggestions should continue to be recorded for possible post-release implementation in a patch or sequel.

Beta Phase Entry Criteria

The following Beta phase criteria are typical for a console game:

1. All features and options are implemented. The game is "feature complete."

2. The code passes at least 100% of platform TRC. Toward the end of Beta, the game should be ready for a "pre-certification" submission to the platform manufacturer. This process allows the platform manufacturer's QA team to test the game against the latest TRC and warn of any potential compliance issues.

3. The game can be navigated on all paths. Any bugs that may have closed off portions of the game are eliminated.

4. The entire user interface is final.

5. The game is compatible with all specified hardware and software configurations.

6. The game logic and AI is final. Programming is complete on the "gameplay" of the game. The game knows its own rules. All AI profiles are complete.

7. All controllers work. Those third-party peripherals that have been chosen by the development team (and the publisher) to be supported function with the game.

8. Final artwork is implemented. There should be no placeholder artwork left. Beta is the phase when most screenshots, trailers , and running footage will be taken to use in the packaging and to market the game.

9. Final audio is implemented. All placeholder audio is has been replaced with final assets of the voice talent. (There may be a few do-over, or "pick-up" lines that have yet to be integrated, but these should not have an impact on in-game event timing or level scripting.)

10. All online modes are complete and testable.

11. All language version text is implemented and ready for simultaneous release. The game script (both written and spoken) is locked and can be sent forward for translation and integration into the foreign-language versions of the game.

Design Lock

At some point during Beta testing, the project manager should declare the game to be in a state of design lock (sometimes called feature lock). The play testing has concluded. Questions of balance have been resolved as best they can. The focus of the test team at this point should be to continue to run the test cases against the builds in an iterative manner, because each defect fixed at this point may have introduced another defect elsewhere in the game.

Toward the end of Beta, many tough decisions must be made. The teams are tired , tempers are on edge, and time is running out. In this charged atmosphere, with very little sleep themselves , the project team leaders have to make such critical choices as the following:

• Whether or not to implement that last-minute feature enhancement. The designers may have had a great idea at the eleventh hour and are eager to introduce a new feature, character, or level. The project team leaders must weigh the risks of implementing the new feature (and possibly implementing new bugs and schedule slippage) against shipping a perhaps less compelling game on time.

• Whether to cut that level that just doesn't seem to be much fun. Occasionally it becomes clear during the course of testing that a level or other content component is a "problem child," and requires too much work relative to the time left in the schedule to redesign. Cutting it out entirely may be problematic , however, in that the game will require new tests to ensure that the remaining levels run seamlessly around the deleted features. Critical story information may have been presented in the problem level, and other levels will have to be rewritten (and retested) to accommodate this.

• Which bugs to ship with. In many ways, this is the toughest decision of all ‚ which bugs to let go.

Letting Bugs Go

As a gamer, you may have encountered a defect in a game you bought. Your reaction may likely have been, "I wonder how the testers missed this one?" Chances are they didn't miss it.

There will be times, especially later in the project, when the development team determines that they can't (or won't) fix a bug. This can happen for a variety of reasons. Perhaps the technical risks involved in the fix outweigh the negative impact of the defect. Perhaps there is a workaround in place that the technical support team can supply to players who encounter the defect. Perhaps there simply isn't time.

Whatever the reason, each project must have a quick and orderly process in place to determine which defects will be waived, that is, which will not be fixed by the development team. This designation has many different names . Waived bugs can be known as "as is," ISV (In Shipped Version), DWNF (Developer Will Not Fix), or CBP (Closed by Producer). The worst name I've ever encountered for the waived status is "featured," which institutionalized the cynical joke, "That's not a bug, it's a feature." (Not surprisingly, that studio is now defunct , having released too many buggy games .)

Cynicism, defeatism, and defensiveness have no place in the bug waiving process. On the one hand, testers work very hard and want to feel as though their effort matters to the project. On the other hand, developers work just as long (if not longer) and have a duty to ship their game on time. It is crucial that all parties involved maintain an understanding and respect for the role each plays in the overall project team.

Ideally, the senior members of the project team will meet regularly and often to discuss those bugs that the development team have requested to be waived. These can be flagged as "waive requested " or "request as is" in the Status or Developer Status fields of the bug database. The senior members of the project team (the producer, executive producer, lead tester, and QA manager) can meet to evaluate each bug and discuss the positives and negatives of fixing it versus leaving it in the game. Other team members such as programmers or testers should be available for these meetings as needed. This decision-making body is sometimes called the CCB (Change Control Board) or the Bug Committee.

In some cases, where a post-release software update, or patch , is anticipated, a number of bugs will be designated for fixing after the game has been shipped (see "Post-release Testing," later in the chapter). In the case of most console games, however, this is not yet an option.

Once a bug has been waived, it's important to remind both the bug author and the test team as a whole that merely because the bug was waived doesn't mean that it wasn't a legitimate bug. Nor does it mean that they shouldn't continue to find defects with the same level of diligence. It is the role of the test team to write up every bug, every time, no matter when in the cycle they find it. They supply the lead tester, project manager, and the business unit heads with the best information possible about the state of the game so that the best business decisions can be made.