The Public Beta

A public beta test is conducted by sending your game to a large audience sometime before it is released to the market. A public beta test is critical for massively multiplayer games because an internal test team simply cannot duplicate the loads and actions of tens of thousands of players distributed on the internet. Any game can benefit from a public beta, especially if your game targets a mass market audience with a wide range of players. Microsoft consistently uses its thousands of employees to beta test all manner of software, from games to applications software.

Selecting the right beta testers is important to getting good results from the test. These people must be hard core players who are willing to put in some serious time into testing. The last thing you need is a bunch of players that try your game for a few hours and spend the rest of the time on the beta test flaming your development team on Google groups.

Make sure you give your beta testers something in return for their effort and they'll be motivated to give you great feedback on each version they download. You might send them a free copy of the game or waive subscription fees for a few months. Another great idea is to give away prizes to your beta testers. Do whatever it takes to keep them working for you.

If you have thousands of people playing your game it stands to reason you'll have hundreds of people sending you feedback. The signal to noise ratio on this feedback is really low and you have to find clever ways of filtering out the vast majority of issues that your internal test team has already found. One way of doing this is by setting up a newsgroup and let the beta testers sort it out for themselves. Related issues will tend to collect in individual threads, and beta testers will check out the threads before creating a new one.

Another good way to get good test feedback from your beta testers is to take a survey approach. Have them fill out a web based questionnaire after every version. Track the answers in a database so you see how changes your dev team makes are affecting the opinion of the public. This method is great for detecting improvements in the game or how the game compares to the competition.

A Tale from the Pixel Mines

I was involved in the Ultima Online Public Beta, which many folks believe was still continuing well after the product shipped. That wasn't very far from the truth. Massively multiplayer games were in their infancy and no one at Origin had experience with the size and scope of the game we were creating. After a few weeks of testing, the executive staff of the project was called in to a meeting where the question of going live was posed. The game was experiencing rapid improvements and rapid change, but it seemed clear that the audience was ready to start playing for real. Electronic Arts was ready to have Ultima Online ship months before Christmas, ahead of Everquest. After the meeting the IT staff was instructed to wipe the game data back to its pristine state and open the ports for the public. The code still had bugs and was undergoing rapid change, even when the public was paying for the game. There was an outcry from the players and even a frivolous lawsuit, but people kept logging on and sending in subscription fees. Who's to say when the public beta really ended?