Beta Testing


Beta testing is a time-proven quality assurance (QA) technique. When software has been debugged so that all the critical metrics are met, a limited number of end users are invited to use it. As they use it—presumably in a realistic way in their actual environment—other problems occur. Using a structured feedback system, these users then report the circumstances under which these problems occurred to the software's developers.

Eventually, no more critical functionality bugs appear and the software ships.

Although this is a great technique for understanding the problems of software, it's not such a great user research technique. Admittedly, the line can get fuzzy between QA and user research, but there is a line: on one side are the user's needs, desires, and abilities; on the other side are the software's. There are two primary differences: when the research takes place and who participates.

Beta testing is generally done at the very end of a development cycle, right before the site is supposed to go live or the final CDs get pressed. By that point, all functionality has been agreed on, interface decisions have been made, and the identity long decided. Additional knowledge about the users is unlikely to be able to make any fundamental changes in the product. At that point, it's assumed that what is being made is what people want, and what is most important is that it is bug-free. If there are fundamental questions about the product or its users, then it probably isn't ready for beta testing.

Moreover, the people who are invited to test software are not typical users. They're at minimum early adopters and power users, people who are drawn to new functionality, even at the expense of stability. In video game development, for example, there are legendary professional beta testers, experts at breaking other people's software. They serve a very important purpose in making quality software, but not as representative users. Their experiences and perceptions are unlikely to resemble those of any but a small fraction of actual users.

All that said, user researchers should not ignore feedback from beta testing and should work with the QA group in developing beta testing materials. Much like customer support comments (as discussed in Chapter 13), beta testing effectively provides "free" additional information about your user population. It may not be of a typical population using the product in a typical way, but it's part of the population, regardless, and useful insight can be had from the process.

Note

A great way to gather information about users' mental models and expectations is to create a "support" mailing that all the users are on and to encourage them to share their experiences and help each other on the list. These interactions will reveal how people are expecting to use the product and how they're using and misusing it. An unexpected shortcut or workaround by a user can be slightly changed to become a legitimate feature for the product as a whole.

From a user research perspective, several things can be learned from examining beta testing feedback.

  • The makeup of the advanced members of your user population. Who are your bleeding-edge power users? Beta testing programs rarely survey their population and compare it to their "average" population. Using a standard survey, it's possible to understand who the beta population really is.

  • What features they use. Tracking the features where the beta comments cluster, and the kinds of problems that are being reported, can tell you something about the interests and habits of power users, who are often an important segment of the population.

  • Questions about the needs of the population at large. The feedback from beta testing can be used as a trigger for other user research topics. For example, if a lot of HotBot beta feedback refers to the "related page" links, when it wasn't considered to be an important feature even for power users, maybe it's time to research how desired—and understood—that function is for other kinds of users.




Observing the User Experience. A Practioner's Guide for User Research
Real-World .NET Applications
ISBN: 1558609237
EAN: 2147483647
Year: 2002
Pages: 144

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net