How Feedback From Typical Gamers Can Help Avoid Disappointing Outcomes

---

How Feedback From Typical Gamers Can Help Avoid “Disappointing” Outcomes

by Bill Fulton, Games User-Testing Group, Microsoft Game Studios

The Problem

Compared to the giddy expectations of the developers at the kick-off of a project, most games are disappointing, commercially, critically, or both. After all, few people set out to spend that much time and money to produce a game resulting in ambivalent reviews and low sales. Solving this problem is one of the holy grails of game development, as it would remove substantial risk from making games.

The Traditional Analysis of This Problem and the Solution

Why does this disappointment happen? The traditional analysis of the problem is that it is because teams are “too close” to their game to see it objectively, much the way that many parents seem to believe their child is above average. Because of this analysis, a myriad of ways to get feedback from fellow game development professionals (co-workers, publishers, journalists, playtest teams, etc.) has sprung up. While the traditional analysis has some merit and the solution to combat the problem is quite useful, it doesn't seem to explain (or fix) the whole problem.

An Alternative Analysis and Solution

An alternative analysis for why games don’t live up to the expectations of the developer is that professional game developers aren’t like the people for whom they are designing the game: typical gamers. Game developers are so knowledgeable about games and game development that they have a hard time designing for the typical gamer who knows comparatively little about games. (See figure for an illustration.)

This situation of game developers being very unlike typical gamers suggests that when the game is fun for the developers, it may not (yet) be fun for typical gamers, who may find it too hard, or may not find the fun that is in the game. This is similar to the way that modern art is often unappreciated by anyone without a degree in art history. But to make games for the masses, it is the responsibility of the game developer to show typical gamers how to have fun with the game.

Gaming expertise

Many publishers and developers have come to see the problem this way, and have engaged marketing research firms to do focus tests on the game to combat this problem. But often the goal of the focus test is to learn how to sell the game, not how to make the game more fun and accessible for more players. Furthermore, focus tests are often done too late in development to make many changes to the game. Because of the constraint of schedule and emphasis on selling as opposed improving the game and time, many game developers are mixed about focus testing.

A comparison of hypothetical distributions of gaming expertise for typical gamers and typical game developers. This figure illustrates how all game developers know more about games than all but the most dedicated gamers. The point of this figure is to show how game developers can't simply make games that are only accessible to people like themselves, if they want to make a game that the majority of gamers can understand and enjoy.

User-Testing From an HCI Perspective

Getting feedback from consumers for the purpose of improving products is a major goal of the field of usability, a subset of the human-computer interaction (HCI) field. Most major software companies have usability departments staffed with HCI professionals. The games industry has been slow to adopt this practice.

But this is changing—the use of HCI professionals in game development is gaining greater acceptance. One major game publisher has been doing some form of usability work on games since 1998, but other game publishers and developers are beginning to experiment with usability methods as a way to make their games more fun. As more game developers and publishers do usability testing on their games in development, the typical quality of games from those developers and publishers will only get better.

An Example of User-Testing from Age of Empires 2: Age of Kings

Age of Empires 2 (AoE2) is an excellent example of how user-testing from an HCI perspective can improve games. The first AoE game was both a critical and commercial hit. In fact, it sold so well that the only way the sequel (AoE2) could sell any better would be if it expanded beyond the kinds of gamers who played the first AoE.

The developers and publisher decided to “aim for the stars” and make the game accessible to nongamers—that AoE2 would be a game that someone who had never played a computer game would be able to pick up and play. This was a lofty goal, because AoE2 is a complicated game and non-gamers lack the background to learn the game on their own, and because we knew from testing that the first AoE was a difficult game to learn for some experienced gamers.

In order to achieve this level of accessibility, it would be necessary to provide a robust tutorial and do a great deal of user-testing. The details of the testing are better described in a different article, but the following anecdote from the final test of the tutorial gives a bit of flavor:

The final test of the tutorial was done on a Sat., at 10 a.m. At 9 a.m., I noticed an elderly lady (maybe in her 70s or 80s) waiting outside the building. I thought she was lost or looking for someone, but it turned out that she had been scheduled for the test. I was surprised, but she technically fit the kind of people we were looking for (never played a retail computer game, could operate a computer, was older than 40), so I let her in. I apologized for her being given the wrong time for the test, but she told me that she was told 10 a.m. was the time, “but always showed up an hour early for appointments.”

I was a little concerned that she might be put off by the nature of the game (build a nation, raise an army, destroy your neighbors), and offered that she could leave if she wanted. But she thought the idea of testing a game was “interesting” because her grandkids played them, and she wanted to be helpful. So we let her go through the test like all the other middle-aged folks. It was a bizarre sight to see dozens of parent and grandparent types playing Age of Empires 2 in the lab.

After they had completed the tutorial, they were instructed to play a random map game against the computer. Towards the end of the test, I went by the elderly lady and saw that she had the semblance of a nation going—she had several villagers collecting all four resources, and had many of the right buildings built (barracks, granary, mining, etc.). When the Mongol hordes came over the hill and invaded her nation, she did several things right—she hid her villagers, and started to build a (woefully inadequate) army. Unfortunately, she was too slow and got overrun; Age of Empires 2 had just crushed grandmother’s nation. When I escorted her from the lab, I asked her what she thought—she said she could see how her grandkids would like it, but the game wasn’t her “cup of tea.”

While the grandmother didn’t enjoy the game, after completing the tutorial she was able to understand the basics of the game and responded reasonably to being attacked. This was a dramatic improvement over the original AoE, where sometimes even experienced gamers got stuck and couldn’t figure out the game without going to the manual. The reliance on testing AoE2’s tutorial with real people and not just paid game-industry professionals, resulted in a game that almost anyone can pick up and play.

In the end, AoE2 sold dramatically more units than did the first version, in large part due to improvements to the game that stemmed from doing user-testing throughout the development of the game.

Author Bio

Bill Fulton has been doing HCI work on games since 1997, and is one of the founders of the Games User-Testing Group at Microsoft Game Studios. The group’s mission is to get feedback from typical gamers for the purposes of improving games in development. By Dec. 2002, there were 28 HCI professionals providing feedback from typical gamers to games in development including Age of Empires 2, Halo, Project Gotham Racing, Rise of Nations and Freelancer throughout the development process. To read more about HCI and games, see their web site at http://www.microsoft.com/playtest/publications.htm.

Microsoft playtesting lab (Photo by Kyle Drexel)

Tape of Halo user test. The inset shows a player’s hand responses monitored

---