Taking Notes

 < Day Day Up > 



As mentioned, it’s imperative to keep notes of your playtests. You think you will remember all of the comments later on, but what you will really remember is those comments you expected to hear or wanted to hear. If you don’t keep notes, you’ll lose all the really important details of the playtesters’ reactions. These notes should be filed chronologically in a notebook or folder, or entered into a database. Each time you conduct a test, write down the date of the test, all feedback gathered from your testers, and any of your own observations.

Figure 8.7 is a form you can use to capture observations and playtester comments. It is broken into three parts: (1) in-game observations, which are thoughts that you write down while the testers are playing the game; (2) post-game questions, which are questions designed to help elicit opinions about the key aspects of a game system; and (3) revision ideas, which is a space for you to articulate ideas for making the game better.

You may be asking yourself right now, “What should I be testing for?” Don’t worry—that’s the subject of the next two chapters. For right now, here are some general questions you might ask of your playtesters. After you’ve gone through Chapters 9 and 10, you can create your own questions, specifically geared for your own game.

Sometimes it’s more effective to interview people and write down their comments than have them fill out forms. This is because filling out forms is a chore. Interviewing playtesters one-onone is a very good method, but if you can’t do that, then group interviews are also fine. Just make sure to encourage everyone to speak their mind, and try to keep the more aggressive members from dominating the discussion.

start figure

You’ll find that sometimes not all of the questions on the form will be relevant. For example, if you are testing for interface flaws, then data about the overall play experience may be less important to capture. We encourage you to tailor this form to your specific needs. Many of the questions will be unique to a game, so it’s important for you not to rely on our questions but to create your own. Questions designed to get at issues that you have with your particular game will be the most valuable to you.

end figure

Figure 8.7: Observations and Playtester Comments

IN-GAME OBSERVATIONS

Dramatic elements

[Your thoughts as you watch the testers play.]

  • 1. Was the game’s premise exciting?

  • 2. Did the story enhance or detract from the game?

  • 3. As you played, did the story evolve with the game?

  • 4. Is this game appropriate for the target audience?

  • 5. On a piece of paper, graph your emotional involvement over the course of the game.

  • 6. Did you feel a sense of dramatic climax as the game progressed?

  • 7. How would you make the story and game work better as a whole?

IN-GAME QUESTIONS

 

[Questions you ask the testers as they play.]

 
  • 1. What did you feel as your turned ended?

  • 2. Does the navigation seem confusing?

  • 3. Why did you move to that location?

  • 4. Why are you pausing there?

 

POST-GAME QUESTIONS

Procedures, rules, interface, and controls

[Questions you ask the testers after they’ve played.]

  • 1. Were the procedures and rules easy to understand?

  • 2. How did the controls feel? Did they make sense?

  • 3. Could you find the information you needed on the interface?

  • 4. Was there anything about the interface you’d change?

  • 5. Did anything feel clunky or awkward?

  • 6. Are there any controls or interface features you’d like to see added?

General questions

 
 

End of session

  • 1. What was your first impression?

  • 2. How did that impression change as you played?

  • 3. Was there anything you found frustrating?

  • 4. Did the game drag at any point?

  • 5. Were there particular aspects that you found satisfying?

  • 6. What was the most exciting thing about the game?

  • 7. Did the game feel too long, too short or just about right?

  • 1. Overall, how would you describe this game’s appeal?

  • 2. Would you purchase this game?

  • 3. What elements of the game attracted you?

  • 4. What was missing from the game?

  • 5. If you could change just one thing, what would it be?

  • 6. Who do you think is the target audience for this game?

  • 7. If you were to give this game as a gift, who would you give it to?

Formal elements

 
  • 1. Describe the objective of the game.

  • 2. Was the objective clear at all times?

  • 3. What types of choices did you make during the game?

  • 4. What was the most important decision you made?

  • 5. What was your strategy for winning?

  • 6. Did you find any loopholes in the system?

  • 7. How would you describe the conflict?

  • 8. In what way did you interact with other players?

  • 9. Do you prefer to play alone or with human opponents?

  • 10.What elements do you think could be improved?

 
 

REVISION IDEAS

 

[Ideas you have for improving the game.]

start sidebar
Getting The Most Out Of Focus Groups

by Kevin Keeker, Designer, Microsoft Game Studios

Kevin Keeker spent the early years of his career working on game projects as a usability engineer. Here he shares some insight into the psychology of focus groups and how to get the most out of them.

Many people believe that focus groups are a good way to evaluate their games. I’ve learned that focus groups aren’t the best way to gauge the quality or popularity of your ideas. Instead, focus groups should be used to generate ideas for your game. A well-run focus group is one where the participants are encouraged to speak freely and disagree with one another if necessary. This environment can generate ideas that will fuel your own creativity and provide a glimpse into the common points of wisdom and key disagreements in your gaming audience. This sidebar describes why focus groups are better for generating ideas than evaluating them. Then it provides a few pointers to help you achieve either objective.

Let’s say that you’re designing a snowboarding game and you’re feeling pretty good about it. You know that you’re making the game for teens and young adults. You know that you need a great sense of speed, big air, tons of attitude and crazy tricks. You’ve been tuning the basic play of the game with usability feedback from your teens and young adults. They’re able to pull off the tricks and find some of the fun scenarios that you’ve positioned around the course. Meanwhile, you’ve got to refine the attitude part.

Music is a huge part of snowboarding culture. You know that. You know that the kids like the punk rock. After all, you make videogames. You’re just a 30-year-old man-child. So, you talk to some labels, pick some tunes and plan a focus group to validate your musical choices.

This is all a lot of fun, until the dozen boarders in your focus group room go into heavy posturing. “What are your favorite bands?” Some start eagerly throwing out names. Others snipe at these suggestions. A third set of participants sinks sullenly back in their seats, while a couple of boarders drift away into the powder.

To reel everyone back in, you remind the group that this is a brainstorm by eagerly accepting all suggestions and going around the room one-by-one. This generates a pretty sizable list of bands with most of the overlap in tastes centering on (expensive) bands with some widespread popularity.

Thankfully, lots of the bands mentioned could be labeled punk if you’re just a little generous in your categorization. At least you can be confident that you’ve validated punk as an enjoyable musical style for most of the snowboarding crowd.

Now you move on to the music you’ve picked. You play a song. Ask people to give it a thumbs-up or thumbs-down. And please explain their opinion. You notice the participants noticing each other. They look around the room as they make their decision. In the end the bands that are familiar names receive the clearest enthusiasm. At least a few people have heard of them. Most of the songs receive halfhearted enthusiasm. A few of them no one likes. During the wrap-up, you ask for an overall consensus on the musical selection. A few people passionately argue for something other than punk music. The group as a whole agrees that variety is the key.

You’re left with a very uneasy consensus. What do you do now? You could go with your gut. But then the focus group has been a waste of time and a truckload of money. You try to sum thumbs and go with the songs that evoked the least ire. But that leaves you at risk of a very bland musical selection.

This scenario points out the fundamental problem in focus groups. They’re very good for generating ideas and very poor at validating them.

Group interaction seeds individual creativity by encouraging us to examine differences between our opinions and those of others. The thoughts of others remind us of the way we feel ourselves. The differences between our ideas and others spur us to distinguish our ideas. They also encourage us to try out alternate perspectives and potentially incorporate elements of those perspectives into our own ideas. Creativity is this process of incorporating new elements into our ideas and putting together disparate ideas to create new ideas.

However, a similar process can lead people to avoid stating differences with others. It takes effort to disagree with others and to generate a plausible reason why you differ from others. Furthermore, there’s a good chance the other people won’t like you if you disagree with them. These are two good reasons to avoid disagreeing by accepting a common consensus. Disagreeing becomes significantly harder if you perceive that you’re the only person with an opinion. This perception comes quickly in group settings where one person may state an opinion and others may quickly agree. The onus is then on the dissenters to come forward. But the dissenters may take time to reevaluate their position. These delays in disagreement further support the appearance that there is consensus.

So, what do you do? As a designer who has worked in user-testing, I have a leg up. I know how to ask questions the right way. When you want to generate ideas, you create a group brainstorming session (focus group) and do everything that you can to encourage people to disagree in a safe and constructive fashion.

When you want to evaluate ideas, you survey people individually. Give each person a concrete list of alternatives and ask them to choose or rank those alternatives. To get a clear answer, you need to present people with trade-offs. Do you want this song or that song? Rank these songs in terms of which songs you’d most like to see in the game. Preferably you also present these choices in the context of a clearly defined value system. It’s not enough to know that X is better than Y. You need to know and maybe Y are good enough to be included in the game. Ask each user to clearly mark the boundary above which songs are good enough to be in the game and below which songs should be dropped. Another way to do this is to ask users to compare the songs to ones that they think would be good, bad, or adequate for the game.

Author Bio

Kevin Keeker trained as a social and personality psychologist at the University of Illinois and at the University of Washington before stumbling into usability engineering. Since 1994 he has worked on a variety of entertainment and media-related products at Microsoft. After managing Microsoft Game Studios' usability group, he shifted focus to apply his user-centered design experience as a game designer on Xbox sports titles.

end sidebar

A good way to begin is to identify key areas of your game you need input on and create questions geared to get feedback on those areas. Write down more questions than you plan to use and then rank them in order of importance. Then group the top questions by type, as we did previously. You can develop your own categories of questions and structure. It really comes down to the type of information you wish to gather and how your playtesting sessions are structured.

One thing to avoid is getting carried away and overwhelming your playtesters. If you ask someone twenty or more questions in a row, they’ll become exhausted and may stop answering accurately. Remember, it’s not the number of questions you ask but the quality of the responses.



 < Day Day Up > 



Game Design Workshop. Designing, Prototyping, and Playtesting Games
Game Design Workshop: Designing, Prototyping, & Playtesting Games (Gama Network Series)
ISBN: 1578202221
EAN: 2147483647
Year: 2003
Pages: 162

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net