Conducting A Playtesting Session

---

So now that you have all these strangers in your office or living room, what do you do with them? At this point, many game designers panic. They don’t know how to present their game or where to begin. First, take a deep breath, and then think of your role. Your job is no longer that of the game designer and auteur, but that of an investigator and guide, who must lead these individuals through the game and uncover what they truly think.

The most common mistake is for the game designer to sit down and begin reading off the rules and describing their vision for the game. Especially if it’s a physical prototype, the designer feels compelled to start by jabbering away for thirty minutes, making certain to articulate every facet of the game, even those that aren’t there yet, until the testers’ eyes glaze over. Remember, people learn by playing, not by listening. Let them start playing. Allow them to make mistakes. See how each person approaches the game. Maybe your rules are confusing. Provide answers if they get stuck, but for the most part, let your testers figure it out. You will tend to learn more the less you speak.

A good way to control your impulses is to create a “script” for yourself to follow. In this script, you take on the character of a researcher, not the game designer. Your script should include at least the following sections, perhaps several others, depending on the type of test you are doing:

• Welcome the playtesters and thank them for participating.

• Remind the playtesters that you are testing the game, not their skills. Any difficulties in playing the game will help you to improve your design. Give the playtesters the rules (physical prototype), or let them start the application (software prototype).

• Ask the playtesters to begin playing as soon as they are ready.

• Ask them to talk out loud throughout the game about what they are thinking, questions they may have. Warn them that you won’t be able to answer their questions; you just want to know what they are. You are just an observer here. You won’t be stepping in to help them, not because you don’t want to, but because you need to see where problems exist with the game and how they solve those problems.

• When they are finished playing, interview the playtesters or use the following discussion methods to elicit any additional feedback.

• Thank the playtesters for their assistance and feedback. Provide a token gift of thanks if you can afford it.

Exercise 8.4: Writing a Playtest Script

Write a script for the playtest session you set up in Exercise 8.3. Be sure to cover areas of your game that you have questions about. Don’t lead or suggest ideas to the playtesters.

The most difficult part about this process will be learning to listen without responding to every point. You, as the designer, invariably feel a strong attachment to whatever it is you’ve created. This game is your masterpiece. It is something people will judge you by. It’s an extension of yourself. And most designers have the overpowering urge to become defensive. They don’t want anyone criticizing their baby and will do anything to elicit positive feedback.

The Iterative Design Process

by Eric Zimmerman, Co-Founder and CEO, gameLab

The following is adapted from a longer essay entitled “Play as Research” which appears in the book Design Research, edited by Brenda Laurel (MIT Press, 2004). It appears here with permission from the author. Iterative design is a design methodology based on a cyclic process of prototyping, testing, analyzing, and refining a work in progress. In iterative design, interaction with the designed system is the basis of the design process, informing and evolving a project as successive versions, or iterations of a design are implemented. This sidebar outlines the iterative process as it occurred in one game with which I was involved—the online multiplayer game SiSSYFiGHT 2000.

What is the process of iterative design? Test, analyze, refine. And repeat. Because the experience of a player cannot ever be completely predicted, in an iterative process design decisions are based on the experience of the prototype in progress. The prototype is tested, revisions are made, and the project is tested once more. In this way, the project develops through an ongoing dialogue between the designers, the design, and the testing audience.

In the case of games, iterative design means playtesting. Throughout the entire process of design and development, your game is played. You play it. The rest of the development team plays it. Other people in the office play it. People visiting your office play it. You organize groups of testers that match your target audience. You have as many people as possible play the game. In each case, you observe them, ask them questions, then adjust your design and playtest again.

This iterative process of design is radically different than typical retail game development. More often than not, at the start of the design process for a computer or console title, a game designer will think up a finished concept and then write an exhaustive design document that outlines every possible aspect of the game in minute detail. Invariably, the final game never resembles the carefully conceived original. A more iterative design process, on the other hand, will not only streamline development resources, but will also result in a more robust and successful final product.

Case study: SiSSYFiGHT 2000

SiSSYFiGHT 2000 is a multiplayer online game in which players create a schoolgirl avatar and then vie with three to six players for dominance of the playground. Each turn a player selects one of six actions to take, ranging from teasing and tattling to cowering and licking a lolly. The outcome of an action is dependent on other players’ decisions, making for highly social gameplay. SiSSYFiGHT 2000 is also a robust online community. You can play the game at www.sissyfight.com.In the summer of 1999, I was hired by Word.com to help them create their first game. We initially worked to identify the project’s play values: the abstract principles that the game design would embody. The list of play values we created included designing for a broad audience of nongamers, a low technology barrier, a game that was easy to learn and play but deep and complex, gameplay that was intrinsically social, and finally, something that was in line with the smart and ironic Word.com sensibility.

These play values were the parameters for a series of brainstorming sessions, interspersed with group play of computer and non-computer games. Eventually, a game concept emerged: little girls in social conflict on a playground. While every game embodies some kind of conflict, we were drawn towards modeling a conflict that we hadn’t seen depicted previously in a game. Technology and production limitations meant that the game would be turn-based, although it could involve real-time chat.

Once these basic formal and conceptual questions had begun to be mapped out, the shape of the initial prototype became clear. The very first version of SiSSYFiGHT was played with post-it-notes around a conference table. I designed a handful of basic actions each player could take, and acting as the program, I “processed” the actions each turn and reported the results back to the players, keeping score on a piece of paper.

Designing a first prototype requires strategic thinking about how to most quickly implement a playable version that can begin to address the project’s chief uncertainties in a meaningful way. Can you create a paper version of your digital game? Can you design a short version of a game that will last much longer in its final form? Can you test the interaction pattern of a massively multiplayer game with just a handful of players?

In the iterative design process, the most detailed thinking you need at any moment is that which will get you to your next prototype. It is, of course, important to understand the big picture as well: the larger conceptual, technical, and design questions that drive the project as a whole. Just be sure not to let your design get ahead of your iterative research. Keep your eye on the prize, but leave room for play in your design, for the potential to change as you learn from your playtesting, accepting the fact that some of your assumptions will undoubtedly be wrong.

The project team continued to develop the paper prototype, seeking the balance between cooperation and competition that would become the heart of the final gameplay. We refined the base ruleset—the actions a player can take each turn and the outcomes that result. These rules were turned into a spec for the first digital prototype: a text-only version on IRC, which we played hotseat-style, taking turns sitting at the same computer. Constructing that early, text-only prototype allowed us to focus on the complexities of the game logic without worrying about implementing interactivity, visual and audio aesthetics, and other aspects of the game.

While we tested gameplay via the text-only iteration, programming for the final version began in Director, and the core game logic we had developed for the IRC prototype was recycled into the Director code with little alteration. Parallel to the game design, the project’s visual designers had begun to develop the graphic language of the game and chart out possible screen layouts. These early drafts of the visuals (revised many times over the course of the entire development) were dropped into the Director version of the game, and the first rough-hewn iteration of SiSSYFiGHT as a multiplayer online game took shape, inspired by Henry Darger’s outsider art and retro game graphics.

As soon as the web version was playable, the development team played it. And as our ugly duckling grew more refined, the rest of the Word.com staff was roped into testing as well. As the game grew more stable, we descended on our friends’ dot-com companies after the workday had ended, sitting them down cold in front of the game and letting them play. All of this testing and feedback helped us refine the game logic, visual aesthetics, and interface. The biggest challenge turned out to be clearly articulating the relationship between player action and game outcome: because the results of every turn are interdependent on each player’s actions, early versions of the game felt frustratingly arbitrary. Only through many design revisions and dialogue with our testers did we manage to structure the results of each turn to unambiguously communicate what had happened that round and why.

When the server infrastructure was completed, we launched the game to an invite-only beta-tester community that slowly grew in the weeks leading up to public release. Certain time slots were scheduled as official testing events, but our beta users could come online anytime and play. We made it very easy for the beta testers to contact us and e-mail in bug reports.

Even with this small sample of a few dozen participants, larger play patterns emerged. For example, as with many multiplayer games, it was highly advantageous to play defensively, leading to standstill matches. In response, we tweaked the game logic to discourage this play style: any player that “cowered” twice in a row was penalized for acting like a chicken. When the game did launch, our loyal beta testers became the core of the game community, easing new players into the game’s social space.

In the case of SiSSYFiGHT 2000, the testing and prototyping cycle of iterative design was successful because at each stage, we clarified exactly what we wanted to test and how. We used written and online questionnaires. We debriefed after each testing session. And we strategized about how each version of the game would incorporate the visual, audio, game design, and technical elements of the previous versions, while also laying a foundation for the final form of the experience.

To design a game is to construct a set of rules. But the point of game design is not to have players experience rules—it is to have players experience play. Game design is therefore a second-order design problem, in which designers craft play, but only indirectly, through the systems of rules that game designers create. Play arises out of the rules as they are inhabited and enacted by players, creating emergent patterns of behavior, sensation, social exchange, and meaning. This shows the necessity of the iterative design process. The delicate interaction of rule and play is something too subtle and too complex to script out in advance, requiring the improvisational balancing that only testing and prototyping can provide.

SiSSYFiGHT 2000 interface from software prototype

In iterative design, there is a blending of designer and user, of creator and player. It is a process of design through the reinvention of play. Through iterative design, designers create systems and play with them. They become participants, but do so in order to critique their creations, to bend them, break them, and re-fashion them into something new. And in these procedures of investigation and experimentation, a special form of discovery takes place. The process of iteration, of design through play, is a way of discovering the answers to questions you didn’t even know were there. And that makes it a powerful and important method of design. SiSSYFiGHT 2000 was developed by Marisa Bowe, Ranjit Bhatnagar, Tomas Clarke, Michelle Golden, Lucas Gonze, Lem Jay Ignacio, Jason Mohr, Daron Murphy, Yoshi Sodeka, Wade Tinney, and Eric Zimmerman.

Author Bio

Eric Zimmerman is a game designer and academic exploring the practice and theory of gaming. Eric has been making games in the game industry for ten years, and presently runs gameLab, a company he cofounded with Peter Lee. GameLab creates experimental online single-player and multiplayer games. Before gameLab, Eric collaborated with Word.com on the underground online hit, SiSSYFiGHT 2000 (www.sissyfight.com). Other titles include the PC CDROM games Gearheads (Philips Media, 1996) and The Robot Club (Southpeak Interactive, 1998). Eric has taught game design at MIT, NYU, Parsons School of Design, and School of Visual Arts. He is the coauthor with Katie Salen of Rules of Play (MIT Press, 2003) and the co-editor with Amy Schoulder of RE:PLAY (Peter Lang Press, 2004). See also his article on page 312.

SiSSYFiGHT 2000 game interfaces

Please don’t succumb to the devil of playtesting. Ignore your ego. If you’re going to gain anything from a playtesting session, you have to transform yourself. Imagine that you are someone else. You are no longer the designer of this game. Instead, you are an analyst hired to uncover the truth. Your job is not to have these people love the game or you, it’s to discover what they don’t want to tell you or know how to tell you.

Far too many designers fail in this regard. Either they refuse to allow in any negative comments or they stop testing altogether, because it’s too painful. If you pressure your testers or try to control them, you’ll find that they’ll gladly fall in line. You invited them to your office or home, and they don’t want to upset you. They want to please you. And they will tell you whatever it is that you want to hear. So if you are determined to hear only good news, then that’s what you’ll get. It may make you feel like a genius, but it won’t make your game any better.

Embrace the criticism you receive from your playtesters. Even if you feel awful inside, remind yourself that you need to hear the problems because you cannot fix the problems if you don’t know what they are. And it’s better to hear the bad news now than later from a game critic. Don’t let this chance slip past.

There are times when the criticism can get a bit heavy. If you are using a group, one tester may be particularly nasty and begin to sway the others. There’s a nice way to give feedback and a not so nice way, and some people don’t know the difference. They probably haven’t been trained in how to give constructive criticism or participate in group discussions, so please be forgiving. Your job is not to change them or make them better people—it’s to learn what you can from them.

Many professional usability facilities isolate testers for this very reason. However, you may not have that luxury. So, it helps to make it clear at the beginning of the session that you are open to feedback and want everyone to be honest, but at the same time there is a certain etiquette you’d like the testers to follow. Everyone should respect each other’s opinions. There is no right or wrong answer, and no tester should ever criticize another tester’s ideas. Above all, everyone should stick to the game, keeping their comments focused on how the game functions. If you lay down some good rules for the discussion at the outset, you should avoid most problems.

On rare occasions, there will be a tester who acts particularly destructive. This is especially damaging in group settings. If you spot this problem, the best thing to do is politely guide the conversation towards the comments of other testers. If that person continues to behave inappropriately, you should excuse them or ask them to complete a task on their own, like a survey.

Be careful not to abuse your authority. Most people want to be helpful, and it’s rare that someone is trying to sabotage the playtesting session. Before you take any action, be sure to look inside yourself for the answer. Are you being too sensitive? Is the criticism truly harmful or is this person unaccustomed to giving feedback? How are the other testers reacting to this person? It’s true that one bad seed can skew results, casting a negative spin on everything, but do not jump to conclusions. Your ultimate goal is to take what you are given and learn from it, not silence anyone who says something that you don’t like. It’s a fine line that many people have trouble seeing, but with time and experience, you should be able to hone your skills.

Figure 8.3: Playtesting sessions for physical prototypes: Matt Kassan of Atari and Richard Wyckoff of Pandemic Studios give student designers feedback on their designs.

You’ll make mistakes, but if you’re dedicated, it will serve you well. Becoming a good session leader is something that will help you throughout your career. The same skills can be applied to your production team. In addition to playtesters, you need your team’s input and constructive criticism, and the best way to elicit this is to make your entire production a safe environment, where everyone is encouraged to speak their mind while being careful not to personally criticize each other. If you apply the same rules described earlier to all of your group meetings, you will wind up with a far more productive and motivated team that feels invested in the product you are creating together.

Exercise 8.5: Playtesting Your Game

Conduct the playtesting you set up in Exercise 8.3. Use the playtesting script you wrote in Exercise 8.2 to keep the session on track. Take notes in your playtesting journal from Exercise 8.1 recording feedback and problems.

---