Finding the Right Testers

Finding the right testers is perhaps one of a game designer s biggest challenges in playtesting his game. Not just anyone will be able to playtest a game effectively. Almost any player can tell you whether he likes your game or not, but a surprisingly small number will be able to explain why they do not like it and what you might do to improve it. Of course, getting feedback from someone s general impression of the game can be useful: that was fun or that was tedious or that was too hard are all pieces of information you will be able to apply to your work in order to make your game better. Truly useful advice, however, comes in a more constructive form: When I was fighting the twelfth clown on level three, I thought he was too hard to defeat. I had no idea what I was supposed to do to kill him, or whether the attacks I was attempting were having any effect at all. I thought maybe I was supposed to roll the boulder at him, but I could not figure out how to do so. In this example, the playtester has provided the designer with very specific information about the problem and a detailed explanation of why he thought it was not much fun to play. Playtesters who can do that sort of analysis consistently are extremely rare, making a talented playtester a truly priceless asset for your team.

A key part of working with testers effectively is knowing them well enough to know how seriously to take their opinions and what biases they may have. Different testers will have different motivations, which will necessarily color the opinions they give you. This is why picking a random person off the street to test your game can sometimes be ineffective , since you have no past experience with him and hence do not know whether you can trust his opinion or not. When you do have experience with a particular tester, you will be able to know if that person has any shortcomings. For example, some testers can be best described as whiners who complain about everything, even things that do not need fixing. Other testers may be shy, only saying, Maybe you should look at the power of the Elephant Rider unit, when what they truly mean is, Obviously, the Elephant Rider completely throws off the game. Try your best to understand the personalities of the testers you will be working with; it is key to effectively using the feedback they give you. And in the end, regardless of the testers tendencies, if a number of them bring up a similar issue, then something probably is wrong, even if they are diagnosing the problem incorrectly. Similarly, if you have enough random people off the street tell you something is amiss with your game, it is probably worth investigating.

Who Should Test

There are various types of playtesters a project may have, and it is a good idea to have some from each group working on your project. No one type of tester can provide all of the feedback you need for your project. Indeed, it makes sense for there to be a good number of testers, since having a broad range of opinions can be essential to getting beyond individual bias and understanding if your game plays well or not. While arguments can be made for keeping the size of your team small, especially in terms of designers and programmers, with playtesters more truly are merrier.

The first type of playtester is a member of the development team. These are certainly not the only testers you should have, and some would not technically consider these people playtesters. However, throughout the project, it is important to have your team members playing your game. This serves multiple purposes. First, it keeps them enthused about the project. Assuming development is going well, they see to what end their art, sound, code, or level construction is being used. Second, as they see their work in action, they are better able to understand how it might be improved. And third, they can provide you feedback about how the game is working and what you might do to improve it. Toward the end of the project, in particular, as all of the art, most of the code, and the levels are completed, the members of the development team will be able to provide essential feedback about sections of the game that might need some last-minute improvements. Of course, members of the development team are very close to the project, and as a result may be far from objective in their comments about it. Furthermore, since they have been playing the game for so long, they will have trouble seeing it with a fresh set of eyes and their opinions will be skewed accordingly . Also, since they have contributed to the project, they may tend to like or dislike their own work for personal reasons. Similarly, they may like or dislike the ideas of other members of the team not because of the merits of the ideas themselves but rather because of their opinions of that person. Despite these drawbacks, getting playtesting feedback from the members of your team is essential.

The second type of playtester to have is the traditional playtester. This is someone who starts playtesting your game around the stage it enters alpha and is actually fully playable , and continues until the project ships. Often these playtesters spend half of their time tracking down bugs in the code, but they also provide vital feedback about how the game is playing, whether it is too easy or too difficult, if the controls are intuitive or obtuse, and so forth. On fully funded projects, these testers are typically paid employees who spend a full workweek playtesting your game and providing bug reports . Typically these testers love computer games and play a lot of them, both as part of their job and in their off time. Therefore, their opinions of how the gameplay needs to change are understandably skewed to the perspective of the hard- core gamer. Also, since these testers work on the project for such a long time, they can become used to certain inherent problems with the game, and may stop complaining about those shortcomings.

Many first-impression testers were used to refine and perfect the interface in The Sims .

The third class of playtesters are first-impression testers. WillWright, in his interview in Chapter 22, refers to these people as kleenex testers since at Maxis they are used once and then never used again. Wright used them extensively to test the GUI for The Sims . These are people who are neither on the development team nor testing the game full time. Instead, these testers play the game for a short period of time and provide their gut reaction as to how well the game plays. This may be for a few hours or a few days. These first-impression testers are useful because they see the game as first-time players would. They can provide essential feedback about unintuitive controls, unclear presentation of information, or unfairly difficult portions of the game. It is essential when observing these first-impression testers that the developers say nothing to coach them or influence how they play. Furthermore, the important point about first-impression testers is that you must keep bringing in new ones, since a human can only truly have a first impression of a game once; after that they are tainted by their knowledge of how the game works. Especially toward the end of the project, when the development team is extremely familiar with the game and the traditional playtesters have played it for a thousand hours or more, first-impression testers can be essential to making sure the game is not too hard to learn to play.

The fourth type of playtesters includes game designers or developers not actually working on your project. These are people whom you know and trust and whose opinions you respect. They may not be able to test your project full time as traditional testers can, but the feedback they provide can be extremely useful. Fellow game designers who are not working on your project will be able to play your game and provide insight about its strengths and weaknesses in ways that other testers cannot. These testers understand game design in a way that allows them to analyze how your project may come up short and how it might be improved. Many experienced game designers will use these testers particularly early in the process, when they are still trying to get a sense of whether their new game design is truly compelling or not. These game designers turned testers will be better able to overlook the game s obvious shortcomings at this early stage, such as bugs or incomplete features, and can look beyond to see if the game shows the promise of becoming a good game in the future. Steve Meretzky, in Chapter 10, mentions how useful the Imp Lunches were. At these lunches, the Infocom implementers would gather to discuss their different game design ideas. When a new Infocom title first became playable, other implementers would be the first to start testing the game, while there was still time to make any fundamental changes necessary. Of course, fellow game designers will typically be too busy to spend a lot of time playing your game and giving you feedback. Whatever feedback these fellow designers give you can be extremely helpful, both in helping you pinpoint problem areas you had not anticipated, as well as reassuring you that your design is on the right course, if it actually is. If you are not fortunate enough to have developer friends with enough free time to assess your game, there are numerous design consultants in the industry who, for a fee, are available to review your work and provide valuable feedback.

The fifth class of testers that I find to be of particular value are non-gamers. All of the types of testers I have discussed thus far have, for the most part, been pretty big fans of games. They will have an especially high tolerance for the things that games traditionally do badly , such as having overly complex controls or simply being too hard to play. Having some people who are not big enthusiasts can provide fabulous feedback, pointing out fundamental problems that hard-core gamers will overlook and forgive. These testers can be literally anyone: the guy who comes to fix the coffee machine, a neighbor, a team member s parent, or literally someone right off the street. As long as they will be honest about what they think of your game, anyone s opinion can be valuable here. Combining the third group, first-impression testers, with non-gamer testers can be particularly useful in determining if an interface is too confusing or the game is too unforgiving. These testers will seldom be able to provide constructive feedback on how you might improve your game, but they will be able to point out fundamental problems in a way that other testers cannot.

Who Should Not Test

There are a number of people or groups of people whom you typically cannot trust as playtesters. These are people whose opinions are colored by their own personal motivations, or who may be unwilling to provide truly objective opinions. Though you may be forced to hear the feedback of these people, it is important to understand the motivations behind their comments so that you can apply their advice appropriately.

The first of these inappropriate testers is your boss. A key part of the game designer s relationship with a playtester is being able to get the playtester s feedback and then apply it as the designer sees fit, not as the playtester dictates. Playtesters often do not understand the game well enough to provide the best solution for a problem they encounter, and if your boss is the person who has found the problem it is likely he will try to impose a solution on you, even if it is not the best one for the situation. Some bosses may be wise enough to understand that, as the game s designer, you know how best to fix an issue. They show you the problems, and don t care how you fix it. Nonetheless, getting advice from someone who is signing your paycheck cannot be the same as advice from someone who is in a less dominant position.

The second class of people ill suited to testing your game includes anyone from the marketing department. Marketing people have too many conflicting agendas when looking at your game and are unlikely to tell you what they actually think of it. Instead, they will attempt to figure out what the target demographic wants. As I have mentioned repeatedly in this book, it is extremely hard to anticipate what an audience other than yourself will like or dislike, yet this is what marketing people attempt to do. You do not want their second-guessing, which when it comes to gameplay is wrong as often as it is right, to muddle up your game. Furthermore, the opinions of these people are likely to be colored by whatever is hot in the industry right now, often falling into the game of the week syndrome. All this does not mean their feedback does not have value, merely that you need to understand how it is affected by their own agenda. Regardless, you should not consider feedback from them the same way you consider the rest of your playtesting data.

A third group of people who should not test your game consists of those who are too close to you personally , be they your close friends from way back, your family, or your significant other. When these people look at your game, though they may claim they are being objective, their true agenda is often to strengthen their relationship with you. As a result they will be hesitant to criticize your game too harshly. Some friends may understand that the best way they can strengthen their friendship with you is to tell you the truth, but many will sugarcoat their opinions out of misguided kindness. It is true that many authors use their spouses as their first and most effective line of criticism, and if you can develop a relationship that is that honest it can be a wonderful thing. But the fact remains that many relationships are not that honest, and you should not deceive yourself that their feedback is completely objective.

The fourth type of people that you do not want to have testing your game is idiots. Idiots tend to say idiotic things and have idiotic opinions, and as a result will not be of much help to you. It is best to notice and isolate idiots as soon as possible and, if you must work with them, learn to ignore everything they say. Of course, I am exaggerating; idiots certainly do not dominate testing teams . But every so often you will come across a tester whom you are better off ignoring completely.

The fifth group is testers who think that they are designing your game for you. These testers may have some useful suggestions, but mostly will try to get you to change aspects of your game not because they are wrong but simply because they would have done it differently. A truly good tester will recognize that you are the driving artistic force behind the project and that the game will reflect your individual preferences. They will suggest ways to strengthen the game, instead of ways to simply change it.

A sixth group to be wary of is extremely hard-core fans, particularly those who are fanatical about your game s genre or, in the case of a sequel, the previous version of the game. These testers will tend to see every difference in your game from other games in the same genre as being a serious design flaw and will, as a result, stifle whatever creativity you may try to incorporate in your new game. Appealing to the established fans of your franchise can be quite important for sequels; yet following every bit of their advice may result in a game that is not sufficiently different from its predecessors.