Data Gathering

---

So far we’ve mostly discussed how to obtain qualitative feedback, but you may also want to go after quantitative feedback, such as recording the time it takes someone to read the rules, or counting the number of clicks its takes to perform a certain function, or tracking the speed at which a player advances in level. You may also ask testers to rank ease of use of certain features on a scale of one to ten, or to choose between several options to see what features are most important to them.

The type of data you gather depends upon the problems you wish to solve. If the game feels clunky and people are taking too long to get started, then measuring the time they spend on procedure to determine where the trouble spot is, might be a good approach. However, if the problem is that the game doesn’t feel dramatic enough, a series of qualitative questions may produce superior results.

Exercise 8.8: Gathering Data

Go back to your original prototype and think of three pieces of quantitative data you can measure that will answer three clearly defined questions you have about the gameplay.

If you’re successful at gathering quantitative data, you may suddenly find yourself buried in statistics. It’s nice to have stats on every conceivable aspect of your game, but if you don’t know how to interpret the numbers, they aren’t much use. We recommend that you conduct your data gathering with clearly defined objectives in mind. Before you set out to measure something, write down your assumptions and purpose. What is it you want to prove or disprove? Then structure your test to either affirm or deny the hypothesis. For instance, you may feel that a certain feature in the game is causing a problem, so you design an experiment that measures the time it takes people to reach a specific point in the game with and without that feature. You may also combine this with a qualitative approach, where you ask the testers how they feel about the new feature. The combination of the qualitative and quantitative should give you the answers you are looking for.

You can take your data gathering as far as your imagination and creativity will permit. Some computer game developers even create software tools to record game statistics in special files during playtesting sessions. This is a sophisticated form of keeping version notes. The developers then write special code to help analyze this data and determine the effectiveness of different game elements and features.

For example, the developer might analyze the effectiveness of all units in an RTS prototype using the statistics gathered from actual playtests. If the data shows that one unit is dominating the others, the developer can then tweak that unit’s variables accordingly and re-test. They may make the dominating unit more expensive to build or less powerful. Or they may tweak the variables of other units to balance the game.

Although statistical analysis techniques like this are powerful tools, it is not a replacement for the designer’s creative judgment on how to tweak game variables. This is because statistics can be misleading. If playtesters are new to the game, they may not be using certain units as efficiently as they could because they haven’t learned the subtleties of play yet. Or, at the other end of the spectrum, if the testers are experienced with the game, they may have set opinions about how to use the units and not see some innovative new way of playing. The bottom line with all data analysis is that it’s a good tool that should be used in combination with other playtesting methods in order to have the best overall results.

---