Chapter 11: Surveys

Overview

You know whom you want as your ideal target audience, but do you know who your real users are? You may know some things about which parts of your site are being used and what kinds of issues people complain about. These things may even point to the fact that your actual audience is the people you want, but how certain are you?

Qualitative techniques such as focus groups, think-aloud usability tests, and contextual inquiry give you insight into why people do the things they do when using your product, but they can't accurately outline the characteristics that differentiate the people using your product from the population in general. Only quantitative techniques can predict how many of your users are teenagers or whether they desire the new features you're considering developing. Knowing your audience's makeup can tell you on whom to concentrate your qualitative research and, more important, can give you information about what qualities define your audience.

Who is using your site? Are they the people you had built your site for, or are they completely different from the people you expected? What do they value about your service? Is that what you had anticipated, or are they using your service for something else? Unless you can get the opinions of a large section of your audience, you won't know what makes (or doesn't make) your product popular.

The best tool to find out who your users are and what their opinions are is the survey. A survey is a set of questions that creates a structured way of asking a large group of people to describe themselves, their interests, and their preferences. Once the results are counted, statistical tools can be used to examine your audience, revealing broad characteristics and allowing you to extract interesting subgroups of users. When done correctly, surveys can produce a higher degree of certainty in your user profile than any qualitative research method or indirect analysis of user behavior such as log files.

Surveys can answer such questions as

• How old are your users?

• What kind of Internet connection do they have?

• Is your user population homogeneous, or does it consist of a number of distinct groups?

• What do they want? Does the product provide it?

• What do they like about the product? What do they dislike?

However, surveys can easily go wrong. If not designed carefully, they can ask the wrong people the wrong questions, producing results that are inaccurate, inconclusive, or at worst, deceptive. Webbased surveys are especially vulnerable because, lacking any direct contact with respondents themselves, they depend on the perceptions people have of themselves and their ability and willingness to accurately report those perceptions. Without direct contact, they can't tell you what services they really use or whether their descriptions of themselves are accurate. They can only tell you what they think.