Chapter 4: Understanding Users--Qualitative Research

The outcome of any design effort must ultimately be judged by how successfully it meets the requirements of both the user and the organization that commissioned it. No matter how skillful and creative the designer, if she does not have a clear and detailed knowledge of the users she is designing for, what the constraints of the problem are, and what business or organizational goals the design is hoping to achieve, she will have little chance of success.

What and how questions like these are best answered by qualitative research, not metrics or demographics (though these also have their purpose). As we shall see in this chapter, there are many types of qualitative research, each of which plays an important role in filling in a picture of the design landscape of a product.

Qualitative versus Quantitative Research

Research is a word that most people associate with science and objectivity. This association isn't incorrect, but it biases many people towards the notion that the only valid sort of research is the kind that yields the supposed ultimate in objectivity: quantitative data. The notion that numbers don't lie is prevalent in the business and engineering communities, even though we all rationally understand that numbers—especially numbers ascribed to human activities—can be manipulated or reinterpreted at least as dramatically as words.

Data gathered by the hard sciences like physics is simply different from that gathered on human activities: electrons don't have moods that vary from minute to minute, and the tight controls physicists place on their experiments to isolate observed behaviors are next to impossible in the social sciences. Any attempt to reduce human behavior to statistics is likely to overlook important nuances, which though they might not directly affect business plans, do make an enormous difference to the design of products. Quantitative research can only answer questions about how much or how many along a few reductive axes. Qualitative research can tell you about what, how and why in rich, multivariate detail.

Social scientists have long realized that human behaviors are too complex and subject to too many variables to rely solely on quantitative data to understand them. Usability practitioners, borrowing techniques from anthropology and other social sciences, have developed many qualitative methods for gathering useful data on user behaviors to a more pragmatic end: to help create products that better serve user needs.

The remainder of this chapter will focus on qualitative research techniques that support the methods described in subsequent chapters. At the end of the chapter, we'll briefly discuss how quantitative research can, and cannot, be used to help support this effort.

The value of qualitative research

Qualitative research helps us understand the domain, context, and constraints of a product in different, more useful ways than quantitative research does. It also helps us identify patterns of behavior among users and potential users of a product much more quickly and easily than would be possible with quantitative approaches. In particular, qualitative research helps us understand:

• Existing products, and how they are used

• Potential users of new or existing products, and how they currently approach activities and problems the new product design hopes to address

• Technical, business, and environmental contexts—the domain—of the product to be designed

• Vocabulary and other social aspects of the domain in question

Qualitative research can also help the progress of design projects by:

• Providing credibility and authority to the design team, because design decisions can be traced to research results

• Uniting the team with a common understanding of domain issues and user concerns

• Empowering management to make more informed decisions about product design issues that would otherwise be based on guesswork or personal preference

It is the experience of the authors in their practice that qualitative methods, in addition to the benefits described above, tend to be faster, less expensive, more flexible, and more likely than their quantitative counterparts to provide useful answers to important questions that lead to superior design:

• What problems are people encountering with their current ways of doing what the product hopes to do?

• Into what broader contexts in people's lives does the product fit and how?

• What are the basic goals people have in using the product, and what basic tasks help people accomplish them?

Types of qualitative research

Social science and usability texts are full of methods and techniques for conducting qualitative research, and readers are encouraged to explore this literature. In this chapter, the authors focus specifically on techniques that have been proven in our practice over the last decade, occasionally drawing attention to similar techniques practiced in the usability field at large. Rather than bog down in theory, we try to present these techniques from a more pragmatic and less theoretical perspective. The qualitative research techniques we discuss include:

• Stakeholder interviews

• Subject matter expert (SME) interviews

• User and customer interviews

• User observation/ethnographic field studies

• Literature review

• Product/prototype and competitive audits

STAKEHOLDER INTERVIEWS

Research for any new product design, though it must end with understanding the user, should start by understanding the business and technical context in which the product will be built. This is necessary not only to ensure a viable and feasible end result, but also to provide a common language and understanding among the design team, management, and engineering teams.

Stakeholders are any key members of the organization commissioning the design work, and typically include managers and key contributors from engineering, sales, product marketing, marketing communications, customer support, and usability. They may also include similar people from other organizations in business partnership with the commissioning organization, and executives. Interviews with stakeholders should occur before any user research begins.

It is usually most effective to interview each stakeholder one-on-one, rather than in a larger, cross-departmental group. A one-on-one setting promotes candor on the part of the stakeholder, and ensures that individual views are not lost in a crowd. Interviews need not last longer than about an hour, though follow-up meetings may be called for if a particular stakeholder is identified as an exceptionally valuable source of information.

The type of information that is important to gather from stakeholders includes:

• What is the preliminary vision of the product from each stakeholder perspective? As in the fable of the blind men and the elephant, you may find that each business department has a slightly different and slightly incomplete perspective on the product to be designed. Part of the design approach must therefore involve harmonizing these perspectives with those of users and customers.

• What is the budget and schedule? The answer to this question often provides a reality check on the scope of the design effort and provides a decision point for management if user research indicates a greater (or lesser) scope is required.

• What are the technical constraints? Another important determinant of design scope is a firm understanding of what is technically feasible given budget, time, and technology constraints.

• What are the business drivers? It is important for the design team to understand what the business is trying to accomplish. This again leads to a decision point, should user research indicate a conflict between business and user needs. The design must, as much as possible, create a win-win situation for users, customers, and providers of the product.

• What are the stakeholders' perceptions of the user? Stakeholders who have relationships with users (such as customer support representatives) may have important insights on users that will help you to formulate your user research plan. You may also find that there are significant disconnects between some stakeholders' perceptions of their users and what you discover in your research. This information can become an important decision point for management later in the process.

Understanding these issues and their impact on design solutions helps you as a designer to better serve your customer, as well as users of the product. Building consensus internally will help you to articulate issues that the business as a whole may not have identified, build internal consensus that is critical for decision making later in the design process, and build credibility for your design team.

SUBJECT MATTER EXPERT (SME) INTERVIEWS

Some stakeholders may also be subject matter experts (SMEs): experts on the domain within which the product you are designing will operate. Most SMEs were users of the product or its predecessors at one time, and may now be trainers, managers, or consultants. Often they are experts hired by stakeholders, rather than stakeholders themselves. Similar to stakeholders, SMEs can provide valuable perspectives on a product and its users, but designers should be careful to recognize that SMEs represent a somewhat skewed perspective. Some points to consider about using SMEs are:

• SMEs are expert users. Their long experience with a product or its domain mean that they may have grown accustomed to current interactions. They may also lean towards expert controls rather than interactions designed for perpetual intermediates. SMEs are often not current users of the product, and may have more of a management perspective.

• SMEs are knowledgeable, but they aren't designers. They may have many ideas on how to improve a product. Some of these may be valid and valuable, but the most useful pieces of information to glean from these suggestions are the causative problems that lead to their proposed solutions.

• SMEs are necessary in complex or specialized domains such as medical, scientific, or financial services. If you are designing for a technical or otherwise specialized domain, you will likely need some guidance from SMEs, unless you are one yourself. Use SMEs to get information on complex regulations and industry best practices. SME knowledge of user roles and characteristics is critical for planning user research in complex domains.

• You will want access to SMEs throughout the design process. If your product domain requires use of SMEs, you should be able to bring them in at different stages of the design to help perform reality checks on design details. Make sure that you secure this access in your early interviews.

USER AND CUSTOMER INTERVIEWS

It is easy to confuse users with customers. For consumer products, customers are often the same as users, but in corporate or technical domains, users and customers rarely describe the same sets of people. Although both groups should be interviewed, each has its own perspective on the product that needs to be factored quite differently into an eventual design.

Customers of a product are those people who make the decision to purchase it. For consumer products, customers are frequently users of the product; although for products aimed at children or teens, the customers are parents or other adult supervisors of children. In the case of most enterprise or technical products, the customer is someone very different from the user—often an IT manager—with distinct goals and needs. It's important to understand customers and their goals in order to make a product viable. It is also important to realize that customers seldom actually use the product themselves, and when they do, they use it quite differently than the way their users do.

When interviewing customers, you will want to understand:

• Their goals in purchasing the product

• Their frustrations with current solutions

• Their decision process for purchasing a product of the type you're designing

• Their role in installation, maintenance, and management of the product

• Domain-related issues and vocabulary

Like SMEs, customers may have many opinions about how to improve the design of the product. It is important to analyze these suggestions, as in the case of SMEs, to determine what issues or problems underlie the ideas offered, because better, more integrated solutions may become evident later in the design process.

Users of a product should be the main focus of the design effort. They are the people (not their managers or support team) who are personally trying to accomplish something with the product. Potential users are people who do not currently use the product, but who are good candidates for using it in the future. A good set of user interviews includes both current users (if the product already exists and is being revised) and potential users (users of competitive products and non-automated systems if appropriate). Information we are interested in learning from users includes:

• Problems and frustrations with the product (or analogous system if they are potential users)

• The context of how the product fits into their lives or workflow: when, why, and how the product is used, that is, patterns of user behavior with the product

• Domain knowledge from a user perspective: What do users need to know to accomplish their jobs

• A basic understanding of the users' current tasks: both those the product requires and those it doesn't support

• A clear understanding of user goals: their motivations and expectations concerning use of the product

USER OBSERVATION

Most people are incapable of accurately assessing their own behaviors (Pinker, 1999), especially outside of the context of their activities. It then follows that interviews performed outside the context of the situations the designer hopes to document will yield less complete and less accurate data. Basically, you can talk to users about how they think they behave, or you can observe it firsthand. The latter route provides superior results.

Many usability professionals make use of technological aides such as audio or video recorders to capture what users say and do. Care must be taken not to make these technologies too obtrusive; otherwise the users will be distracted and behave differently than they would off-tape.

Perhaps the most effective technique for gathering qualitative user data combines interview and observation, allowing the designer to ask clarifying questions and direct inquiries about situations and behaviors they observe in real-time.

LITERATURE REVIEW

In parallel with stakeholder interviews, the design team should review any literature pertaining to the product or its domain. This can and should include product marketing plans, market research, technology specifications and white papers, business and technical journal articles in the domain, competitive studies, Web searches for related and competing products and news, usability study results and metrics, and customer support data such as call center statistics.

The design team should collect this literature, use it as a basis for developing questions to ask stakeholders and SMEs, and later use it to supply additional domain knowledge and vocabulary, and to check against compiled user data.

PRODUCT AND COMPETITIVE AUDITS

Also in parallel to stakeholder and SME interviews, it is often quite helpful for the design team to examine any existing version or prototype of the product, as well as its chief competitors. Doing so gives the design team a sense of the state of the art, and provides fuel for questions during the interviews. The design team, ideally, should engage in an informal heuristic or expert review of both the current and competitive interfaces, comparing each against interaction and visual design principles (such as those found in later chapters of this book). This procedure both familiarizes the team with the strengths and limitations of what is currently available to users, and provides a general idea of the current functional scope of the product.