Conducting Design Research

Anthropologist Rick E. Robinson has outlined three main rules drawn from anthropology for conducting design research:

• You go to them. Designers shouldn't read other people's research on their research subjects from the comfort of their offices. Designers shouldn't make subjects come to them, to an artificial testing environment in an unfamiliar location. Observing the environmentwhere activities are performedis an essential component of any research.

• You talk to them. Designers shouldn't just read about their subjects. Nor should they only ask other people about them. Designers should have subjects tell their own stories in their own manner. The nuances of how a story is told can often tell a designer as much as the story itself.

• You write stuff down. The human memory is faulty. If designers can't write down what they see and hear directly as they do their research, then they should do so immediately afterward.

What Not to Do

Years of marketing methodology have left their mark on ideas about research. The first thing that most people think of when they think about talking to users is assembling a focus group. Don't do this. Focus groups are artificial constructs that, like juries, can be swayed and manipulated by strong participants, throwing off natural results. And that's to be expectedfocus group facilitators assemble people into a synthetic group in an artificial setting (usually a conference room with a two-way mirror) and pepper them with scripted questions. This is not a good way to do design research. Rule #1: You go to them.

Nor is it a good idea to rely solely on the research of others, unless they are on the design team. Without knowing the circumstances and methods of the research, designers typically can't verify that the results are good and that they record the data that is most important to the designer: what the subjects did, said, or made and the environment they were in. This dictum is especially true for data derived from marketing research. Marketing research typically focuses on demographics and attitudessome of the least interesting data from a designer's point of view. Rule #2: You talk to thememphasis on you.

Designers shouldn't rely on a videotape or transcript to capture what they need to remember. Reviewing audio or videotape is a tedious process and will seldom be done, except to find specific moments. Transcripts of tapes, while useful, take time to create even when using a transcription service, and the designer may need the information before the transcript is complete. And there is always the possibility of that dreadful moment when the video camera doesn't record or malfunctions. Designers need to take their own research notes, both for safety and simply to focus their observations. Rule #3: You write stuff down.

Ethical Research

When conducting research, designers should strive to treat their subjects ethically. Not only is this the right thing to do, but it will yield better results, since the subjects will likely open up more if they know and feel that they (and their data) are being treated well. Ethical research requires following these guidelines:

• Get informed consent from subjects. The designer should tell the subject that he or she is conducting a research study and explain the purpose. The subject must understand what is going on and agree to participate, preferably in writing. With studies involving minors, parental or guardian approval in writing is a necessity. Exceptions to this guideline are observations in public spaces where it would be impossible or impractical to get consent from everyone in view.

• Explain the risks and benefits of the study. Some studies carry with them risks. The designer may hear or see something that the subject doesn't want him or her to. The presence of a designer could be dangerous or make certain tasks cumbersome. But the designer should also explain what he or she hopes will improve as a result of the study ("We're going to build a better system for tracking shipments of ball bearings"), both to reassure the subject and to ensure good research results.

• Respect the subjects' privacy. Never use subjects' real names or other data that might identify them. Blur or hide faces in photographs. This will ensure that anything that subjects do or say won't have personal repercussions for them.

• Pay subjects for their time. People's time is valuable, and people who give some of it to provide insights to designers should be paid for it, at least a token amount. This payment doesn't necessarily have to be cash, although it should have value to the subjects.

• If asked, provide data and research results to subjects. Some subjects will want to see what you have recorded and the outcomes of the research. Designers should respect these requests.

Costs and Time

One myth of design research is that it is expensive and time consuming. And while it can besome rare design research projects cost millions of dollars and take place over yearsmost design research takes place over days or weeks and costs in the tens of thousands of dollars. It is time and money well spent.

The materials necessary for design research can be as simple as a notebook and a pen, or as complicated as specialized software and video-recording equipment. Ideally, a research team will have two at least of everything: two notebooks, two cameras (in case one breaks), four pens. The research team itself should consist of (at least) two people who can trade off interviewing/moderating duties during research sessions.

The amount of time spent doing research can vary widely but even a single day spent doing research will improve the outcome of the project. Ideally, however, designers will want enough time to interview and observe a representative group of users. In most cases, this will be more than 10 people, but fewer than 40. Time needs to be set aside not only for doing the research itself, but also for recruiting subjects, which can be quite time consuming itself. There are firms that, when given a set of criteria, will recruit research subjects. Often companies will have a list of customers to recruit from, and there are also ways to recruit users over the Web. However it is done, recruiting takes time. Generally speaking, most design research takes from a week to two months to execute from beginning to end.

What to Look For and How to Record It

When in the field, designers can get overwhelmed with the amount of data they are suddenly receiving. Often the designers are in a strange environment interacting with strangers. The newness of everything makes everything seem important. But the designer needs to focus on observing the things that are truly essentialnamely, specific activities, the environment where activities take place, and the interactions among people that take place during activities.

Designers can help themselves focus by creating a hunt statement. A hunt statement is a tool for narrowing down what the designer is researching and why. Hunt statements typically take this form: I am going to research X so that I can do Y. X is often an activity, and Y is usually a project goal or subject area. Here's an example: I am going to research how civil engineers use PDAs on the job so that I can build a PDA for them. Hunt statements should be developed before doing research so that there is a purpose to each piece of research. The more specific the hunt statement, the better.

Patterns and Phenomena

In the field, the main thing a designer looks for are patterns. These can be patterns of behavior, patterns in stories, patterns of responses to a questionany action or idea that keeps recurring. The rules of thumb are these:

• See or hear it once, and it's a phenomenon. Write it down.

• See or hear it twice, and it's either a coincidence or a pattern emerging. Write it down.

• See it or hear it three times, and it's a pattern. Write it down.

Sometimes patterns won't emerge until after the research data has been analyzed. Sometimes a pattern is obvious in the midst of doing the research. Indeed, one good rule of thumb is that when you start noticing many patterns, you've likely done enough research to draw some meaningful conclusions.

Phenomena are particularly interesting to a designer as well. Unusual behaviorsespecially unusual methods of workingcan suggest directions that will benefit other people in their work. Say an accountant has created a different use for a spreadsheet; perhaps this approach can be built into the spreadsheet so that others can use it as well.

It's never a good idea to do research alone. Having a second pair of eyes, ears, and hands is immensely valuable for observing, listening, and recording, and for discussing and analyzing the research data afterwards. Two people observing the same phenomenon can draw (at least) two distinct conclusions from it, provided both saw it in the first place. Sometimes another person can be valuable simply to help capture the rich data being observed. Patterns can be subtle and easily missed.

Field Notes

Writing down observations and key phrases is essential. Paper notebooks are best and less distracting than laptops or PDAs, unless the environment is one where a notebook may be more conspicuous, such as in an office environment.

All field notes should start the same way: recording the name of the person doing the research and the day, time, and place where the research is taking place. These details are crucial, especially for reference later in the project when these items can provide cues for recalling details. ("Remember that woman in the diner? The one we talked to last Tuesday. What did she say again?") Although the designer may record subjects' names and other data to provide compensation, for instance, this data should be kept separately from field notes, which should use pseudonyms instead of real names to preserve the anonymity of the subjects. Another thing to leave out, no matter how tempting, are personal opinions about the subjects, the observed activities, or overheard conversations, such as "That person is an idiot." Doing otherwise is simply asking for trouble. Subjects, clients, and teammates may want to see the field notes, and showing bias in them is not only unprofessional, but bad research. Bias in research can't be helped, but it can (and should) be minimized.

It's a good idea, however, for the designer to have a separate area on the page to jot down thoughts and feelings that arise during the research sessions, including possible patterns. This should be a place to capture quick reflections or flashes of insight that can be explored more fully later.

Other findings that should be written down in the field notes are:

• Exact quotes with indications of emphasis and toneBob: "I sure do love these controls" (said sarcastically).

• Sketches of the location, annotated with comments and detail.

• The history, steps, and context of any activities.

Still pictures should be taken when and where feasible. Ideally, these will be printed, attached to the accompanying field notes, and annotated with captions or other notes.