Diaries

Directly observing people as they use a product over time is difficult, time consuming, and likely to affect how they use it. Diary studies are, as the name implies, based on having a group of people keep a diary as they use a product. They track which mistakes they make, what they learn, and how often they use the product (or any thing else that is of interest to the experience researchers). Afterward, the diaries are coded and analyzed to determine usage patterns and examined for common issues. Even with minimal analysis, they can provide a source of feedback that reveals patterns that would be difficult to identify otherwise. They are best done on fully functional products.

For practical reasons, indirect observation can provide good knowledge at a much lower cost. Diary studies are one of the least expensive ways to track how use of your site changes over time for specific users (virtual usability testing, described in Chapter 16, and clickstream log analysis, described in Chapter 13, are other ways, but they both involve a lot more data processing, labor, and expense). Letting people track their own progress can give you a view into their experience without actually having to stare over their shoulders—literally or metaphorically—for months at a time.

Diaries are also one of the only geographically distributed qualitative research methods. You can have people fill out diaries around the country (or the world, for that matter) without leaving your office. This allows you to perform research into internationalization and into how cultural and geographic differences affect people's experiences of your product. Before beginning a diary study, you need to determine several things based on how often people use your product.

• The duration of the study. There should probably be at least a half dozen diary entries for trends to be observable (though this of course depends on many factors). If the product is used once a week, the study will need to go on for about two months to be able to measure a change. However, if the product is expected to be used every day, then a week may be sufficient to see changes in people's use.

• The study's sampling rate determines the levels of detail of the trends that you can observe. The more frequently people fill out diary entries, the more subtle changes you can notice in their experience. However, changes happen at all levels of detail, and there are important monthly and yearly trends even in products that get used many times a day.

Since people aren't diary-filling machines, picking a sampling rate and duration that won't overly tax their time or bore them is likely to get you better-quality information. For example, a study of how moderate search engine users learn a new search engine started by recruiting people who searched roughly once per day (with a different search engine than the one being researched). Picking daily searchers defined the maximum sampling rate since asking them to fill out more than one diary entry per day would not have produced any additional information.

There are two kinds of diary study: unstructured and structured. Unstructured diary studies are participant driven. Within a loose structure, the "diarists" relate their everyday experiences, tracking their learning and the problems they encounter as they encounter them. Structured diary studies resemble extended surveys or self-administered usability tests. Under the remote guidance of a moderator, diarists perform specific tasks and examine specific aspects of the product, reporting their experiences in a predetermined diary format.

Note

Recruiting people who are dedicated enough to fill out diaries over an extended period of time will likely bias the kinds of people you find. The diligence that comes with such willingness may also bring along other qualities that are not representative of your general user population. However, the value you get from conducting diary studies will likely offset this bias, though you should be aware of it when analyzing your results.

Five to ten participants should be enough for a small research group (or even a single researcher) to manage. Participants should be recruited for their availability throughout the testing period, their willingness to commit to regular participation, their ability to articulate their experiences, and—in most cases—a similar experience level among all the participants. Punctuality is also nice, but difficult to screen for.

Unstructured Diaries

Completely open-ended diary studies are rare. Even so-called unstructured diary studies have some structure to focus on issues that are important to the developers.

After being recruited, screened, and invited to participate, diarists are instructed in what's expected of them. The instructions should be specific, brief, and complete. They should give the diarists a set of guiding principles about what kind of behavior should be recorded while encouraging active participation.

Sample Diary Email Instructions

Thank you for participating in our evaluation of HotBot, Wired's search engine. We are in the process of evaluating it for redesign, and we would like your input so that we can base the design changes on people's daily experiences with it. For the duration of this research, we would like you to use HotBot for as many Internet information research needs as you can. Your thoughts and experiences are very valuable to us.

For the next month, you will receive this instruction sheet along with a diary form twice a week (on Mondays and Thursdays) in email.

It's a good idea to remind people of their obligations and to set some expectations. Attaching the instructions to every email reminds the participants of how they should think about filling out the diaries.

We would like you to fill out the diary form and email it to <[e-mail]> before the next one arrives (mailing it on the same day is fine). We estimate that it will take 20–30 minutes to fill out completely.

If you can't email the diary to us by the time the next form arrives, contact us as soon as possible. Contact information is at the bottom of this sheet.

Contact information should be easily accessible and people should be reminded it's there whenever appropriate.

The form is designed to help you describe your experiences while using HotBot to look for information. You don't have to fill it out at any specific time, and you can write as much as you want. But we would like you to fill out as much as you can.

Here are some things to keep in mind while you're filling out the form.

• Relate as much as you can about your experiences, positive and negative, big and small. We are interested in all of it, no matter how minor it may seem at the time. We are especially interested in experiences that were surprising or unexpected.

• Our goal is to make HotBot work better for you. If you can't get some feature to work, it is not your fault. Please describe any such situations in detail.

As with all situations where people report their experiences, they should be assured that they're not the ones being tested, it's the product, but that it's their responsibility to accurately report such failings.

• Whenever you try a feature of HotBot that you have not used before please tell us about it, whether or not you were able to get it to work. Describe the situation in which you used it in detail.

People may not know when they're using something new, but it doesn't hurt to encourage them to look for and record novel situations.

• If you have a problem with HotBot, but are then able to solve it, please describe the problem and your solution in detail.

• Please include the specific search terms you used whenever possible.

Encourage specifics when possible. Search terms and products are relatively easy to note down, so it's feasible for people to record them, but more abstract or labor-intensive tasks should be avoided. If people are asked to write down everything they clicked on and every checkbox they selected, they're likely to spend all of their time writing about the minute elements of their experience instead of the more important issues.

• If you're not sure about whether to put something in the diary or not, please put it in.

• If you have not performed any searches by the due date of the form, please mark it as such and return it.

If you have any questions, please don't hesitate to contact Mike

Kuniavsky at <[e-mail]> or [phone].

After the instructions are sent out, it may be useful to review them with the diarists either in person or through some other real-time method (the phone, instant messaging, etc.) so that subtleties can be clarified.

Along with the instructions, there should be a standardized form for reporting experiences. The form of the diary research is quite flexible, in general, and you should feel free to experiment. The form's goal is to give the diarists a relatively open-ended forum while encouraging them to share their goals, problems, and insights. The forms can be distributed as plain text email, as HTML email (only if you know that all of your participants can accept HTML email), or as URLs that point to individual HTML forms.

The specific content of the diary forms will depend on the product and your goals for the research. The forms should be first tested for coherence and timed not to take too much of the diarists' time (a typical diary entry should take 20 minutes or less to write, 30 minutes at the longest).

As with all experience research techniques, setting out specific goals at the beginning of the study is key to the study's success. As the responses start coming in, they should be carefully read. If it appears that the kinds of responses being received will not meet the goals of the research, the format of the study should be adjusted. Fortunately, this is pretty straightforward since the diary forms can be easily changed and the instructions adjusted (though all changes should be prominently noted in the instructions to avoid confusion). Even diary studies where people are given no specific instructions will probably provide some interesting results (although few studies give no guidance at all).

Structured Diaries

Adding structure to a diary is a way to make sure that specific aspects of the experience are examined. An unstructured diary can produce great information about aspects of the product that you don't care about and never touch on ones you most need to research. Explicit instructions to diarists give you a better chance that they comment on what's important. However, that same structure can bias the responses. The participants may look at parts of the product they have not looked at before, describe it in ways that they normally wouldn't, or use it in novel ways.

There are three different kinds of structured diaries: diaries that are structured like surveys, diaries that appear like usability tests, and diaries that resemble problem reports.

Survey-Structured Diaries

One kind of structured diary is more like an extended survey, with a set of questions that the diarists answer to complete each diary entry. Questions can repeat in order to measure directly how the diarists' experience changes or focus on different areas of the product with every entry.

For example, a more survey-structured version of the HotBot diary form could look like this.

As described earlier, asking people for the specifics of their actions is worthwhile as long as it's not burdensome. So instead of asking for all the things the participant may have searched for since the last diary entry, this form asks for just the most recent search.

Open-ended questions give participants the opportunity to explain their experiences in depth. Asking people to present a narrative of the changes in their experience, although prone to the bias of selective memory, can compress the analysis process, providing direct insight into how people's views and thoughts are changing.

This form adds several questions about a feature cluster on the front door interface. This allows the researchers to get feedback on a specific part of the product.

Usability Test Diaries

Another kind of diary study is more like a remote usability test, where the diarists are asked to perform specific tasks that exercise certain features or expose them to certain content. They are asked to evaluate these parts of the site and describe their use of it. Over time, you can track which of the features participants have been introduced to they actually use. This type of study is most useful when studying how the use of specific features changes over time.

Again, it's useful to set people's expectations for what kind of information they are expected to provide and how they should approach the task in order to make it easier for them to provide the information you need and not to feel bad if they can't do it. (Loosely based on Reiman, John," The diary study: A workplace-oriented research tool to guide laboratory efforts." In Proceedings of INTERCHI, 1993 Amsterdam, ACM, New York.)

This form introduces the participants to a specific set of features and creates a task that can be more quickly accomplished using those features. By examining the diarists' descriptions of their search process, it's possible to see whether or not the Advanced Search interface is understandable and whether people use it after an initial introduction. Question 3 is also modified to request information about the timing of recent searches in order to see if people's search duration changes based on exposure to additional features (of course, this is too coarse of a measure to see anything but the most extreme behavior changes).

Problem Report Diaries

Still another kind of diary form resembles a problem report or a critical incident report. It's filled out whenever the diarists have problems or insights. Although this does not have the focus or depth of a remote usability testlike diary, it minimizes the amount of bias that the study has on the users' behavior, and it highlights the most painful parts of the interface. Such problem report studies can be run continuously throughout the product lifetime to document the kinds of problems people have. Although somewhat "soulless," they're easier to deploy on a large scale, thus increasing the total rate of feedback. They're not as good for tracking use systematically since they depend on people remembering to fill them out when they have problems or insights.

Depending on diarists' diligence and their perception of what a problem or an insight is, filling out this form can become a big chore. Defining the specific parameters that the development team is looking for, communicating that to the diarists, and creating a strong incentive (whether material or in terms of reminders) reduces the no-response rate somewhat. Some amount of ambiguity will always remain, but the scope of what the participants need to write down should be specified as closely as is reasonable.

Continually evaluate how well the diary forms are working to address the goals of the research. If they're not working (if the diarists are not discussing what you had hoped, if they're not responding as completely as you'd like, etc.) try to identify the problem and change the forms appropriately.

Managing Responses

One of the keys to a successful diary study is the management of responses and respondents. Since much of the content is self-initiated, it's easy for people to treat the diaries as optional. To maximize response, you have to give people incentives and remind them of their obligations.

If participants are not sufficiently encouraged, there will be a high dropout rate, which is equivalent to a high self-selection rate and skews the responses. The people who finish an underencouraged diary study will likely not be representative of your target audience.

Incentives should be proportional with the amount of time that it takes participants to complete the work. Six diaries may well take a total of three hours to complete, with an extra hour of overhead spent managing them. They should be paid at a rate somewhere between 60% and 100% of what you would pay participants who came to your lab. Assuming a standard rate of $60 per hour, this means that each participant in a two-month study that requires two diaries per week should receive roughly $200 as an honorarium at the end of the study. It is possible to do this less expensively (one diary study I was involved in paid $50 and a T-shirt), but you risk high dropout rates—only one person out of eight finished that study. You may want to encourage completion with additional surprise incentives (T-shirts, restaurant gift certificates, etc.) placed throughout the process. When working within a company where incentives may be inappropriate, consider asking the participants' managers to allocate time to fill out the diaries.

Reminders can be important tools in bridging the gap that happens when you're not doing research face to face. Reminding the participants of their commitment and of the importance of that commitment to the product can inspire them to provide better feedback—and to put the diary on their to-do list. Prompt telephone follow-up to missed entries tells the participants that you're still there and that their feedback is important to you. Regular email communication thanking them for their responses and answering their questions also reinforces this, but don't spam them with a dozen "keep up the good work!" messages a day.

How to Analyze It

Analyzing diaries is similar to analyzing focus groups. The techniques discussed in Chapter 9 mostly apply here and will not be repeated. Starting with the goals of the research, the contents of people's responses are systematically examined for common themes. These themes are then pieced together into trends.

The first part of analysis is simply reading the diaries. This should start as soon as entries begin coming in. Don't wait until the end of the research. Read the entries and begin comparing the responses between the participants. Compose hypotheses about people's behavior and address the initial goals. Note what other trends appear. As more diaries come in, test and modify these hypotheses based on the new information.

Diary analysis can stop with careful reading, but there are more rigorous techniques that can be applied. Formal analyses, although more time consuming, can reveal use patterns that are not obvious in regular analysis, and they bring rigor to generalizations made from the data. Rather than saying "most people learned about the advanced options after roughly a month" after a formal analysis, you can say "four out of six participants were regularly using the advanced options five weeks into the study."

Rigorous Trend Extraction

To make trends clearer, issues raised by the diary entries can be abstracted so that the same kinds of phenomena are called by the same name, regardless of who wrote them or when.

Coding the diaries, as described in Chapters 9 and 13, is an organized method of tagging observations with consistent labels. Once a workable set of codes is established, each diary entry is examined and tagged with codes that represent categories of comments that occur in the entry. You may want to code for items that support or oppose your hypotheses. This is also a good time for pulling out representative quotations.

Note

Follow up with an email or a phone call if a diary entry is ambiguous. The responses will, of course, be filtered through the person's memory, but a specific question (or asking them to look at the Web page or section of the product under discussion) can cut to the root of a problem or provide a nugget of insight, saving you guesswork.

Once you have all the entries coded, start organizing the entries. The themes along which you group responses will depend on the questions you want to answer, and there is an infinite number of ways it can be done. There may be interesting groups of comments in a single person's experience, such as when he or she learns to use the tool better and notes down ever more sophisticated comments, or clusters could appear across several people's experiences if everyone learns at the same rate. Similarity between behavior can be found in the content of what people say, in the relationship between people's comments, when they say it, and which features the comment refers to (some features may have lots of comments about them, others few).

Timelines can often immediately show you a number of interesting patterns that would be otherwise difficult to observe. For example, the HotBot study could map the amount of times "Advanced Search options are mentioned" (one of the coding categories) against time in order to see whether people mention the advanced options more as they use the product.

The process of coding and organizing should be enough to give you an idea of what kinds of trends are apparent in the data. Once you have a list of information clusters, however, you have to relate them to people's behaviors somehow.

Some relationships are obvious. Lee, in Figure 12.1, doesn't discuss Advanced Search options often. However, as the research progresses, he discusses them more and more. Likewise, even though the users didn't discuss Advanced Search options at first, by the fourth week four of the five talked about them at least once.

Figure 12.1: Mentions of Advanced Search options in HotBot diaries.

List as many of these trends as you can find. Some trends may be obvious, and your confidence that they represent an actual behavioral pattern may be high. Others may be more difficult to pick out (do people use HotBot Advanced Search options more by the 12th week than they did at the 6th week? Maybe, but it's hard to tell for sure). You should label the various trends ("The Third-Week Effect") in order to differentiate them and discuss them later.

Make Hypotheses

Once you have your list of trends, it's time to try to understand their causes. Since this isn't pure science (it's highly applied), certainty in the absolute truth of your hypotheses doesn't have to be complete. In order to make product development decisions, it's often enough to make guesses that are pretty likely to be right.

Thus, it's time to make hypotheses. Why does Lee's discussion of advanced options seem to grow with time? It could be the fact that he becomes more comfortable with the options the more he uses them. It could also be because he's searching more. A quick check of his diaries can show which is the case. If it is because he's growing more comfortable with the options, then that could point to a learning curve that could be researched by looking at the other diaries.

If you couple your diary study with follow-up focus groups or interviews, you can then check your hypotheses against people's actual experiences. Working together, you and the diarists can isolate the most important parts of their experience with the product.

Self-Reporting

When you ask people to report on their own experiences, you ask them to step outside their normal perception of themselves and comment on their behavior. Some people have no problem with this, providing accurate, honest answers. Others have more difficulty. Even if they want to help you and think that they're telling you everything, they may not feel comfortable admitting failure or revealing that they don't know something. This needs to be taken into account when reading anyself-reported information such as support comments, survey questions, or interview responses, but it's especially important in diaries. Often the diary entry is your only contact with the person, and there's no way to know the objective reality.

As in journalism, self-reported actions should be independently verified whenever possible. Ask for clarification of important statements. Follow up with additional interviews or research. If you're reallyambitious, compare one diary entry to another from a different day to see if the reported behavior matches. And always keep a grain of salt handy.