Chapter 1. Introduction: Nothing to Hide

4 Where We Got Our Data

How We Did the Book Study
The "Thinking Aloud" Method
Site-Specific Testing
Sites Tested
Web-Wide Testing
What if a Site Has Changed?

17 Tell Me Again: Why Do I Need to Do User Testing?

User Testing in Three Days
The Exceptions

We'll begin by letting you in on a secret: There are no secrets to doing good usability studies. It's simply a matter of knowing how and where to look, and then documenting your observations.

Usability works because it reveals how the world works. Once you discover how people interact with your design, you can make it better than your competitor's.

This book tells you how people behave on the Internet and what makes Web sites fail or succeed. How do we know this? Because we've spent hundreds of hours looking. If anything, that's our big secret: that we know how to conduct user research. Yet even that is not a secret because we teach a course on how to conduct usability studies correctly.

As much as we try to teach companies to do their own user research and to employ valid methods , however, we keep getting asked for advice on how to make Web sites better. Many people would rather know what works in general than spend time testing their own designs. With so much already known and documented about user behavior on the Web, why not base your design on this general knowledge? Then you can always fine-tune it by conducting research on industry-specific questions that apply to your particular site.

We are writing for sites that are not among the Web's Top Ten most-visited list. It's actually good news if you're not on the Top Ten most-visited list.

Fair enough. That's why we wrote this bookto bring the best of our extensive knowledge on general usability together in one handy place. The observations here are based on issues that we have seen again and again with many different users as they have tried to use many different Web sites. The guidelines translate our empirical observations into general advice about what usually works on the Web. Although there are exceptionswhich is why we advise that you test your own sitethese guidelines apply about 90 percent of the time, and the vast majority of Web sites would be better if they complied with them.

Where We Got Our Data

All of our findings and guidelines are based on empirical evidence from two sources. First we rely on our testing of 716 Web sites with 2,163 users around the world. Most of this research was conducted in the United States, but we also ran sessions in Australia, Belgium, Canada, Denmark, Finland, France, Germany, Israel, Italy, Japan, Hong Kong, Korea, Switzerland, and the United Kingdom. Many of these studies were done for our consulting clients , so the details are confidential. But this vast research also provides general insights into user behavior, particularly when we observe the same findings in highly diverse industries.

Other studies were conducted in the process of writing research reports about special issues. Most of the guidelines from these studies are only important if you are working on the exact problem we researched, but we have also abstracted valuable general insights from the thousands of specific observations in these studies. Thus most of what we say here is based on general lessons from a huge number of Web sites and users, whether they were tested for proprietary projects or for our own studies.

Our second source of information was a special study that we conducted for this book. When we talk about "the study" here, we're referring to this smaller set of data.

How We Did the Book Study

We tested 69 users, 57 in the United States and 12 in the United Kingdom, for this book. Slightly less than half (32) were male and slightly more than half (37) were female , in an even distribution of ages from 20 to 60. Each was paid $100 for participating. We didn't test teenagers or senior citizens for this book, though we occasionally offer insights on these special groups, based on the separate studies we have conducted with them.

The users had a broad range of job backgrounds and Web experience. We screened out anybody working in technology, marketing, Web design, or usability because they rarely represent mainstream users. People who work in the biz know too much and have difficulty engaging with a design as regular users. Instead, they tend to criticize the design based on their personal design philosophy, which is invalid as usage data. In fact, whenever you hear a user throw around insider terminology like "information architecture" in testing, you probably have to discard most of what they say.

All the users had at least one year's experience using the Web. We almost never test people who are completely new to the Web because all we would find is that the Web in general and browsers in particular are difficult user interfaces that take some time to learn. We wouldn't learn much that would help us design better Web sites because completely new users wouldn't get very far into the sites.

The experience of brand-new users is generally not that important for Web sites anyway because new users rarely visit one of them on their first Internet expedition. Of course, this is not true if you are Yahoo!, AOL, MSN, or similar Internet service providers that make their sites the default homepage for their customers. However, we are not writing for these few exceptional sites. We are writing for mainstream corporate, e-commerce, news, and government sites, and others that are not among the Web's Top Ten most-visited list. It's actually good news if you're not on the Top Ten list because when users visit your site, they already will have learned the basics elsewhere of how to use the Web.

The standard rule for user testing is to employ the equipment that most users are likely to have. For this study, we tested on a Windows machine, running the latest version of Internet Explorer. The monitor had a screen resolution of 1024 by 768 pixels. For Internet connectivity, we used a broadband connection: Depending on the test location, it ranged from 1 to 3 megabits per second (Mbps).

The "Thinking Aloud " Method

In our studies, users are tested one at a time so that they don't bias each other. In each session, the test user sits by a computer while the test facilitator, and sometimes one or two additional observers, sits nearby. If there are many observers, it's better to test in a usability laboratory with two rooms that are separated by a one-way mirror so that the observers can be hidden from view. But with a small number of observers, it works equally well to have them sit behind the user so that they are out of sight and therefore out of mind.

Hearing a user's "thoughts" allows us to understand why they do what they do, and this information is invaluable in testing.

The book study was conducted with the "thinking aloud" method, which is our preferred approach for almost all usability tests. In this method, users are asked to think out loud as they work with the interface. Hearing a user's "thoughts" allows us to understand why they do what they do, and this information is invaluable. It's nice to know that users, say, clicked the wrong button and couldn't check out of an e-commerce site. But if you want to improve the checkout process and thus increase sales, you need to know why people click the wrong buttons .

We also made two video recordings of each test session: one of the computer monitor and one of the user's face and upper body. These recordings include a sound track with the user's comments. For most usability projects, you don't really need to review the recordings because the main design problems are obvious after the test sessions. For practicality, it is usually best to quickly fix the problems and increase a company's business as soon as possible. But for a research project like ours, it's good to be able to review them and make sure that we have an accurate record of everything the user did.

Sites Tested: Large Companies

Nestl

Cummins (energy company)

United States Postal Service (USPS)

American Honda Motor Co.

Burger King

Medium-Size Companies

BankOne

Dianon Systems (health diagnostics)

Dime Savings Bank of Williamsburgh

Escalade Sports

House of Blues

Pixar Animation Studios

Small Companies

Black Mountain Bicycles

GW Eye Associates

E-commerce

Atlantis , Paradise Island

Bath & Body Works

Kitchen Etc.

Movies.com

The Sharper Image

Whistler Blackcomb Mountains (ski resort)

Government and Nonprofits

American Heart Association

California State Parks

City of San Diego

J. Paul Getty Trust

Her Majesty's Revenue & Customs (United Kingdom customs authority)

United States Social Security Administration (SSA)

Site-Specific Testing

For part one of the book study, we systematically tested 25 Web sites that cover a range of genresfrom industries like automobiles and financial services to entertainment sites and intellectually oriented medical and cultural sites.

You can see from their homepages that the sites also exhibited a wide variety of design styles, from somewhat primitive to overly glamorous. All in all, our basic goal was to test a good cross-selection of current Web sites.

We didn't pick any site because we wanted to dig into that company's Web strategy. Similarly, our comments on these sites should not be construed as criticism of the companies or teams behind the sites. There are many reasons why a Web site might have a bad design, beginning with a lack of resources. For the purposes of this book, we don't really care why. No matter how understandable the reasons for a mistake, it's still a mistake that our readers should be warned against committing on their Web sites, and that's why we present it here.

Homepages for the 25 sites that were tested systematically in the usability study we conducted for this book. The homepages are shown as they would initially appear in a browser window on a 1024-by-768 monitor.

www.nestle.com

www.cummins.com

www.usps.com

www.honda.com

www.bk.com

www.bankone.com

www.dianon.com

www.dimewill.com

www.escaladesports.com

www.hob.com

www.pixar.com

www.blackmountainbicycles.com

www.drgordonwong.com

www.atlantis.com

www.bathandbodyworks.com

www.kitchenetc.com

www.movies.go.com

www.sharperimage.com

www.whistlerblackcomb.com

www.americanheart.org

www.parks.ca.gov

www.sandiego.gov

www.getty.edu

www.hmrc.gov.uk

www.ssa.gov

It should be noted that the book study was not funded by any of the organizations whose sites we tested. We covered all costs ourselves so that we could be free to speak the truth in reporting on them.

This part of the study was a scavenger hunt of sorts. We gave each user three or four specific tasks to do on each site. While these tasks would not uncover every last usability problem on a site (bigger sites especially offer much more functionality than a few tasks can cover), they were enough for our purposes: to assess how well sites support the most typical goals users have when visiting them.

Some of the tasks we asked users to perform were:

Go to www.usps.com and find out how much it costs to send a postcard to China.
Go to www.sandiego.gov and find the name of the city council member for one area.
You are planning a family reunion at Sugarloaf Ridge, California. Go to www.parks.ca.gov and make a reservation for a campsite that can hold 35 people.
You are looking for a snack that you can eat during your workout. Go to www.nestle.com and see what products they have for you.
Go to www.pixar.com and see if you can find out how they came up with the idea for the movie Monsters, Inc.
You want to visit the J. Paul Getty Museum this weekend . Go to www.getty.edu and find out the cost to enter and the time it opens.
You want to put $1,000 in the bank and keep it there for a long time. Go to www.bankone.com and find the accounts with the best interest rates.
You read an article about how fuel cell technology may change the world. Go to www.cummins.com and find the top two pros and cons of fuel cell technology.

All of these tasks were eminently feasible to do on the Web sites in question. We almost never ask users to "do the impossible " on a particular site. We observe plenty of difficulties just from watching them try to do the tasks that a site is supposed to support, so that's all we test.

Web-Wide Testing

In the site-specific testing, users were told where to go and were expected to stay there while performing their tasks. This is the way most usability studies are conducted, and is great if you want to find out how elements of a particular site's design works. Of course, that's not the way users work in real life. People have the entire Internet at their fingertips and they'll often jump from one Web site to another to complete a task.

In most usability studies, people are told what site to use, but that's not the way users work in real life. So we also gave ours a range of tasks and told them to go anywhere they pleased.

For this reason, in part two of our testing we gave users a range of tasks and told them to go anywhere they pleased. We call these "Web-wide tasks" because users had the entire Web to choose from. These tasks represent a wide range of activities, from highly commercial pursuits to curiosity -based inquiries, and all can realistically be done on the Web today.

The main downside of this approach is that users go to different sites even when working on the same problem, so we didn't get to systematically test those sites with a range of users. The upside is that we get to see how people construct their solutions across multiple sites, as they do when they are not in the lab.

For this section of the testing, we gave each user one of 12 tasks:

You and your family are interested in going on vacation to Mazatlan, Mexico. Find a travel offer that is both appealing and affordable for your family.
You have some extra time during the week and want to do something to help the community. Find a community service program that would be a good fit for you.
Uncle George is interested in getting a personal computer for his home. He's mainly going to use it to surf the Web, e-mail, and print up digital pictures. Find a computer that you would recommend to him.
Your nephew's birthday is coming up. He's going to be five years old and likes reading. Subscribe to a magazine that he might want.
You're thinking about investing $10,000 in a retirement plan. Find the best way to invest your money.
You're planning to purchase a new home and need to finance it. Find a company that will give you the best service and rates.
There's been a lot of news lately about viruses on the Internet. Find the best way to protect your personal computer from viruses.
You're doing a report on Marie Curie. Find out who she is and some of her most famous accomplishments.
What are the three main reasons that Siberian tigers are an endangered species?
Find out what a "let" is in tennis.
In August 2003, the United States and Canada experienced an enormous blackout, leaving 50 million people without power. Find out the main cause of the blackout .
A good friend complains about throbbing pain that usually radiates from one eye to the forehead, temple, and cheek. What might your friend be suffering from?

What if a Site Has Changed?

We include many screen shots in this book because it's much easier to absorb abstract principles when they are illustrated with specific examples. During the year we spent writing the book, however, several of the sites that appear in screen shots here have been redesigned. In some cases, the companies conducted their own usability studies. In others, site representatives attended seminars in which we show video clips of user behavior from recent studies. Thus, even though they didn't know about this book, they still saw previews of our tests of their sitesin effect getting a free usability study!

In either case, this means that if you check out a site after seeing it in this book, it is very likely to look different. Does this make the analysis in this book less relevant to your project? Not at all. The principles and guidelines that a screen shot illustrates are relevant long after a site has changed. In fact, many findings from our user testing in 1994 continue to be seen in studies in 2006 and will probably be found again by unlucky testers in 2020 and beyond.

An example of this is the Web site of the British Broadcasting Corporation (BBC). Jakob and Marie Tahir happened to use bbc.co.uk in Homepage Usability: 50 Websites Deconstructed to illustrate the issue of easy access to archives of previously featured homepage stories. The lack of archives was one of the BBC site's usability problems in 2001 and it continued to be a problem in 2005, as the screen shot shows.

The day before this screen shot was taken, the BBC site featured a great recipe for roast goose with sage and onion stuffing and applesauce. How can you find this recipe? An experienced user might be able to conjure up the correct page through a search, but the average user would be at a complete loss.

www.bbc.co.uk

As the second screen shot shows, the BBC was working on an archive in late 2005. If you visit the site now, the homepage may well feature the archive. If so, the BBC will finally have done what we suggested in 2001. Does that change the importance of the comments in Homepage Usability? It doesn't, because our guideline to provide an archive of homepage feature stories continues to be relevant for millions of other Web sites. Only 41 percent of corporate homepages follow this guideline, so 59 percent could improve their usability today if they pay attention to the mistake the BBC made in 2001.

Beta release of BBC's service to archive all stories that have run on the homepage so that users can find them later: Let's say that you remember seeing a mouthwatering photo of Rick Stein's Christmas goose when you visited the site yesterday , but you didn't have time to read the recipe. Returning a day later, you discover that the BBC has moved on to a story about Franz Ferdinand's favorite music (see previous screen shot). Even if you like this Scottish rock band , they are not going to help you cook your goose. If you happen to know that the URL for BBC's beta-release archive is www.bbc.co.uk/homearchive, you can easily find the story by scrolling down the list of yesterday's features until you recognize the photo. Hopefully, by the time you read this, the BBC will have promoted the homepage archive from a beta test to a regular feature with a link on the homepage itself.