Chapter Eight. Usability Testing

You don't hear things that are bad about your company unless you ask. It is easy to hear good tidings, but you have to scratch to get the bad news.

” THOMAS J. WATSON, SR., IBM EXECUTIVE

It's strange ”not to mention exasperating ”to think that a company would go so far as to conceive, design, and manufacture a product without ever ensuring that people can use it, and just as important, that they like using it. But if my personal experience is any indication, it happens with alarming frequency. I can't count the number of times I've used an appliance and thought to myself , "Surely, the people who designed or produced this have never tried to use it."

A case in point: my clock/radio. It looks great. It sounds great. But it has the worst interface imaginable for programming the radio's preset-station buttons . Instead of having users simply hold down a "Set station" button after they've tuned in the preferred station, it requires them to first press an "Enter" button, then press one of the buttons used to jump to another station. Enter? Why in the name of human factors would I "enter" something before I've selected it? Shouldn't I select ”and then enter?

This would only be a minor annoyance were it not for the fact that the radio has no backup battery to protect against a power failure. That means every time my home loses power, I have to rediscover how to reset the radio station buttons. If the designers of this device had to use it themselves for a while, they would surely have encountered the same problems and made appropriate changes.

That's why it's essential for speech-recognition system designers to conduct both usability tests and pilot tests before a system is deployed on a large scale. Essential? Couldn't we just skip the usability test, do a pilot test, and see what kind of feedback we get? No. You can't assume a system works fine just because no one has complained about it. As Thomas Watson, Sr., pointed out, people are unlikely to tell you bad things unless you ask them first. And even if people can use an application, that doesn't necessarily mean that they want to use it. Our goal is to create systems that people actually prefer to use, and the only way to ensure this preference is through usability testing.

Usability testing is sometimes confused with quality assurance (QA), but the two are very different. QA usually measures a product's performance against its specifications. For example, QA on an automobile would ensure that the components function as specified, that the gaps between the doors and the body are within certain tolerances, and so forth. QA testing would not determine whether a vehicle is easy for people to operate , but usability testing would. In a speech application, QA ensures that the appropriate prompts do in fact play at the right times and in the right order. This kind of testing is important, because designers generally shouldn't assume that an application will work to "spec." QA testing can tell us a great deal about a system's functionality. But it can't tell us if the target population for the application can use it ”or will like to use it.