Usability Testing Strategies

People have written entire books on testing. My goal here isn’t to teach you how to test software. Instead, my goal is to teach you how to test for usability bugs. Of course, before you can test for usability bugs, you have to know what usability bugs are. And to figure out what usability bugs are, read this book, and read as many other books and articles on usability as you can. But don’t just take our word for it; talk to the customers using your previous software versions; listen to the beta testers; listen to the tech writers when they have a gripe; listen to everybody. In other words, get the thoughts of the people.

In addition to listening to others, however, you (or your testers) can search for usability bugs yourself. In the following sections, I talk about ways to do testing that is relevant to usability.

Searching for Usability Bugs

How exactly do you search for usability bugs? Here are three separate areas the QA team needs to explore:

• Bugs that the original design never covered

• Bugs that violate the original design

• Bugs in the original design

For example, sometimes programmers (or an entire team of programmers) might end up going off on a tangent, adding some new feature that wasn’t mentioned in the original design documents. They might, for example, add some fancy export feature, allowing the program to export data to Microsoft Excel. While this might be a good feature, the fact is, it wasn’t in the final design documents and therefore shouldn’t be in the program. What should the testers do? They should not test out the feature, making sure it creates valid Excel files. Instead, they should write up a bug report stating that an extraneous feature is present and must be removed.

As another example, the design documents might clearly state a set of menu items under the View menu, but the programmer didn’t include them all, and the programmer added a few extras or perhaps simply renamed them. The testers should catch this problem. Instead of testing the new menu items, the testers need to file a bug report stating that the menu is not laid out as expected.

In other words, if the software engineers and architects and all those nice people did their job, they planned for usability early on. They designed the software to be highly useable. But the programmers who wrote the code might have diverged from the original designs.

Such divergences from the original design would be no surprise. Software development houses all too often give too much freedom to a programmer’s whim of the day. As I mentioned earlier in the book, a programmer might arbitrarily declare that the product can open only 10 documents at once, simply because 10 seemed like a nice number. Similarly, a programmer might be given the chore of designing a dialog box. The engineering documents might have simply said, “We will create an Options dialog box,” and left the details up to the programmer. (This happens all too often!) These bugs would fall into the first category, bugs that the original design never covered.

But on the other hand, a programmer might ignore the design document. The design document might have presented a very nice dialog box that is highly useable. But the programmer might decide that it is flawed and then go on her own way, designing it completely differently. While the people who designed the dialog box on paper or in a design document had the best of intentions, the programmer is the one who ultimately builds the thing.

In both cases, this is where the QA team comes in. Somebody must catch these problems. The architects and team leaders and all those people who should know better might not catch the problems, because their job isn’t to go through and test every last feature of the product. They probably don’t have time. But the test engineers do have time, because testing every feature is indeed their job responsibility. (And I mean every feature. That’s what I talk about in the next section, “Doing a ‘Full Pass’ Test on Your Software.”)

The third category, bugs in the original design, might create the biggest political problems in the whole development process, because declaring a bug in the original design might not exactly make the designers happy. The original designers might yell things like, “Oh, come on, if the users can’t figure this out, they must be stupid!” And that’s why the QA team needs to have a certain amount of showstopping authority. The QA team must be given the authority to say no to such problems and to demand that they be fixed. And that probably means one of the top people in the organization must give the QA team such authority.

So how exactly do the people in the QA team find the usability bugs? They do a full pass on the software, which I discuss in the next section.

REAL WORLD SCENARIO: The Microwave That Reminds You (and Reminds You, and Reminds You…)

“Okay! I get the point! Now leave me alone!”

That’s how I feel when I’m dealing with my microwave oven. Over the past few years, I have moved a lot for various reasons. Because of my mobile tendencies, a couple years ago when it came time to buy a new microwave oven, I based my decision on one factor: I bought the lightestone that was within the size range I wanted. And fortunately, the lightest one was also a pretty good one: it had a rotating platter and that ever-so-important popcorn button, among other features.

But the microwave oven also has a “feature” that is, to be honest, seriously annoying. When I cook something in the oven and walk away, I listen for the beep to tell me when the cooking is done. But if I don’t return to the oven and open the door, the microwave will beep again in 30 seconds. Okay, that’s fine; my mother used to forget about the coffee mug full of water she was warming in the microwave to make instant coffee, and other family members would find it six hours later. (I can say this because she doesn’t read my books. She just puts them on the bookshelf and brags to her friends about them!) Because some people forget about the stuff they just heated in the microwave, I can understand the need for the 30-second reminder.

But this isn’t just a 30-second reminder. It’s a 30-, 60-, 90-, 120-, 150-, 180-, 210-second reminder, and probably more. (But I wouldn’t know. By the sixth or seventh beep, my blood pressure is so high that you can be sure I have removed the food from the oven!)

Maybe some microwave oven designers are reading this book. If so:

Dear Mr./Ms. Microwave Designer:

How do you turn off that darn reminder beep???? I’m considering donating your microwave oven to charity and buying a different brand of oven, except that I care about the poor people and don’t want to curse them with this abomination that you call a microwave oven. Help!

Irritably yours,

Jeff Cogswell

As best as I can figure, the microwave has no way to turn off this warning. I think next time I’ll buy a different brand of microwave oven. The other brands don’t have this problem. And I don’t care if the one I get does weigh a bit more!

Doing a “Full Pass” Test on Your Software

Test engineers sometimes speak of a “full pass” in their test. A full pass is where they use an automation program, and they test every single feature of the software, pushing in as many combinations and permutations as possible. If they are testing a program that allows the user to type an e-mail address into an edit control, they write a test script that will grind through all sorts of possible entries for the edit control, including several examples of text that is not a valid e-mail address, as well as text that is a valid e-mail address. And with the help of automation, they can grind through thousands of possibilities in minutes. They can then test the response of the software to determine whether the software responded correctly. Did the entry of each invalid e-mail address cause the software to issue a friendly message or did some get through? That’s what the test engineers must determine. But such a test is not a test of usability.

Where does the usability test come in? The usability test is determining whether the software’s response to an invalid e-mail address was, in fact, polite and easy for the users to handle. That, of course, requires a lot more subjectivity and less automation. The automation software can’t parse the error message and determine whether some people will be offended by it.

But the automation software can (or at least should be able to) test for some items. For example, the automation software should be able to detect a system modal box. The automation software should be able to spot cascading dialog boxes. The automation software should be able to spot misspellings in the window text. And if not, then either the QA engineers need to write better scripts or they need to find better automation software. (Ahhh, you have to appreciate the irony of test software that itself isn’t highly useable because its QA team didn’t do a usability test on it!)

As for the usability tests that can’t be automated, the only real way is for a real, live human being to sit down and go through every single feature of the software, keeping an eye out for usability problems. Doing so is a full pass of usability.

But how can the test engineer realistically do a full pass? He needs a game plan, a written document guiding him through it. Further, he needs to enlist the help of his trusty automation utility to help him make sure he goes through all the features, one by one. And creating this game plan will typically require the help of multiple people, just to ensure that every feature is covered. The test engineer could then check off each feature, noting any problems with usability. And if he encounters any problems, he can then submit a bug report, assigning a severity level.

And all this, of course, requires that the test engineer and the team that helped create the test are all familiar with usability standards. In other words, test engineers need to read all the same usability books (such as the one in your hands) that the designers read! (If you’re in charge of a software company, that means you need to order an extra hundred copies of this book you’re holding. Everybody needs a copy, and I wouldn’t mind having a bestseller on my hands!) Also, remember that a usability test doesn’t just surround the GUI. I group other issues in with usability, as I mentioned in Part II of this book. For example, if some function in the software runs in a tight loop and causes the CPU usage to fly up to 100 percent, the QA team should catch this.

Performing a Consumer Usability Test on Your Software

Try as you might, the only real way to find out how consumers will respond to your software is to let the consumers try it. Test engineers have a Catch-22 on their hands. They want to put themselves in the shoes of the consumers, yet at the same time, the company needs them to be engineers with a strong computer background. And unless you’re designing tools for computer people, that is a contradiction, or a Catch-22.

Some people like to use the term beta test when they issue an early release for people to try out. I prefer the term consumer usability test. The reason is, they hope that the beta testers won’t encounter any traditional bugs. Or, if they do, they hope that the QA team already knows about them. Yet, traditional bugs are all too often what software shops are hoping that the beta testers will find. Instead, you want your consumer testers to spot and report usability problems in addition to the few traditional bugs that might slip past QA.

You don’t want the consumer testers to be readers of this book. (Of course, if they want to buy this book, please don’t stop them. Instead, find different testers. I would never tell somebody not to buy one of my books or any of my software!) Remember, you want your consumer testers to be a good cross-section of your consumer base. You want the testers to consist of typical people who will be using your product.

One good way for them to test is to give them a simple job: Tell them to determine how easy the product is to use. Ask them to be on the lookout for any part of the software that’s difficult to use, confusing, or downright annoying. And have them determine whether the software is a good neighbor. That is, does the computer still run well when your software is running alongside other software packages? Does your software get in the way of the other programs?

Note

However, I do offer such advice with a slight reservation: Some software that, for example, models a complex business process might be a bit complex to use at first, but after somebody uses the software a couple dozen times or more, it might become easy to use.

All these are basic questions and not technical. If you find yourself asking them to run a profiler on the software and by some miracle they actually do it, then you have picked the wrong consumer testers and are asking for the wrong tests. These technical tests are the ones the QA team should be doing. Instead:

RULE

Think nontechnical when assigning consumer usability tests! Don’t require the consumers to have any technical knowledge. In fact, the best case is that they don’t.

And Finally, Remaining Open-Minded to the Test Results

Nobody likes criticism, especially when the criticism is not constructive. Yet, when you send out your software for a consumer test, criticism is exactly what you’re likely to receive, and some of it might be worded in an angry way. But take it seriously! These are the people who really are going to be using your software. Remember:

RULE

One meaning of highly useable is that the software doesn’t anger people!

The QA team, of course, shouldn’t be getting angry. But similarly, you, as a software developer, shouldn’t get angry when you hear back from either the QA team or the consumer testers. (Okay, I’ll allow you to get a little angry at the QA team on occasion. That’s to be expected sometimes, because they’re your coworkers.)

Therefore:

RULE

Remain open-minded about whatever comments the consumer testers have. If they hate a certain part of your program, take their concerns seriously!

You want the best possible software on the planet, right? And you don’t want the consumers to get angry, toss your software in the trash, buy the competition, and then tell all their friends to avoid your software, do you?

Now realistically, you’re going to get some complaints that are not well founded. You’re going to hear from users who are actually having a problem with the operating system but think your software is at fault. You’re going to encounter users who need to learn what a mouse is and how to click the button. That’s to be expected, and you can handle such complaints however you feel is best. But if you get a call such as this:

When I opened the document a second time by mistake, and then made changes, I lost all my previous changes!

Don’t respond with, “Well, don’t do that, you moron.” Instead, consider whether your software needs some kind of safeguard to prevent two openings of the same document from clashing. Sit back, relax, and put yourself in their shoes. They have their job to do, and they’re using your software to help them do that job. Your software isn’t their job; it’s your job. They have some other job where the software plays just a minor role. I’m writing a book right now, for example, and that’s my work. My work isn’t Microsoft Word. But I spend an awful lot of time staring at Microsoft Word, and the last thing I want is for the software to mess up. And if it does mess up, I expect Microsoft to be understanding of my concerns when I call them. (And you can be sure that I will contact them!) And will they be open-minded to my concerns? Let’s hope so.

REAL WORLD SCENARIO: Didn’t Work? Try It Again! And Again! And Again!

The other day I witnessed something strange, and this certainly wasn’t the first time that I’d seen this happen. I was next in line at the checkout of a grocery store. The register ran out of tape when the cashier tried to print the receipt for the customer ahead of me. The cashier walked away, and she returned with a new roll of tape, which she quickly fed into the register. She then typed a three-key sequence into the register. The register responded with an error beep. Frustrated, she typed the same three keys again. Same error beep. So she typed the same three keys yet again. And again, the same error beep happened. She did this several times, the same three keys each time, followed by the same error beep. Now, as a programmer, I was a bit astounded that the woman somehow believed that by her typing the same key sequence over and over, the computer inside the register would somehow finally get it right and do what it was supposed to do. Of course, computers don’t work that way. I watched her try to type in the same sequence no less than 10 times, each time yielding the same error beep.

I then switched to a different checkout lane, and I never got to see how the story was resolved. But on the drive home, I was thinking about what happened. We computer people know that if you feed a computer the exact same data over and over, the computer will usually respond with the same output each time. If you type the wrong password over and over, the computer won’t finally figure that you’re serious and let you in anyway. So why did this woman somehow think the register would sooner or later accept the keystrokes? Did she not understand computers?

And that’s when it hit me: Think about a car. Suppose you go out and try to start your car, and it just won’t go. The engine cranks, but it doesn’t want to start up. What do you do? You try again and again, until you finally get the thing started (as long as you don’t drain the battery). Most objects work this way. If you turn on a fluorescent light, and you see a faint glow but it doesn’t start, you flick the switch off and turn it on again until the light comes on. If you’re using a lighter to start your gas grill, and the lighter doesn’t start, you flick it a few more times until it finally starts.

Computers, however, don’t work this way. Clearly, this woman was typing the wrong keystrokes. But that didn’t occur to her (until the manager probably came over to help her, I imagine). Instead, she simply thought that the computer just wasn’t taking the seemingly correct data, and sooner or later the computer would accept it. In other words, she was basing her understanding of computers on everyday devices that are very different from computers.

Remember this story when programming computers. They aren’t like other devices. What was the solution? Whoever wrote the program for this computerized cash register could have done this poor woman a big favor by displaying an error message such as, “I don’t recognize the numbers 5 7 9” (or whatever keys she typed in). Now the computer, of course, didn’t know what she was trying to do (or it could have done what she wanted, not what she said!), so it couldn’t say, “That’s not the correct keystroke to reset the roll of paper.” But it could have displayed some message that was more descriptive than an insulting beep. Don’t just have your software beep, please. And remember, if for some reason you insist on having your software beep, allow the user to turn it off. I was taking some grad courses at a university recently, and I was using one of the well-known mathematical software packages. I had my laptop with me in the library, and just to be safe, I had the Windows volume controls set to mute. I was in a library, after all. But you know what? When I did something wrong, this crazy program made the computer beep! This beep came through some internal speaker separate from the stereo speakers and it was loud. Everybody turned and glared at me. I apologized and went into the math program’s configuration to turn off all sounds. Well, those sounds in the configuration turned off the wave files but not the beep! I thought all was fine and five minutes later…BEEP! One young woman turned and glared at me, closed her book, gathered her stuff, and walked to the other end of the room and sat down, resuming her studies. Thanks, software. How embarrassing.

Remember, then: Don’t make your software just beep in error. Give a message that’s a bit more descriptive. But if you feel some inner need to provide a beep, allow me to disable it, please.