Why You Need a QA Team (and What They Should Do)

The QA team has a big job on their shoulders. They need to make sure that the software does not get out the door without any bugs. (Yikes.) But having bugs doesn’t just mean that the software explodes the monitor and wipes out the hard drive while posting the user’s Yahoo! e-mail password on the Web. (Sakes, if your software does all that by mistake, maybe it’s time to find a new career.) In addition to catching what we traditionally think of as bugs, the QA team must catch any usability bugs. What are usability bugs? Exactly what you’re trying to prevent by reading this book. A usability bug is any problem in usability. If you create highly useable software, then your software is free of usability bugs. What are usability bugs? In the section “Searching for Usability Bugs” later in this chapter, I list some examples; here are a few more concrete examples of usability bugs:

• The macro recording shortcut system functions as it is expected, but it’s cumbersome to use: The icons on the buttons on the macro recording toolbar are not clear. Also, when the user clicks the Stop button, the Stop button doesn’t become disabled, and the toolbar gives the appearance that the macro is still being recorded.

• When the user is highlighting text by holding down the Shift key and pressing the arrow keys, and she backs up, the text doesn’t become un-highlighted in the usual way. Instead, the highlighting expands in the other direction, which is not standard. This will confuse users.

These are bugs in the software! Even though the features may function as the designers expected, they do not function as the users might expect.

Of course, you don’t want your software to reach the point of QA only to find out that the software is free of regular bugs but totally unuseable. That’s why the programmers need to read this book, not just the QA people. (Usability engineering takes place throughout the development cycle!) If a QA team spends hours upon hours making sure that the software functions as expected and without bugs but does not test for usability problems, and the software is filled with usability problems, then what’s the point in even bothering with QA? Just because software is bug-free doesn’t mean the software isn’t a piece of junk.

The QA team, therefore, needs to be mindful of usability. While testing for the usual kinds of bugs such as memory leaks and general protection faults, they need to also watch for usability bugs.

But these usability bugs they’re going to find are not big things like the presence of cascading dialog boxes. (Those types of usability problems should not even have existed because the developers did their homework and knew better than to put them in in the first place. Right?) Instead, the kinds of bugs the QA team is looking for are the smaller, less-obvious usability bugs. (I talk about the different kinds of usability bugs and ways of tracking them down in the section “Searching for Usability Bugs” later in this chapter.)

Typically, the QA team will probably want to work usability into their schedule of tests. And the managers will want to make sure that the whole QA team is constantly on the lookout for usability problems even when testing for regular bugs.

Why Your QA Department Should Not Be a Stepping-stone to Development

Want to make a new hire angry? Advertise for a software engineer position and interview candidates, pick the best, and when the new hire arrives for the first day of work, tell her, “You will be working in system testing for six months while you learn how our product works.”

If you do that, you will find that you have huge employee turnover. But besides the obvious part of making somebody angry, you’re completely undermining the whole notion of system testing. Why fill such an important team with disgruntled, frustrated people?

Too many companies, especially dot-coms, have made the QA department a stepping-stone to programming. In addition to making programmers angry and creating a QA team that might not live up to its charter, you are also attaching a dangerous stigma to the QA team, resulting in a dangerous caste system within your organization. As a few of the brighter people get “promoted out” of QA, those who remain will start to feel like they have QA written on their shirts in huge scarlet letters. Morale will drop. And bad morale in one team will bleed over into other teams. (Think about it: How many times have you worked for a company that has a terrible morale problem, and if you look more closely you can trace the problem to one small group of people?)

QA is a vital part of an organization. Treat the QA engineers as equals to the people with the title software engineer: Pay both sides the same, and make it clear that they are on the same level. Invite people to make a lateral move from development to QA, with the same pay rate, which is an opposite move from what most people expect.

Who Should Work in QA?

If you read the first paragraph of this chapter, you heard me point out that the QA people are, indeed, engineers. These are not just partially skilled people who know a bit about computers. Instead, you should staff your QA department with highly skilled people who know how to program.

Imagine a bug report like this:

 Test Engineer: Nikki Ward  Assigned to: Jeff Cogswell  Short Description: Illegal Instruction Error  Long Description: When I opened a file, the program  gave me an illegal instruction error. I can reproduce  it by opening the attached file. However, the problem  occurs only on my computer, not Sue’s computer. 

Okay, if you’re the developer who gets assigned this report, you’ll have to open the attached file. And when the bug doesn’t occur, what do you do? Well, you’ll probably have to walk down the hallway to the tester’s computer, try it out, and see the bug occur. Then what? Does the QA computer have development tools on it? Maybe, if the tester is so inclined to install them. But what if it does not? Then what do you do? You could spend the better part of the day trying to reproduce the problem on your own computer. Then when your manager is coming down on you, you may eventually just flag the bug as “cannot reproduce” and move on, leaving it in place.

But suppose instead you receive the following report:

 Test Engineer: Nikki Ward  Assigned to: Jeff Cogswell  Short Description: Illegal Instruction Error  Long Description: When I opened a file, the program  gave me an illegal instruction error. I can reproduce  it by opening the attached file. However, the problem  occurs only on my computer, not Sue’s computer. I ran  the product inside the debugger and discovered that the  problem is a memory overwrite in somefile.cpp, inside  the function somefunction. It seems that the index  of the array is larger than the array itself. 

Well, that’s a lot better information! Memory problems are, after all, some of the hardest bugs to catch, because the results can be so sporadic. If you accidentally write to some memory outside of an array, your program might be fine. Or it might not be, depending on how that memory outside the array is used. And now, thanks to Nikki’s fine bug work (and well-written bug report), you can go into the actual function and figure out why the index is out of bounds. You can then start backtracking, figure out where the real bug is causing the index to get messed up, and so on.

Now because the tester had the development tools on her computer, as soon as the problem occurred she was able to immediately launch the debugger and go from there. Maybe 15 minutes later, she had the problem isolated and then sent in an excellent bug report. You, in turn, spent another half-hour tracking down the rest of the memory problem and fixing it. What am I saying here, then? I’m saying:

RULE

Your testers should be reasonably proficient in programming.

They don’t need to be experts in software development. (Instead, they should be experts in test engineering or at least spending their time becoming experts.) But they should be reasonably proficient in programming. They might not need to know how to derive a template class from a non-template class, but they should know how to use a debugger to trace into a template class function (and not be intimidated by the fact that it’s a template), set breakpoints, inspect the variables, and the like.

They are not, after all, writing code. But they are reading code. If you have studied a foreign language, you know that most foreign language departments at universities have a watered-down version of the studies with an emphasis on reading. For example, a German department might offer a track of classes for learning to speak German. And separately, they might offer Reading German, which doesn’t focus on memorizing all the genders of the nouns and such. Instead, it focuses on being able to read and understand German.

Likewise, the QA team’s knowledge of programming need only be analogous to the Reading German class. The QA team members are, of course, free to learn as much as they can about programming, and indeed, the more they know, the better.

But requiring less programming knowledge does not imply that the testers know less than the programmers. A QA position is not a step below programming. The QA team doesn’t write code, so how can it be on the same advancement ladder as programming? The QA team is a separate career path with its own skill set. Being reasonably proficient in programming is only one job requirement out of many. A test engineer must also be able to do several things programmers don’t do on a regular basis, such as devise an entire testing plan that puts the program through the wringer, testing as many combinations as possible. Thus:

SUGGESTION

A tester’s expected knowledge should not be a subset of a programmer’s expected knowledge. That is, a programmer isn’t a tester who simply “knows more” and has therefore “moved up.”

From my own experience, I think every QA team should have at least one person who either has made a lateral move from development or has been promoted from development. This person isn’t somebody who just couldn’t hold his weight in the development team and out of management frustration got pushed over, kicking and screaming, into testing. This person should instead be a skilled programmer who can easily go into the code and track down problems quickly and efficiently. This person also needs to have engineering skills to be able to create and execute an entire testing plan. But also, lest we lose sight of the focus of this book, the test engineer must be able to test the software from a usability perspective. This often requires being able to stand in the shoes of the average user. Can a typical programmer do that? Usually not. Most programmers are so advanced in computer usage that they have lost sight of what it’s like to be a beginner.

RULE

A good test engineer should be able to stand in the shoes of the end user.

QA and Showstoppers

Do you remember the TV commercial from several years ago by the Hanes company where the woman who was supposedly a QA inspector said, “They don’t say Hanes until I say they say Hanes!”? Well, that’s the way you should run your software organization. Your QA people should be allowed to stop the show when they find a bug that they deem too severe.

A lot of people in the software world simply accept that some bugs will go out in their software. (I have heard lots of mixed opinions about this idea, but the fact is, it’s a common practice.) But some bugs should not be allowed to get out the door.

Certainly, if the latest build causes the software to crash when it first starts up, the software can’t go out the door. That’s a showstopper. But what about usability bugs? If the software functions as is but has some problems with usability, should the QA team be allowed to call a showstopper and demand that the bug be fixed?

To be honest, I just can’t imagine this happening, as much as I’d like it to. I have been in the software business long enough that I imagine most software managers and VPs simply won’t do it. They have a serious deadline that’s already been pushed back three times, and the last thing they want is a tester calling a showstopper because the GUI might confuse some poor sap who doesn’t know how to use computers anyway.

First, let me say that that’s not the only kind of bug I’m talking about here. Usability extends far beyond catering to the neophytes. Second, as much as managers and VPs might resist, it is my hope that they will get their hands on this book and see the light: You simply must treat usability bugs as legitimate bugs.

Of course, testers typically assign a severity level to a bug. A “sev-1” bug (as most shops call it) is a showstopper. A “sev-4” bug is one they plan to never fix, even when that one guy in Upper Zambobia who happens to be a SQL guru and is using your software for pure enjoyment during his snowbound winter runs into the bug.

The question is, then, what usability bugs qualify as sev-1 bugs, or showstoppers? As much as I’d like to suggest that they all should be sev-1 bugs, the reality is they aren’t. Some are worse than others.

I can’t simply list all the types of bugs here and assign severity levels for you. Instead, the QA staff needs to be so accustomed to usability issues that they can gauge just how bad the different bugs are. Here are some examples:

• Did a programmer fail to follow the designs and create a cascading dialog box somewhere deep in the program? I’d say this is a sev-2, maybe sev-3, depending on how often people are likely to use the particular dialog box. It’s not a showstopper. But it should be looked at by a programmer and considered for repair.

• Did the programmer open a system-modal dialog box that freezes up the whole computer? I’d call that a sev-1, or at least a sev-2. Nah, make it a sev-1. A system modal dialog box is something no program should ever do.

• Did a programmer include a task that waits for a long timeout under certain conditions (such as waiting for a dial tone on the modem when no phone cable is connected) but with no way to cancel the operation? A bug like this isn’t going to be the end of the world, but it will definitely be an annoyance to a lot of users. Choose either sev-1 or sev-2, depending on how much you care about annoying your users. (Personally, I’d make it a sev-1, but I’m obsessed with usability, if that’s not obvious.)

Are you getting the idea here? Showstoppers are problems that are simply unacceptable, usability errors that no software should ever commit. These are bugs that seriously hamper the users’ ability to achieve their goals. Sev-2 bugs are those that really should be fixed, but you don’t have to call in the programmers at midnight. They’re bugs that may hinder the usability of your software and might cause the users to employ time-consuming workarounds. Sev-3 bugs and sev-4 bugs are those that would be really nice to fix; they’re typically cosmetic bugs or minor performance bugs.

Now if you’re hoping that I’m going to come up with an example of a sev-4 usability bug, forget it. I can’t think of any, because I personally feel all usability bugs are at least a sev-2. Others may disagree. But it’s your choice: Do you want to have customers get frustrated and choose the competing software?

Here’s a very good example: A friend of mine does a lot of image editing for her eBay auctions, which she does full time. She was looking for a new image-editing software package. She found a trial version of one that was quite good, except it didn’t allow a multiple-level undo, and the undo it did offer was extremely limited. (It would undo an entire batch of commands if the commands were all the same kind, such as floodfill.) But she needed a better undo. Sometimes she made multiple changes to an image and wanted to backtrack through several to compare and then redo the changes to compare again.

REAL WORLD SCENARIO: Help, I’ve Fallen into Development and I Can’t Get Out!

On a personal note, tracking down problems in code and fixing them was always one of my strengths. Yet, because I was also a skilled developer, the one time back in the mid ’90s when I told my boss I wanted to move to testing, he wouldn’t allow me to. “We need you in development,” he told me. That was a big mistake on his part. If you’re a manager and you have a programmer who wants to move to QA, let him go over!

Now you might be worried about two things: First, if the grass looks greener over in QA and the pay scale is the same, you could end up with a brain drain where you lose all your good programmers. Second, what if your star programmer wants to go over to testing?

As for the brain drain problem, trust me, you won’t have that happen. Most people who know how to program prefer to program. But if they really don’t want to program, then why force them to stay? You need sharp minds in all your groups, not just development. So don’t worry about brain drain; it’s a moot point.

But what if your star programmer wants to leave the group? My advice is to move him over to testing. If that will make him happy, then let it be! The reason is that if all your programming talent is confined to a single person, then you have a serious problem on your hands that’s much bigger than simply whether this one person wants to leave. What if the star programmer gets a job offer across town and resigns? Or, heaven forbid, what if the star programmer dies? (It can and does happen, after all.) Then what? Do you close up shop, congratulate everybody on the fun time you’ve had, and retire? Certainly not! You simply cannot allow one member of your development team to be the sole brain while everybody else is simply an appendage to the operation, controlled by the star programmer. Because when (not if, but when) that programmer gets a job offer across town, you’re going to have a serious problem on your hands.

Train your programmers well. Work with them, find mentors, and turn everybody into star programmers. (Start by having them read this book. I’m serious!) Teach them how to design good software, and that way you won’t be relying on a single person who might up and leave.

And when that person does want to leave the development group but stay in the company, count your blessings that this person is still on the payroll, and happily send him over to QA. (Besides, realistically, people will be coming to him on a regular basis for help, and you’ll still have that brain you were afraid of losing.)

Now some people might rank a limited undo as a sev-3 bug or a sev-4 bug and move it out of the bugs and into the “features for next release” category. Well, I hate to break the news to the people who made this particular image editor, but my friend didn’t buy your $100 package and instead went with the competitor. You lost a sale because of this usability bug.

So what level would you assign to this usability bug? I’d make it a sev-1 or sev-2. The software shouldn’t go out with it. But it’s your software and your sales. And maybe you’re competing with my product, and I’ll make sure to include the feature in mine.