Not All Bugs Are Created Equal | Lessons Learned in Software Testing

You would probably agree that a bug that corrupts a user's data is more severe than one that's a simple misspelling. But, what if the data corruption can occur only in such a very rare instance that no user is ever likely to see it and the misspelling causes every user to have problems installing the software? Which is more important to fix? The decisions become more difficult.

Of course, if every project had infinite time, both problems would be fixed, but that's never the case. As you learned earlier in this chapter, trade-offs must be made and risks must be taken in every software project to decide what bugs to fix and what bugs not to fix or to postpone to a later release of the software.

When you report your bugs, you'll most often have a say in what should happen to them. You'll classify your bugs and identify in a short, concise way what their impact is. The common method for doing this is to give your bugs a severity and a priority level. Of course, the specifics of the method vary among companies, but the general concept is the same:

Severity indicates how bad the bug is; the likelihood and the degree of impact when the user encounters the bug.
Priority indicates how much emphasis should be placed on fixing the bug and the urgency of making the fix.

The following list of common classification of severity and priority should help you better understand the difference between the two. Keep in mind, these are just examples. Some companies use up to ten levels and others use just three. No matter how many levels are used, though, the goals are the same.

Severity

System crash, data loss, data corruption, security breach
Operational error, wrong result, loss of functionality
Minor problem, misspelling, UI layout, rare occurrence
Suggestion

Priority

Immediate fix, blocks further testing, very visible
Must fix before the product is released
Should fix when time permits
Would like to fix but the product can be released as is

A data corruption bug that happens very rarely might be classified as Severity 1, Priority 3. A misspelling in the setup instructions that causes users to phone in for help might be classified as Severity 3, Priority 2.

What about a release of the software for testing that crashes as soon as you start it up? Probably Severity 1, Priority 1. If you think a button should be moved a little further down on the page you might classify it as Severity 4, Priority 4.

As you learned in the discussion of the DREAD formula in Chapter 13, "Testing for Software Security," security issues can be difficult to classify. A specific vulnerability could be very hard to expose but, if it is, could allow hackers access to the information in millions of personal accounts. That would most likely be a Severity 1, Priority 1 bug.

Severity and Priority are vital pieces of information to the person or team reviewing the bug reports and deciding what bugs should be fixed and in what order. If a programmer has 25 bugs assigned to him, he should probably start working on the Priority 1's first, instead of just fixing the easiest ones. Similarly, two project managersone working on game software and another on a heart monitorwould use this same information but could make different decisions based on it. One would likely choose to make the software look the best and run the fastest; the other would choose to make the software as reliable as possible. The severity and priority information is what they would use to make these decisions. You'll see later in this chapter how these fields are used in a real bug-tracking system.

NOTE

A bug's priority can change over the course of a project. A bug that you originally labeled as Priority 2 could be changed to Level 4 as time starts to run out and the software release date looms. If you're the software tester who found the bug, you need to continually monitor the bug's status to make sure that you agree with any changes made to it and to provide further test data or persuasion to get it fixed.