Bug-Tracking Systems

By now it should be clear that the bug-reporting process is a complex beast that requires a great deal of information, a high level of detail, and a fair amount discipline to be effective. Everything you've learned so far in this chapter sounds good on the surface, but to put it into practice requires some type of system that allows you to log the bugs you find and monitor them throughout their life cycle. A bug-tracking system does just that.

The remainder of this chapter will discuss the fundamentals of a bug-tracking system and give you examples of using a paper-based approach and a full-fledged database. Of course what you use will likely be customized and specific to your company or project, but in general, the concepts are consistent across the software industry so you should be able to apply your skills to just about any system you're asked to use.

The Standard: The Test Incident Report

Your good friend, the IEEE 829 Standard for Software Test Documentation, defines a document called the Test Incident Report whose purpose is "to document any event that occurs during the testing process which requires investigation." In short, to log a bug.

Reviewing the standard is a good way to distill what you've learned about the bug-reporting process so far and to see it all put into one place. The following list shows the areas that the standard defines, adapted and updated a bit, to reflect more current terminology.

• Identifier. Specifies an ID that's unique to this bug report that can be used to locate and refer to it.

• Summary. Summarizes the bug into a short, concise statement of fact. References to the software being tested and its version, the associated test procedure, test case, and the test spec should also be included.

• Incident Description. Provides a detailed description of the bug with the following information:

  Date and time

  Tester's name

  Hardware and software configuration used

  Inputs

  Procedure steps

  Expected results

  Actual results

  Attempts to reproduce and description of what was tried

  Other observations or information that may help the programmer locate the bug

• Impact. The severity and priority as well as an indication of impact to the test plan, test specs, test procedures, and test cases.

Manual Bug Reporting and Tracking

The IEEE 829 standard doesn't define the format that the bug report should take, but it does give an example of a simple document. Figure 19.5 shows what such a paper bug report can look like.

Figure 19.5. A sample bug report form shows how the details of a bug can be condensed to a single page of data.

Notice that this one-page form can hold all the information necessary to identify and describe a bug. It also contains fields that you can use to track a bug through its life cycle. Once the form is filed by the tester, it can be assigned to a programmer to be fixed. The programmer has fields where she can enter information regarding the fix, including choices for the possible resolutions. There's also an area where, once the bug is resolved, the tester can supply information about his efforts in retesting and closing out the bug. At the bottom of the form is an area for signaturesin many industries, you put your name on the line to reflect that a bug has been resolved to your satisfaction.

For very small projects, paper forms can work just fine. As recently as the early 1990s, even large, mission-critical projects with thousands of reported bugs used paper forms for bug reporting and tracking. There still may be pockets of this today.

The problem with paper forms is that, well, they're paper, and if you've ever walked into a paper-run office and asked someone to find something, you know how inefficient such a system can be. Think about the complex bug life cycles that can occur (an example of which was shown in Figure 19.3), and you'll wonder how a paper system can work. What if someone wanted to know the status of Bug #6329 or how many Priority 1 bugs were left to fix? Thank goodness for spreadsheets and databases.

Automated Bug Reporting and Tracking

Just as with the test case and test procedure documents described in Chapter 18, "Writing and Tracking Test Cases," there's no reason that the IEEE 829 standard can't be brought up-to-date and adapted to work with modern systems. After all, the information for tracking bugs, the data put on the form shown in Figure 19.5, is just text and numbersa perfect application for a database. Figure 19.6 shows such an automated bug reporting and tracking system that represents the type you might encounter in your work.

Figure 19.6. The main window of a typical bug-reporting database shows what an automated system can provide. (Mantis bug database images courtesy of Dave Ball and HBS International, Inc.)

Figure 19.6 shows a top-level view of a bug database containing 3,263 bugs. The individual bugs, their IDs, titles, status, priority, severity, and resolution are shown in a simple listing at the top third of the screen. Further information about the selected bug entry is then shown on the bottom part of the screen. At a glance you can see who opened the bug, who resolved it, and who closed it. You can also scroll through details that were entered about the bug as it went through its life cycle.

Notice that at the top of the screen there is a series of buttons that you can click to create (open) a new bug or to edit, resolve, close, or reactivate (reopen) an existing bug. The next few pages will show you the windows that appear when you select each option.

Figure 19.7 shows the New Bug dialog box, in which information is entered to log a new bug into the system. The top-level description of the bug includes its title, severity, priority, software version info, and so on. The comment field is where you would enter the details of how the bug was discovered. This database conveniently prefills the comment area with headers that guide you in providing the necessary information. If you're entering a new bug, all you need to do is follow the promptsentering your test's objective, the setup steps, the steps that reproduce the bug, what result you expected, what result you saw, and what hardware and software configurations you were using when you saw the bug.

Figure 19.7. A new bug starts its life cycle in the New Bug dialog box.

Once a bug is entered, and really anytime during its life cycle, new information may need to be added to clarify the description, change the priority or severity, or make other minor tweaks to the data. Figure 19.8 shows the window that provides this functionality.

Figure 19.8. The Edit window allows you to add more information to an existing bug entry.

Notice that this dialog box provides additional data fields over what the new bug window provided. Editing a bug allows you to relate this bug to another one if you find one that seems similar. A programmer can add information about how much progress he's made in fixing the bug and how much longer it will take. There's even a field that can put the bug "on hold," sort of freezing it in its current state in the life cycle.

An important feature shown in Figure 19.8 is in the Comments section. Each time a bug is modified, when it's opened, edited, resolved, and closed, that information is recorded in the comment field. At a glance you can see what states the bug has been through over its life.

Figure 19.9 shows the dialog box used when someone, usually the programmer or project manager, resolves a bug. A drop-down list provides different resolution choices from Fixed to Can't Fix to Duplicate. If the bug is fixed, the buildor version number that will contain the fixis entered, and information about what was fixed or how the fix was made is added to the comment field. The bug is then reassigned to the tester for closing.

Figure 19.9. The Resolving dialog box is typically used by the programmer to record information regarding the bug fix.

Many bug databases track not just comments about the fix, but also details of exactly what the programmers did to make the fix. The line of code, the module, and even the type of error can be recorded as it often provides useful information to the white-box tester.

After a bug is resolved, it's typically assigned back to you, the tester, for closing. Figure 19.10 shows the bug Closing dialog box. Because the database tracked every modification to the bug report since it was opened, you can see the decisions that were made along the way and what exactly was fixed. It's possible that the bug wasn't fixed the way you expected, maybe a similar bug had been found and added by another tester, or maybe the programmer made a comment about the fix being risky. All this information will assist you when you retest the bug to make sure it's fixed. If it turns out that it's not fixed, you simply reopen the bug to start the life cycle over again.

Figure 19.10. A bug report ready for closing has its entire history available for review.

Once you use a real bug-tracking database, you'll wonder how a software project's bugs could ever have been managed on paper. A bug-tracking database provides a central point that an entire project team, not just the testers, can use to communicate the status of the project, tell who's assigned what tasks to perform, and, most importantly, assure that no bug falls through the cracks. It's the culmination of everything you've learned in this chapter about how to report the bugs you find.