Section 7.2. Bug Tracking Tools | Practical Development Environments

7.2. Bug Tracking Tools

There are easily a hundred different tools available for tracking bugs. A good place to start if you want to see a list of choices is the Open Directory's Bug Tracking section at http://dmoz.org/Computers/Software/Configuration_Management/Bug_Tracking. Another good location for a list of tools is http://testingfaqs.org/t-track.html. Yet another list of open source bug tracking tools is http://usefulinc.com/edd/notes/IssueTrackers.

7.2.1. Spreadsheets

The simplest structured way to keep track of a small number of bugs is to use a spreadsheet program, such as Excel or OpenOffice Calc. You could use an ordinary text file or a Wiki, but then it's hard to sort the information. Each row in the spreadsheet is a single bug. Some typical column titles might be Bug Number, Summary, Description, State, Owner, and Priority. Sorting and summarizing by different columns can show the bug counts per person, the highest-priority bugs, and so on. Many spreadsheets will let you restrict the kind of data entered into each fieldfor instance, to make sure that the bug number is unique.

The main advantage of this approach is its simplicity. Spreadsheets are commonly available tools and are reasonably well documented. Basic reports are easy enough to produce. Adding a new field to the definition of a bug is as simple as naming another column.

However, the disadvantages of this approach are significant. Most of these are the same reasons why people use databases rather than spreadsheets. As a project grows, the first problem that is likely to be encountered is that only one person can modify the spreadsheet at a time. Another problem is that, unlike with a good database, if the machine running the spreadsheet crashes during modification of a bug, then the data can be inconsistent. Also, as the number of bugs in the spreadsheet grows into the thousands, the time taken for the different reports can grow significantly.

Using a spreadsheet to keep track of bugs should be viewed as a temporary measure, one that will not scale as a project grows. There are many simple, low-cost bug tracking tools available that will scale better than a spreadsheet. Still, a spreadsheet is better than an email folder of reports of possible bugs or a wall full of Post-it notes, but not by much.

7.2.2. Different Server Technologies

There are many bug tracking systems to choose fromperhaps more than all the available SCM tools and build tools combined. Most of these tools meet a fair number of the requirements and goals for a bug tracking system that were listed in Section 7.1, earlier in this chapter. So how to choose tools for evaluation? Some tools are easy to reject because they're no longer actively developedwhen was their last real release? A useful reference for checking up on this is Danny Faught's Boneyard page (http://www.testingfaqs.org/boneyard.html), which is a list of many corporate changes and dead products in this area. Other tools are easy to reject because you can't justify their cost for the project at hand.

Between these two extremes there are still hundreds of tools, both commercial and free, all of which have a dozen or so customer sites and a few glowing testimonials. My suggestion is that if you have used one of these tools beforeor even better, if you've administered it and it worked for a similar projectthen use it again if at all possible (and if you still can bear it). That way you'll avoid spending time discovering bugs in a new product.

If you are interested in knowing what other people and projects are using, then the next five sections should be helpful to you. The tools in these sections (Bugzilla, GNATS, FogBugz, JIRA, and TestTrack) were chosen because each one is relatively well known and represents a different underlying technology. The different server-side technologies used by each tool are:

Bugzilla: Perl scripts called from a web server
GNATS: A standalone server using C and shell scripts
FogBugz: PHP scripts called from a web server
JIRA: A Java servlet running inside an application server
TestTrack: A separate server closely coupled to a web server

7.2.3. Bugzilla

Bugzilla (http://www.bugzilla.org) was one of the first projects to be released by the Mozilla organization. Written entirely in Perl, Bugzilla is licensed under the open source Mozilla Public License. The version described here is 2.18rc3.

The History of Bugzilla

Bugzilla grew out of the bug tracking systems developed for use inside Netscape in the late 1990s. Terry Weissman wrote the first version in Tcl and then ported it to Perl for Version 2. Since April 2000, the project has been led by Tara Hernandez and then Dave Miller. Several forks have occurred and merged, including the Bugzilla used by Red Hat, and IssueZilla, which was used in CollabNet's preconstructed development environment (see Section 3.1.3).

Bugzilla has all the basic features that you would expect from any bug tracking system, and a good number of more advanced ones. Interaction is via a web browser, and notification is by email. Configuration is done by modifying Perl source files, so the local source of Bugzilla can (and should) be versioned with an SCM tool. Other features include the following:

Regular expressions can be used in text searches. The web page for queries can seem daunting when first encountered, but the defaults mostly just work. More complex queries using Boolean operators are also supported.
Reports are a separate feature from queries and are really quite impressive. Three kinds of information can be chosen, then filtered according to interest, and then displayed as tables, various graphs, or pie charts. Data can be exported as comma-delimited values. The historical reports are also of generally good quality.
LDAP is used to control access rights, so there is no need for a separate set of Bugzilla usernames.
The history of each bug is available; it displays in a table which fields were changed by whom and when, and what the changes were.
Each person can configure to a fine level which types of changes to a bug will trigger an email to them.
Email addresses in bugs can optionally be obfuscated to stop address harvesting on public web sites.
Bugzilla supports people voting for different bugs. Bugs with more votes are considered more irritating to more people.
Links from within text to other bugs, URLs, and email addresses are created automatically.
Adding dependencies between bugs is supported.
Spellchecking is available.
There is some limited support for measuring the time spent working on each bug.
Bugzilla is well localized in at least nine different languages and can use multiple language templates at once.
There are tools to check the configuration and to check that the data in the database makes sense to Bugzilla.

The list of projects and companies that use Bugzilla is impressive and includes some of the largest open source projects. The Apache web server, the Linux kernel, and gcc all have thousands of bugs recorded in their Bugzilla systems. There were discussions within Apache about using other bug trackers after Apache's insecurely configured Bugzilla led to the defacing of the main Apache web site, but better security seems to have meant that Bugzilla is still in use there, though some projects are now moving to JIRA.

Documentation for Bugzilla is sufficient but improving, and the mailing list and other resources are very active due to the large number of users and system administrators. Consulting is offered from numerous sites all over the world. Another useful page about Bugzilla administration is Byron Jones's http://bugzilla.glob.com.au, especially for Windows servers. References for how to integrate CVS with Bugzilla appear in Section 7.4, later in this chapter.

Bugzilla's biggest strength is that it is open source. If you are comfortable with Perl and the Perl ways of doing things, then customization is possible. However, because Bugzilla has grown organically, some of the interfaces are not defined clearly enough to make it easy to maintain your modifications as new versions are released. Also, modifying the workflow by adding new states is not easy at all. That said, the many features that already exist in the newer versions of Bugzilla may well be complete enough so that only minor customizations are necessary for your project. Another approach is to purchase a server already configured with Bugzillafor example, from Bugopolis (http://www.bugopolis.com)or to use a Bugzilla hosting service such as BugHost (http://www.bughost.com).

Bugzilla was developed for Unix-based web servers and has only recently begun to support Windows as a server platform. Likewise, it was written to use MySQL as the underlying database and has only recently begun to support other databases (PostgreSQL is one). Documentation of the database schema is sparse. In general, configuration of Bugzilla is still quite complex, even if you are comfortable with Perl and Unix.

Bugzilla is the most popular open source bug tracking tool available. It's used by a large number of projects and companies and does provide a wide range of features after careful installation and configuration, preferably by someone who is comfortable with Perl. One area still under careful scrutiny is security (http://www.bugzilla.org/security), so make sure you read all the documentation when using it on a public server.

7.2.4. GNATS

GNATS (http://www.gnu.org/software/gnats) is a bug tracking system from the Free Software Foundation and is licensed under the GPL.^[1] It's written mainly in C and shell scripts, and has a long history of extensive customizations by each project that has used it. All interaction with GNATS was originally by Unix shell commands and email, but other interfaces also exist. There is a Web interface (Gnatsweb), an Emacs mode, and a GUI written in Tcl/Tk (TkGnats) that runs on Unix and some older versions of Windows. The source's contrib directory also contains a web interface for administering GNATS itself, which is particularly useful since GNATS comes with a fair number of different configuration files.

^[1] GNATS should not be confused with GNAT, which is the GNU Ada Toolchain project.

The History of GNATS

GNATS (recursively defined as "GNats: A Tracking System") has been in existence since the early 1990s. It was originally written by Heinz G. Seidl, and then for a number of years it was developed by Brendan Kehoe and Jason Merrill at Cygnus, and it's still sometimes referred to as the GNU Problem Report Management System (PRMS). Releases based on the 3.1 source continued even after the 3.2 release. Version 4.0 was a major rewrite by Bob Manson, Milan Zamazal, and Yngve Svendsen; it was released in August 2003. As of early 2005, Chad Walstrom is the maintainer of the project.

GNATS is really intended for use on Unix machines, though the Tcl/Tk, email, and web interfaces do allow it to be used from Windows machines. Releases prior to 4.0 used text files to store the data. Release 4.0 changed the data format to a binary one. By default, the states for bugs are:

Open: Initial state for all bugs.
Analyzed: Work has begun on the bug.
Feedback: A fix is being tested.
Closed: The bug has been fixed.
Suspended: Work on the bug has been suspended.

Large projects and organizations that use GNATS as their bug tracking tool include Juniper Networks and FreeBSD (http://www.freebsd.org/support.html). It was also used by the Apache community until March 2002, when it was replaced with Bugzilla. GCC has also moved from GNATS to Bugzilla.

Documentation for GNATS exists and mostly is correct, though not extensive. Reading the source really is the final documentation for GNATS. The mailing lists are all fairly lowvolume as of early 2005.

GNATS interacts with the lowest common denominators of open source projects: email and browsers. It even works with text-based browsers, such as Lynx. Since many of the fields are simple text, regular expression queries work well. The latest version has made adding new fields easier and supports using separate database instances (for use by different projects, for example).

While GNATS does have an audit trail that can be configured to show the changes in different fields, this is not searchable by default. There is no explicit audit trail for changes to the configuration of the tool itself. Generating historical reports is complicated, though FreeBSD does have some Tcl scripts available to help do this.

GNATS is a project that seems to have grown by fits and starts, as each project using it fed their changes back into the GNATS source code. Even though Version 4.0 removed many of the valid concerns about the speed and index-corruption bugs of earlier versions, a number of projects had already stopped using GNATS, usually changing to Bugzilla, the other open source bug tracking system. Still, GNATS seems adequate for most projects and can be extended in many different ways without too much difficulty. If you like your tools to be Unix-based with a terse command line or to be purely email-oriented, GNATS may be worth further investigation.

7.2.5. FogBugz

FogBugz, from Fog Creek Software (http://www.fogcreek.com/FogBUGZ), was designed by Joel Spolsky, who also writes thought-provoking articles on development environments at http://www.joelonsoftware.com. Interaction with FogBugz is by web browser or email, and it uses the PHP scripting language and a web server (Apache or IIS) to interact with a database. FogBugz is licensed commercially but is released with source code for modification by system administrators. The cost is approximately $100 per user, with support wittily priced at five cents per day (which works out to be $18.25 per year, in case you were wondering) but available only in blocks of a year.

The whole design of FogBugz places a priority on usability over functionality. What this means in practice is that FogBugz is really easy for users to learn and will do 80% of what other, more complicated bug tracking tools will do. Adding new users and changing their preferences is straightforward. Changing the text used as labels for different fields is easy enough, since the PHP files are available. Adding a simple text field is apparently possible but not encouraged, since it leads to too many choices, which discourages people from entering bugs. Adding more complex fields that increase choices, or changing fields' default values or the interactions between fields, is definitely not recommended.

Limiting a tool's functionality takes some confidence in these days of bloated software. The following quote from the FogBugz web site demonstrates this confidence:

FogBugz does not provide personal performance metrics for management, because in the real world, whenever people are measured based on the number of bugs they create (or fix, or report), every bug results in an argument. If you try to penalize programmers for writing buggy code, the only thing you can be certain of is that sooner or later, the number of bugs in the bug tracking database will approach zero, while the number of bugs in the software stays the same. FogBugz is not a crutch for your HR department. (http://www.fogcreek.com/FogBugz/40FogBugzInDepth.html)

Of course, you can still write SQL queries to produce any report you wish for, but FogBugz isn't going to make it easy for you to do so.

The different platforms supported for FogBugz servers are Windows, GNU/Linux, FreeBSD, and Mac OS X. Required software for Windows are some data access libraries and the Microsoft Jet, Microsoft SQL Server, or MySQL databases. Unix versions require Apache, MySQL, and PHP. Another alternative is to pay Fog Creek to host a FogBugz instance for you on one of its servers.

Documentation for FogBugz is clear, and the discussion groups are lively and mostly helpful. Support response is apparently excellent. There is one book about FogBugz so far: Painless Project Management with Fogbugz, by Mike Gunderloy (Apress).

Features of FogBugz beyond those usually found in simple bug tracking systems include the following:

Integration with various SCM tools (including CVS, Subversion, Perforce, and Visual SourceSafe) is a core part of the product.
Full-text searching with Boolean operators and prefix matches is available (though depending on the underlying database, this may only be for the title field).
URLs are created automatically when the string http:// appears in text. XML content is unchanged in the fields.
Spellchecking is available.
There is support for estimating how long fixing a bug will take and how long it has already taken.

FogBugz provides robust and easy-to-use bug tracking, but by design does not have some common features. Whether these matter for your project is up to you to decide:

The actions you can perform on each case (bug) are limited to the following: create a new bug, edit, assign, resolve (with a number of different reasons), close, and reactivate. Creating different states for bugs is not permitted, though renaming the existing ones should be possible.
Text fields in the bug are for simple text only, since tab characters and long lines are filtered out. Presumably, formatted files should be treated as attachments, though this makes cutting and pasting text from log messages tedious.
The history of a bug is recorded at the end of the bug, but historical reports are not available.
Everyone can see and edit all bugs.
The filters for reports are all single-valued fields, not multivalued, so searching for all bugs from User A or User B requires an SQL query.

The version of FogBugz described here is 3.1.9. Version 4.0 was released early in 2005 and added more complex access controls; operations such as closing groups of bugs; spam filters on incoming bug reports; and the same discussion group software that hosts their own support forums.

FogBugz does a few things and does them well. It's simple to learn, doesn't scare customers away from submitting bugs, and is well supported. If this is enough for your project, then its very reasonable pricing may make it just what you want.

7.2.6. JIRA

JIRA (http://www.atlassian.com/software/jira) is a Java servlet-based bug tracking tool from Atlassian Software Systems. Interaction is by web browser or email. JIRA is licensed commercially, but the source code is also distributed to commerical users. Prices for commercial use range from $1,200 for the standard edition to $4,800 for the enterprise edition. Academic prices are half the commercial prices; open source and nonprofit use is free. These prices are one-off, per server; they include upgrades and a year's support. Version 3.0.2 was the version tested here.

JIRA provides a wide range of features, lets you change them in a number of ways, and then finally encourages you extend JIRA itself and integrate it with other tools. It is clearly very carefully designed to be easy to use. Administration is all performed through the web interface. Just as Bugzilla finds favor with system administrators who use Perl, JIRA fits well with system administrators who are already working with projects that use Java. For example, the build tool used for JIRA is Ant (see Section 5.5.4).

Features beyond those usually found in simple bug tracking systems include the following:

JIRA has a clean and uncluttered UI.
CVS integration is excellent right out of the box and can be configured from a simple UI.
Integration with other tools such as SCM tools and mailers has a clear and documented API. The schema for the database is also freely provided, though subject to change with each version.
URLs and emails are automatically made into links within text. XML text is not parsed. Long lines are not wrapped or truncated in text fields, which can sometimes lead to the layout looking stretched out.
JIRA provides customizable notification by email and RSS feeds about all changes and about each bug.
JIRA has a very useful default report called Recent History, showing which bugs you have recently visited.
Voting for bugs is supported, and there are reports to show the most-requested bugs.
Localization of JIRA is complete for at least three languages.
There is support for timing different operations in the server, to provide some basic profiling.
An integrity checker for the data is provided.

The requirements for JIRA are relatively platform independent. For the standalone version, only Java is required, since a pure-Java database and web server are provided. When the bug tracking system needs to grow larger, JIRA works with numerous versions of the OrionServer, Resin, Tomcat, JBoss, Weblogic, Jetty, and Oracle OC4J application servers.^[2] The different databases that are supported include Oracle, DB2, MySQL, PostgreSQL, and Sybase. The web pages are said to work with Internet Explorer 5 and 6, Mozilla, Firefox, Opera, and Safari, and they work well with Opera 8.0 for me.

^[2] Application servers are a convenient way of making Java applications accessible remotely. J2EE© (Java 2 Platform, Enterprise Edition) is one common standard for such applications.

The user and administration documentation for JIRA is excellent, as is the documentation for how to modify it. The support is "legendary," and actually does have a rather good record. One of the reasons for this is that the project information for JIRA itself is publicly available at http://jira.atlassian.com. JIRA's list of customers is extensive and growing rapidly.

Features that aren't yet in JIRA that you might want are:

A CLI for each supported platform, though a rudimentary Java API to write one is provided using SOAP or XML-RPC. More complex operations can also be performed using HTTP POST calls.
The ability to change states for a group of bugs in one bulk operation (JIRA does let you reassign a group of bugs in one operation).
Reports that include the history of each bug and groups of bugs.

JIRA is a good example of a highly configurable bug tracking system that also works well by default. Entering bugs is simple, but all the abilities for integration and fine-level control are also provided and documented. JIRA is particularly easy to install for a small-scale standalone server. JIRA is a great choice for projects and organizations that want to be able to customize their bug tracking tool more than is permitted by tools such as FogBugz.

7.2.7. TestTrack

TestTrack (http://www.seapine.com/ttpro.html) is a commercial bug tracking system from Seapine Software, Inc. Pricing for TestTrack Pro begins at $295.00 for an individual license or $795 for a floating license that everyone in a project can use. Support typically costs 20% of the purchase price per year. The version tested here was TestTrack Pro 7.0. Seapine also sells Surround SCM, an SCM tool that integrates well with TestTrack.

TestTrack is highly customizable, but system administrators don't have to see the underlying details of the database to make the changes. Servers are available for Windows (in English and French), GNU/Linux, Solaris, and Mac OS X. The client is available only for Windows, though a Perl API is provided for remote access. Other platforms connect to the servers by using a browser. The backend database can be the default proprietary one (xBase), Oracle (natively for speed), or any other database with an ODBC connector.

Features beyond those usually found in simple bug tracking systems include the following:

There is a standalone application for developers and customers to submit bugs using email or the Web, and there is good support for involving customers directly with the bug tracking system.
TestTrack supports spellchecking of text fields.
State transitions can be customized, and diagrams of the states are automatically created.
Administrators can easily change the labels used for each field or users' names.
Encryption of the data exchanged between client and server is supported, as is cryptographic signing of bugs.
TestTrack has basic integration with Visual Studio, as does Surround SCM (which can be used to replace Visual SourceSafe). TestTrack also provides integration with other SCM tools including CVS, ClearCase, Perforce, and Visual SourceSafe.
LDAP or Active Directory can be used for user authentication, and a fairly sophisticated set of administration options for users and their passwords is provided.
Release notes can be automatically built from bugs that have been marked as part of a particular release.
Import and export of all dataincluding bug data, reports, and user detailsis in XML, which makes it somewhat easier to exchange data with other tools or future bug tracking systems.

The list of customers who use TestTrack is in the hundreds, with some good-sized companies listed. Seapine's documentation has received awards from a number of technical documentation societies, and TestTrack has also won Jolt awards from Software Development magazine. There is a moderated discussion site for TestTrack; it has lots of messages, but also plenty of unanswered questions.

Some of the drawbacks of TestTrack are:

No CLI: While the data is all fairly easy to extract in XML, there is no out-of-the-box CLI. You can use Perl and SQL to access the database directly, or use the SOAP SDK add-on, but that's extra.
Lots of tabs: The Windows client is faster and has a cleaner interface than most browser-based clients, but it's still hard to see all the information about a bug on one screenthere are seven horizontal tabs and four vertical ones to choose from! Not surprisingly, the web interface feels slower than the Windows client.
Separate license server: TestTrack and Surround SCM use a license server, which is another potential central point of failure.

Two other aspects of TestTrack struck me as unusual, though they're not really problems:

Drop-down parentheses: The Advanced Find screen has drop-down menus for selecting the number of parentheses in each part of complex queries with AND and OR. This looks a little odd at first, but actually works well enough for up to three or four levels of parentheses.
Separate registry editor: There is a separate Windows application for modifying TestTrack-related values in the Windows registry, which made me wonder why the changes couldn't be made using the existing administration tools.

TestTrack is a flexible and reasonably powerful bug tracking system that is not too hard to administer. It's especially attractive if you can use the Windows client, which feels faster than any of the web-based clients.

7.2.8. Custom Bug Tracking Tools

When you're frustrated with a particular tool, sometimes it's tempting to oversimplify what the tool involves. For example: SCM is really just copying some files to a safe place; build tools really just run a series of commands; and bug tracking tools are really just a frontend to a database. Don't let yourself be deceived by this shallow thinking! One reason why there are so many bug tracking tools available is that lots of other people thought along these lines, and their tools really are just frontends to databases. These are not the tools you're looking for.

The requirements for a bug tracking tool that are listed in Section 7.1, earlier in this chapter, are much harder to achieve. If you feel you have tried all the different tools you can, and none of them do what you want, then why not modify one of the more substantial open source ones?