The Bug Database | Game Coding Complete

I've worked with a few different bug databases. Good ones can actually help you predict when you'll ship your game (more on that later). Bad ones can make everyone's life a living hell. Don't think for a moment that you can do without one. Even better, convince yourself that writing your own database in MySQL is a horrible idea; go out there and find something that works today. Your job, after all, is to write a fantastic game for your players. Don't waste time writing, fixing, and tweaking a home-grown bug database.

A Tale from the Pixel Mines

When I was at Origin Systems the bug database was not much more than a simple system built on FoxPro, an off the shelf database system. The bug database was difficult to use, extremely slow, and didn't provide useful information beyond simple bug counts. This system, and I use that term loosely, motivated me to make a horrible decision when I was the project director on Ultima VIII. I decided not to use the database at all. Instead, I decided it would be better for the team to use paper bugs, one bug per piece of paper clipped on a clipboard. What a tragic error! A few months before the ship date the test team called me into a meeting where they were concerned about the high bug count. I didn't show a high bug count (maybe 2 to 3 inches of paper) in my "records." The reality was that somehow an enormous stack of bugs, many hundreds of them, had never been assigned to the team. A few others and I spent hours sorting through the mess, assigning bugs, and coming to a serious understanding about the magnitude of my error in judgment. Since then, I've been very serious about bug databases.

Perhaps the best bug database I've ever used is RAID. Don't go searching for it on the Internet because you won't find it for sale. It is the internal application Microsoft uses to track bugs. It's a proprietary client application to a Microsoft SQL back end, and it has a number of important features that every good bug database must have. There are some excellent bug databases out there, and I'm sure some will run circles around RAID. The particular database product you choose should at least provide a few basic features. On non-Microsoft projects, I use PVCS Tracker for our bug database. It's not quite as good as RAID, but it gets the job done. It has all of the features a good bug database needs.

It goes without saying that the database should have flexible searching criteria. Almost any game will have hundreds, if not thousands, of issues. Finding all the bugs assigned to a particular person is pretty simple, but searching through the thousands of bugs that have been closed for a particular text string in the bug history might be a lot tougher. Inevitably you'll remember something about a previous bug that has relevance to today's project stopping problem. If the bug database puts every byte of information at your fingertips you'll be able to make good use of solutions in the can.

Essentially, you want a database that exposes an ad-hoc query mechanism that lets you construct your own queries. Stay away from a database that uses a simple menu system or dialog box. Those might be nice for some executives out there but programmers and other folks in the trenches want to construct very specific searches. If your database lets you construct SQL queries or something similar, you're way ahead of the game.

Here are some examples of common queries to a bug database:

Find all my active bugs for this milestone sorted by priority: Individual developers on a team will use this query most every day.
Find all active bugs, sorted by team member: Team leads will use this query to get a bird's eye view of the bug fixing effort.
Find all active bugs, sorted by severity: Team leads can use this query to see how serious the bug problems are getting.
Find all resolved bugs that we've fixed in the previous build: This is a good one to gauge the efforts of the testing team. They shouldn't be letting resolved bugs pile up.
Find all resolved bugs fixed in the next build: If your resolved bug count is getting higher than 20 bugs per tester, you should consider firing a new build to the test team.

Storing and categorizing information in a database is just the starting point. You need to be able to make use of that information to increase the quality of your game. Take a look at the fields that are commonly used in most every bug database, and how they can be used.

Title or Short Description: A one line description of the bug. A good title is really useful for scanning enormous lists of bugs when you are assigning them to team members or just estimating whether or not you'll see the sun the following weekend.
Who Found the Bug: The name of the tester, preferably in the form of their email address. Sometimes developers must talk directly to a tester. Never keep your testers and developers blind to each other in the bug database.
When: The date and time the bug was found. This information will be extremely useful in finding out how long bugs stay active in the database.
Revision Number: The version number of your game. It's not enough to use the date and time of the build. Make things easy on yourself and put a version number easily accessible in the application.
Machine Configuration: This is nice if it is automatically recorded when testers write bugs. It should contain everything and then some: CPU speed, system RAM, VRAM, hard disk, OS, additional hardware, or anything that might be important.
Severity: A subjective field at best, but it's still important to rate the bug by its severity to give the programmers a chance to sort their bugs. After all if a bug nukes a hard drive it's a good idea to get it fixed fast.
Issue Type: This field is a general classification. Good examples of issue types include code defect, documentation, localization, specification issue, suggestion, or work item. Use this field to direct the bug to the right group before it is assigned.

The bug description should be incredibly detailed. If it is done properly a programmer should be able to diagnose the problem quickly. It should include instructions to reproduce the bug, what the bug did, and what was expected. Take a look at the following example:

===== Opened by a-joekit on 01/29/2002 10:44AM =====
Description:
Selecting Undo at the end of a game after several games have been played (two or more) and then playing to end again results in an inaccurate number of games played in scorecard. There is one less than there should be.
Repro:
1. Open Trump.
2. Open Gin Rummy.
3. Play a game to completion (i.e., past 100 points, not just one hand unless you lose that badly). You may use "A" to have it auto play but this will not repro if you select AI vs. AI at the beginning because you do not have the Undo button.
4. Play another game to completion.
5. Press Undo after viewing the scorecard from your second game.
6. Play the game to completion again.
7. Look at the scorecard.
Results:
The number of games is one less than it should be. If your scorecard previously showed 2 games played, it will now show only one. If you play 3 games and then undo and complete, it will show 2 games, etc.
Expected Results:
Scorecard should accurately reflect the number of games played.

The description is a short summary of the bug. When the programmer follows the repro steps exactly, they should be guaranteed to see the same results in the debugger, hopefully catching the bug in the act. The results are what the tester observed after the repro steps were followed, and the expected results were what the tester expected to observe.

Gotcha

Some bugs cannot be reproduced 100%. As hard as the testers try, they might not be able to find a clear set of steps that will always result in the bug appearing. All good testers check each questionable bug before writing it up. If they can't reproduce the bug every single time with the steps they suggest, they must also include the percentage chance that it will occur. Every developer should insist on this from their test team, because if a bug only happened once and was never seen again it shouldn't have a high priority. The percentage chance might also be a valuable clue to a programmer, so always get it in the bug report.

You might wonder why we bother writing the expected results—shouldn't it be obvious? The answer is decidedly not. Testers make use of the design document heavily when they create their test plans, and where the design document leaves off their imaginations and creativity picks up. Their expectations of your game's response to their wacky input might be completely justified either by the design document or their intuition.

The most commonly searched field in a database is the status field. Most databases I've worked with have a status field with three states:

Active: The bug belongs to the development team and is in the process of being resolved.
Resolved: The bug belongs to the test team where they will check the game to see if the programmer actually managed to fix the bug without breaking five other things.
Closed: The bug has been addressed and belongs only to the database.

An important feature of many databases that I've worked with is that some database fields, especially the status field, are secure and can't be edited directly. It wouldn't be appropriate (as much as some programmers would want to) for a development team member to simply 'close' a bug without having it run though the test team. Even if all fields are editable, a record of each change to the bug should be kept in a running log of the bug's history. This is extremely useful for mining information out of the database. You'll get to see a great example of this when we see the next bug report.

When the test team works like crazy they'll write hundreds of bug reports and dump them onto the development team, usually on a Friday afternoon! Usually the lead programmer and the project director will sit in front of the bug database and try to make reasonable decisions about assigning bugs to others on the development team. The bug database must make this process fast and painless or it will be deemed useless by people that need it most. Make sure your bug database can do the following things:

Edit many bugs at the same time (such as assigning all selected bugs to one person).
Page to the next bug or the previous bug with a hotkey or toolbar button. You'll be doing this a lot.
Add a team member.
Minimize keystrokes, such as having field value prediction like Quicken!

Now that the bugs are all assigned, your team members will want to be able to look through the bug on their list. There are a couple of good fields that the team can use to prioritize bugs and bug fixing:

Priority: Bugs with a higher priority get worked on first. The production team usually sets priority so the team leads can direct effort in the right place. The test team might weigh in on priority also, so keep them in the loop.
Milestone: If a bug doesn't need to be fixed until a certain milestone is reached, make sure you reflect that in the bug. This is a great way of focusing development effort on the problems that will help the team make the next milestone.

Gotcha

When you have tons of bugs on your plate you tend to mechanically work on each one as it is sorted, without really even looking at the bug. Sometimes you'll even wonder "why in the hell is this bug assigned to me? That code isn't mine!" or "Hey, I thought the game was supposed to work that way." Always approach a bug with 10% skepticism. The bug might not even be a real bug. This is especially true when everyone is tired and overworked. Try to keep just enough neurons firing to make your own judgment calls about each bug before you start to fix it.

When the bug is finally resolved the following fields should get filled in:

Resolved By: This should be the email address of the person who resolved the bug. If the bug needs to bounce back for additional work, this is the person who will get it first.
Resolved Date: By subtracting the opened date from the resolved date, you can create a report that analyzes how long it takes for your development team to fix bugs on average. This can be invaluable for predicting your ship date.
Resolved Revision: This field is critical, because it tells the testers when they can expect a build that has the fix. If the developer gets this wrong when he resolves the bug, the tester might test it too early and find the bug still exists. They'll send it back to the developer, who will emphatically deny that the bug exists and that the entire test team is smoking crack. You get the idea: Get this field right and avoid a brawl.
Cause: This field allows the developer to record why the bug occurred. This is a great candidate for a drop down box, and here are some suggested values:
- Dev Error: The bug was a development error; most bugs fall into this category.
- Dev Missing Code: The bug was caused by code that didn't exist when the developer thought it was supposed to be there.
- Not Yet Implemented: The testers saw something that didn't exist yet and it wasn't supposed to exist.
- Specification Change/Flaw: The bug was actually in the design document and not a code problem.
- Content Change: A change in any asset other than code fixed this bug: art, sound, help files, or whatever.
- External Change: The bug was fixed by updating a licensed library, hardware driver, or something like that.
Resolution: This field tells the test team how the developer fixed the bug. Some values that are good for this are:
- Fixed: The bug was fixed by the developer.
- Not Reproducible: The bug could not be reproduced by the developer using the steps given by the tester.
- Won't Fix: It is agreed by the development team and the test team that the issue is a problem, but it won't be fixed.
- By Design: This bug is actually a feature of the game.
- External: The bug was fixed by something external to the development and test teams.
- Postponed: The bug will be fixed in the next version of the game, assuming there is a next version of the game.
- Duplicate: This happens when two or more bug reports actually refer to the same issue. It is important that when a bug is marked as a duplicate, that the bug database retain a record of the original bug.

Beyond those fields it's also a great idea to make use of the bug history to record some details about the exact nature of the fix. It's not uncommon to see snippets of code in our bug history showing the change and why the code was broken in the first place.

A Sample Bug Report

I promised you a good bug with some decent bug history, and here it is. This is a bug report written up by one of the senior testers at Microsoft on a version of Freecell that I wrote for one of their card game products. It turns out that I didn't quite understand the nuances of Freecell. It also turns out that the original programmer of Microsoft Freecell, Jim Horne, chimed in to set me straight.

===== Opened by jkburns on 05/17/2002 05:02PM =====
DESCRIPTION:
The automatic movement of cards to the foundation piles by the engine is inconsistent—sometimes cards are moved, other times cards can be moved and are not.
REPRO:
1. Open Trump.
2. Go to Freecell.
3. Move a card to one of the 4 free cells such that the next card up in the tableau pile can be moved to the foundation pile.
4. Move a card from one tableau column to another such that a card that can be moved to the foundation pile is exposed.
ACTUAL:
The first case (#3) results in an automatic move of the card to the foundation pile. The second case (#4) results in nothing happening.
EXPECTED:
Anytime a card can be automatically moved to the foundation piles, it should be.
NOTES:
===== Edited by jkburns on 05/17/2002 05:16PM =====
Cards in the free cells are not automatically moved to the foundation piles either...
===== Assigned by joye.mcburnett on 05/28/2002 02:54PM =====
—> AssignedTo: Active -> mike.mcshaffry
===== Assigned by robjera on 05/28/2002 02:55PM =====
—> AssignedTo: mike.mcshaffry -> joye.mcburnett
Please check for consistency with Windows Freecell.
===== Assigned by robjera on 05/28/2002 02:56PM =====
—> AssignedTo: joye.mcburnett -> jkburns
===== Edited by jkburns on 05/31/2002 09:05AM =====
—> AssignedTo: jkburns -> Active
I have extensively played the PC version of Freecell, and my conclusion is that it is really buggy... It seems to automatically move cards to the foundations if the rank is less than or equal to 3, but then it will suddenly remove all cards and report that you won. One game I played I had 4's as the top cards of my diamonds and hearts foundation piles, and 5's as the top cards of my clubs and spades foundation piles; I made a move and the game ended saying I won through a dialog! But I had to manually move the 4's and 5's to the foundation piles when they were exposed. Sometimes the cards in the free cells are not moved to the foundation piles, but I think this has to do with the fact that it avoids automatic movements of 4's and up...
Here's what I think should happen if we are to have automatic movements implemented for freecell, since the actual freecell game seems inconsistent as well:
- anytime a move is made (any move), all fully exposed cards, including those cards in the free cells, are scanned to see if they can be placed on a foundation pile top.
- if this scan results in card(s) moving to the foundation pile, then the card(s) exposed from this move should be checked to see if they can be moved to the foundation pile—a recursive check until the newly exposed cards are determined not to be a legal move to the foundation piles...
I don't know if there is a spec out there that more clearly outlines the logic behind automatic movements in freecell, perhaps it is doing the right thing after all, I just can't figure out why some cards when exposed are not moved to the foundation piles. I'll see if I can get in touch with the windows freecell devs for clarification.
I think if we are to have the automatic movement of cards to the foundation piles implemented, it should be consistent and fully implemented.
===== Assigned by matt.lamari on 06/04/2002 02:09PM =====
—> AssignedTo: Active -> mike.mcshaffry
===== Resolved as By Design by mike.mcshaffry in 02.00.06.0016 on 06/05/2002 09:30AM =====
—> AssignedTo: mike.mcshaffry -> jkburns
Hi John - Freecell isn't buggy as far as I can tell, but the rules of when a card can go onto the foundation automatically are a little weird. The automatic move is only made if the card in question isn't needed. The card is deemed "needed" if there exists anywhere in the tableau columns another card that can legally sit on top of the card that move automatically to the foundation.
In Freecell, it is illegal to move a card away from a foundation pile once it is placed there. This means that if cards were automatically moved as soon as they could, the game could easily end up in an "unwinnable" state.
Do you think you're observing what I describe above?
===== Activated by jkburns on 06/17/2002 03:12PM =====
—> AssignedTo: jkburns -> Active
Mike, you are correct that the cards 1 less than the card in question are considered before automatically moving it to the foundation piles, but cards of rank 2 are an exception. I spoke with Jim Horne, the creator of Freecell, and he explained the logic of automatic movements from the original windows FreeCell. He was pretty cool about telling me about FreeCell! Here is the email that he sent me:
Ok, here it is. Note that this is not the most aggressive logic, which is by design. It is, however, very clear why each card can be moved to the home cells.
I go through each card in the free cells, and the cards on the bottom of each column looking for cards that are useless (defined below), and repeat that until no more are found or until no cards are left.
A card is useless if:
1. If it's an ace, it's useless with no further checks.
2. If it's a deuce, all that needs to be checked is if the corresponding ace is already up in a home cell. If so, it's useless.
3. Any other card is marked useless if both cards which are one number smaller and of the opposite color are already up in a home cell.
I verified this in Windows FreeCell: there can be an ace of the opposite color on the tableau and a deuce will still automatically move up to the foundation pile if its ace is placed there. This did not happen in Trump's FreeCell. I think this is the only discrepancy between Windows FreeCell and Trump FreeCell automatic movement logic...
===== Edited by jkburns on 06/17/2002 03:59PM =====
In addition to the above, I just noticed the following: An ace that is fully exposed when the game begins, if moved to the free cells, is automatically moved to the foundation piles in Windows FreeCell, but not in Trump FreeCell. In Trump, the ace stays in the free cell...
===== Edited by jkburns on 06/17/2002 05:04PM =====
I have also noticed that the cards in the free cells are not checked for automatic movement to the foundations, either in Windows FreeCell, if a card is in a free cell that, upon a user move, should be automatically moved to the foundation piles (given rules described above), it is. In Trump this is not the case for cards in the free cells.
===== Assigned by joye.mcburnett on 06/18/2002 08:30AM =====
—> AssignedTo: Active -> mike.mcshaffry
===== Resolved as Fixed by mike.mcshaffry in 02.00.06.0019 on 06/18/2002 01:54PM
=====
—> AssignedTo: mike.mcshaffry -> ben.roberts
Man, you're good. No fair getting to talk to the original programmer, either, I think that's cheating!
Anyway, the result of my labor should be an exact as possible replica of Windows Freecell, using the rules he describes. Really the only thing I did was:
1. Implemented Rule #3 by looking at what exists in the home cells, versus what didn't exist in the tableau columns. A subtle, but meaningful, difference.
2. Implemented Rule #1 and Rule #2 specifically, as special cases of Rule #3.
3. Run the first check right after the deal, instead of right after the first move.
4. Made sure that all rules check top cards in the tableau as well as the free cells—that wasn't being done.
That should do it. Enjoy!
===== Assigned by marshall.andrews on 06/20/2002 11:23AM =====
—> AssignedTo: ben.roberts -> jkburns
===== Closed by jkburns on 07/01/2002 10:18AM =====
—> AssignedTo: jkburns -> Closed
Verified as fixed in 02.00.06.0019—rules as stated above are implemented correctly.

The bug history contains a lot of information, not the least of which is a conversation between the tester and me. It also shows each change that was made to the bug during its lifetime as it was being assigned to various developers, managers, and testers.

Other Nice Features

There were some extremely nice features in some of the bug databases in my past. These features made life much easier. If you can find a bug database with all these features don't ask how much it is, just go get it. You'll be glad you did.

Customizable fields let the test and production teams add very game specific information to the database, and make searching on these fields much more robust. An excellent example would be the addition of a "Mission" field to an action adventure game, or a "Map Coordinates" field to an RPG or MMP game. Driving games might have a field for the kind of car or the weather conditions. This information helps the team figure out who is the best person to fix the bug, and where the majority of problems collect.

The bug database should be remotely accessible via the web or proprietary client. The team leads can log in from wherever they are (airport, home, trade show) and deal with any problems or issues. It goes without saying that third-party developers are far flung from their testing team, and sending the bug database via FTP just won't cut it. The bug database is constantly changing and should always be available to every team member wherever they are.

The database should be able to include file attachments as a part of the bug. This can be invaluable for getting a test script, screen shot, saved game file, or other piece of information from the test team to the developers. A web based bug database makes this trivial, since you can simply put a link in the bug history.