Statistical Analysis of Your Bug Database

Another good title for this section should be "Looking into Your Crystal Ball." Once the team can consistently fix more bugs than the test team writes, you can use your bug database to predict your ship date. If that seems impossible or farfetched, I completely understand. I once felt the same way. It comes down to simple mathematics, really.

First there is a cold hard fact to consider. There is a maximum sustainable rate of bug debt your team can retire each working day. Every project is different, but after looking at the outcome of many different computer game projects Microsoft pegs this rate is somewhere between 3 and 5 bugs per team member per day.

A Tale from the Pixel Mines

Programmers can fix only 3 to 5 bugs per day, huh? I can already hear you jeering and calling me a fool. I understand. I did the same thing to the test manager at Microsoft, and it turned out he was completely right. I know how many bugs I can fix per day, and it's more than five, especially if I'm really cranking. I thought this number was closer to 10 or 15 bugs per day. I got into a huge disagreement over this fact on a conference call with one of the test managers at Microsoft. I told him that my developers were the best and they'd blow his socks off, yadda, yadda. Six weeks later I was eating crow, on a conference call, with the same people in attendance. Sure, my developers were fixing 10 bugs per day and more, but the test team was still writing bugs. At the end of the day, the analysis clearly showed that our development team was right on the expected average at 3.75 bugs per day per team member over the life of the project.

It turns out that you don't even want to fix too many more than that, assuming they are real bugs that require some planning, checking, and whatnot. Serious problems require more than an hour to fix correctly, and if you fix them too fast, you're likely not paying due care and attention to your bugs.

Believe it or not, your team can retire bugs no faster than five bugs per team member per day. Period. It doesn't matter how good they are, they are still human.

Let's assume you've topped the bug curve, and the number of active bugs is consistently dropping. It's time to predict your ship date:

 active bug count / ( number of developers n bug fix rate ) = days until zero bugs. 

Here's a real life example. Our last project had 5 programmers, and 1 producer/designer. On June 15, the project had just shy of 300 active bugs. Using the formula above, we show:

 300 / ( 6 * 3.5 ) - 14.285 days until we hit zero bugs. 

It turns out we hit our ZBR date in around 16 working days. Our average fix rate was a little slower than we projected, but we didn't work weekends either.

Certainly one of the worrying times on a project is when the bug count is climbing, and perhaps climbing rapidly. This will occur naturally as the programming team is concentrating all of its effort on finishing new features rather than working on bugs. Take a look at the graph shown in Figure 14.1.

Figure 14.1: Daily Active Bug Count Graph.

You can see a few humps early on where the development team attempted to keep the bug count in check. The amount of time they could devote to fixing bugs wasn't enough to keep the active bug count from growing. This wasn't a problem because we always calculated the maximum allowable active bug count. This maximum count is the number of bugs that can be fixed from the code complete milestone (where programmers are doing nothing but fixing bugs) to the ZBR date (the scheduled date for reaching zero active bugs for the first time). Use the same formula to solve for the active bug count given a certain number of workdays and developers:

 number of developers * bug fix rate n number of working days = maximum active bugs 

In our case this formula worked out to be:

 6 developers * 3.5 bugs fixed per day n 40 days = 840 bugs 

There is a small problem with that formula, however subtle. It assumes that every developer on the team has an equal share of bugs. That is never the case. One of the programmers is usually stuck with a higher than average count. Remove the "developers" part of the formula to determine the maximum number of bugs allowable for each person:

 3.5 bugs fixed per day * 40 days = 140 bugs. 

If you run these numbers yourself and find your counts are too high, what can be done? You already know you have a few options just by looking at the formula. You can clearly increase the number of days you are working by putting in some weekend time. You can also increase the number of bugs the team fixes per day by working longer hours or "punting" low priority bugs. To some extent you can add some extra development help, although they don't ever match the expected bug fix rate. The formula is pretty clear, and there are no other inputs.

This kind of analysis helps a team figure out how much effort to put into bug fixing all along the development of the project. Use these numbers to set the throttle. If you burn too hot your team will be dead tired before the final bugs are fixed, which are usually the hardest ones. Burn too lean and you'll find the active bug count spiraling out of control. Most importantly, use these numbers to justify giving your team a weekend off every now and then. If they're under their expected bug count they deserve it.