Track Changes Until You Throw Away the Project

[Previous] [Next]

Version control and bug tracking systems are two of the most important infrastructure tools you have because they give you the history of your project. Although the developers might say they can keep everything in their heads, the company needs to have some record of what's been accomplished on the project in case the entire development team wins the lottery and everyone quits the next day. Because most teams don't adequately maintain their requirements and design documents throughout the life of a project, the only real documentation becomes the audit trail in the version control and bug tracking systems.

I hope I'm preaching to the converted. Unfortunately, I keep running into teams that haven't yet started using these tools. As someone interested in history, I feel you have to know where you've been to know where you're going. Putting these two tools to use is the only sure way to learn that lesson. By monitoring the outstanding bugs and bug fix rates in the bug tracking system, you can better predict when your product will be ready to ship. With the version control system, you'll get an idea of your "code churn," the volume of changes, so that you can see how much additional testing needs to be done. Additionally, these tools are the only effective way to judge whether you're getting any results from changes you implement in your development cycle.

When you bring a new developer to your team, these tools can pay for themselves in a single day. When the new developer starts, have her sit down with the version control and bug tracking software and begin working her way through the changes. Good design documents are ideal, but if they aren't available, the version control and bug tracking systems at least provide a record of the code evolution and highlight any trouble areas.

I'm talking about these two tools in the same breath because they are inseparable. The bug tracking system captures all the events that might have driven changes to your master sources. The version control system captures every change. Ideally, you want to maintain the relationship between reported problems and actual changes in the master sources. By correlating the relationship, you can see your cause and effect for bug fixes. If you don't track the relationship, you're left wondering why certain changes to the code occurred. Invariably, in later versions of the product, you have to find the developer who made the change and hope he or she remembers the reason for the change.

Some products are integrated and automatically track the relationship of the master source change to the bug report, but if your current systems don't, you'll need to maintain the relationship manually. You can track the relationship by including the bug number in the comments that describe the fix. When you check the file back into version control, you'll need to identify the bug number you're fixing in the check-in comment for the file.

Version Control Systems

The version control system isn't just for your project's master sources. Anything and everything related to the project—including all test plans, automated tests, the help system, and design documents—needs to go into the version control system. Some companies even include the build tools (that is, the compiler, linker, include files, and libraries), which allows them to completely re-create the shipped version of their product. If you have any question about whether something should go in version control, ask yourself whether maintenance programmers could use the information in a couple of years. If you think they could, that information belongs in the version control system.

Controlling Changes

Tracking change is vital; however, having a good bug tracking system in place doesn't mean that developers should be allowed to make wholesale changes to the master sources whenever they want. Such carte blanche would make all the tracking pointless. The idea is to control the changes during development, restricting certain types of changes to certain stages of the project so that you can have an idea of the state of the master sources on a day-to-day basis. The best scheme I've heard of for controlling changes comes from my friend Steve Munyan, and he calls it "Green, Yellow, and Red Times." In Green Time, anyone can check in anything to the master sources. The earliest parts of the project are usually fully in Green Time because at this point the team is working on new features.

Yellow Time is when the product is in a bug fix phase or nearing a code freeze. The only code changes allowed are for bug fixes—and only bug fixes. No new features or other changes are permitted. Before a developer can check in a bug fix, a technical lead or a development manager must approve it. The developer making the bug fix must describe the bug fix he's making and what it affects. In essence, this process is a mini code review for every single bug fix. On some teams I've been on, the product was in Yellow Time from day one because the team liked the code review aspects of this stage. We did loosen the approval requirements so that any other developer could approve changes. The interesting outcome was that because of the constant code reviews the developers caught many bugs before they checked the code into the master sources.

Red Time occurs when you're in a code freeze or near a key milestone and all code changes require the product manager's approval. When I was a product manager, I even went to the extent of changing the permissions in the version control system so that the team had read-only access. I took this step mainly because I understood what the developers were thinking: "This is just a little change; it will fix this bug, and it won't hurt anything else." The developers' intentions were good, but that one little change could mean that the entire team would have to restart the test plan from the beginning.

The product manager must strictly enforce Red Time. If the product has a reproducible crash or data corruption, the decision to make the change is essentially automatic because you just do it. In most cases, however, deciding whether to fix a particular bug is less black and white. To help me decide how critical a bug fix was, I always asked the following questions with the company's needs in mind:

• How many people does this problem affect?
• Is the change in a core or a peripheral part of the product?
• If the change is made, what parts of the application will need to be retested?

The answers to these questions provided the criteria I needed to allow or to decide against the change.

The Importance of Labeling

One of the most important commands that you can learn to use in your version control system is its label command. Microsoft Visual SourceSafe calls it a label, MKS Source Integrity calls it a checkpoint, and PVCS Version Manager calls it a version label. Different version control systems might refer to the label command in different ways, but whatever it's called, a label marks a particular set of master sources. A label allows you to retrieve a specific version of your master sources in the future. If you make a labeling mistake, you might never be able to retrieve the exact master sources used for a particular version. That could mean that you might not be able to discover why a particular version is crashing.

When deciding what to label, I've always followed these three hard-and-fast rules:

1. Label all internal milestones.
2. Label any build sent to someone outside the team.
3. Label anytime a daily build occurs.

In all cases, I follow a scheme of "<project name> <Milestone/Reason> <Date>" so that the label names are descriptive. With the first two rules, I'd never release the team from Yellow Time or Red Time until I applied the labels.

The third labeling rule is one that many people forget. Your quality engineers are usually working with the daily build, so when they report a problem, it's against a particular version of the master sources. Because developers can change code quickly, you want to make it simple for them to get back to the exact version of the files they need to reproduce the bug.

Common Debugging Question

---

What do we do if we're having trouble reproducing builds sent to others outside the team?

Every time you do a build for someone outside the team, you should make a complete copy of the project build directory to CD or tape. This copy will include all your source files, the intermediate files, the symbol files, and the final output. Also include the installation kit that you sent the customer. You might even want to consider copying the build tools. CDs and tapes are inexpensive insurance against future problems.

Even when I've done everything possible to preserve a particular build in version control, I've still experienced cases in which a rebuild produced a binary that differed from the original. By having the complete build tree archived, you can debug the user's problem with exactly the same binaries that you sent.

Bug Tracking Systems

In addition to tracking your bugs, the bug tracking system makes an excellent vehicle for jotting down reminders and keeping a to-do list, especially when you're in the process of developing code. Some developers like to keep notes and to-do lists in notebooks, but essential information often gets lost between random hexadecimal number streams from a debugging session and the pages on pages of doodling that you used to keep yourself awake in the last management status meeting. By putting these notes into the bug tracking system and assigning them to yourself, you consolidate them in one place, making them easier to find. Additionally, although you probably like to think that you "own" the code you work on, you really don't—it belongs to the team. With your to-do list in the bug tracking system, other team members who have to interface with your code can check your list to see what you have or haven't done. Another benefit of including to-do lists and notes in the bug tracking system is that fewer details fall through the cracks at the last minute as a result of you forgetting about a problem or another issue. I always find myself running the bug tracking system so that I can quickly jot down important notes and to-dos right when I think about them.

I like to reserve the lowest priority bug code in the system for notes and to-do lists. Flagging notes and to-do lists as lowest priority bugs makes it easier to keep them separate from the real bugs, but at the same time, you can quickly raise their priority if you need to. You should also structure your bug metrics reports so that they don't include the lowest priority bug code, because it will skew your results.

Don't be afraid to peruse the bug tracking data either. All the unvarnished truths about your products are there. When you're planning an update, run through the bug tracking system and find those modules or features that had the most bugs reported against them. Consider adding some time in your schedule to allow team members to go back and strengthen those sections.

When deploying your bug tracking system, make sure that everyone who needs it has it. At a minimum, everyone on the development team and the technical support team needs access to it. If your bug tracking system supports different levels of access, you might also want to think about allowing others, such as sales engineers (technical experts who are part of the sales organization and help the salespeople as needed when they're selling a complicated product) and marketing representatives, to have access as appropriate. For example, you might want to allow certain sales and marketing people to enter bugs and feature requests but not to view existing bugs. These two groups are generally out talking to customers more than your typical engineers are, and the customer feedback they can supply can be invaluable. Having them log their requests and problems in the same system that everyone else uses is efficient and practical. The idea is to have one central place where all problems and feature requests reside. If you store this information in multiple locations, you're more likely to lose track of it.

Choosing the Right Systems for You

Numerous version control systems are available. Some might be easier to use than others or offer more features, but the real issue in choosing the best version control system comes down to your specific requirements. Obviously, if you're in a shop that has high-end requirements, such as multiple platform support, you're going to need to look at one of the more expensive systems. If you're a small team targeting just Microsoft Windows development, however, you can consider some of the less costly alternatives. Make sure you spend some time doing some hard evaluation of the system you're thinking about implementing, especially in trying to predict what you'll need in the future. You're going to be living with your version control system for a while, so make sure it will grow with you. And keep in mind that just as important as choosing the right version control system is using a version control system in the first place: any system is better than no system.

As for bug tracking systems, I've seen many people try to limp along with a homegrown system. Although doing a project with a homemade system is possible, I'd recommend investing in a commercial product. The information in the bug tracking system is too vital to put into an application that you don't have the time to support and that can't grow to meet your needs six months or a year into the project.

The same criteria apply for choosing a bug tracking system as for choosing a version control system. Once, as a product manager, I decided on a bug tracking system without spending enough time looking at the most important part, reporting bugs. The product was easy enough to set up and use. Unfortunately, its reporting capabilities were so limited that we ended up transferring all our existing bugs over to another product right after we hit our first external code milestone. I was rather embarrassed for not having evaluated the product as thoroughly as I should have.

As I mentioned earlier in the chapter, you should definitely consider a bug tracking product that offers integration with a version control product. In the Windows marketplace, most version control systems support the Microsoft Source-Code Control Interface (SCCI). If your bug tracking system supports SCCI as well, you can coordinate the bug fixes with particular file versions.

Some people have described code as the lifeblood of a development team. If that description is accurate, the version control and bug tracking systems are the arteries. They keep the lifeblood flowing and moving in the right direction. Don't develop without them.