Track Changes Until You Throw Away the Project

Version control and bug tracking systems are two of the most important infrastructure tools you have because they give you the history of your project. Although the developers might say they can keep everything in their heads, the company needs to have some record of what's been accomplished on the project in case the entire development team wins the lottery and everyone quits the next day. Because most teams don't adequately maintain their requirements and design documents throughout the life of a project, the only real documentation becomes the audit trail in the version control and bug tracking systems.

I hope I'm preaching to the converted. Unfortunately, I keep running into teams that haven't yet started using these tools, especially bug tracking systems. As someone interested in history, I feel you have to know where you've been to know where you're going. Putting these two tools to use is the only sure way to learn that lesson. By monitoring the outstanding bugs and bug fix rates in the bug tracking system, you can better predict when your product will be ready to ship. With the version control system, you'll get an idea of your "code churn," or the volume of changes, so that you can see how much additional testing needs to be done. Additionally, these tools are the only effective way to judge whether you're getting any results from changes you implement in your development cycle.

When you bring a new developer to your team, these tools can pay for themselves in a single day. When the new developer starts, have her sit down with the version control and bug tracking software and begin working her way through the changes. Good design documents are ideal, but if they aren't available, the version control and bug tracking systems at least provide a record of the code evolution and highlight any trouble areas.

I'm talking about these two tools in the same breath because they are inseparable. The bug tracking system captures all the events that might have driven changes to your master sources. The version control system captures every change. Ideally, you want to maintain the relationship between reported problems and actual changes in the master sources. By correlating the relationship, you can see your cause and effect for bug fixes. If you don't track the relationship, you're left wondering why certain changes to the code occurred. Invariably, in later versions of the product, you have to find the developer who made the change and hope he remembers the reason for the change.

Some products are integrated and automatically track the relationship of the master source change to the bug report, but if your current systems don't, you'll need to maintain the relationship manually. You can track the relationship by including the bug number in the comments that describe the fix. When you check the file back into version control, you'll need to identify the bug number you're fixing in the check-in comment for the file.

Version Control Systems

The version control system isn't just for your project's master sources. Anything and everything related to the project—including all test plans, automated tests, the help system, and design documents—needs to go into the version control system. Some companies even include the build tools (that is, the compiler, linker, include files, and libraries), which allow them to completely re-create the shipped version of their products. If you have any question about whether something should go in version control, ask yourself whether maintenance programmers could use the information in a couple of years. If you think they could, that information belongs in the version control system.

The Importance of Including Unit Tests

Even though I just said to check in everything that could be of use, one of the biggest problems I've seen in development shop after development shop is not including the unit tests in the version control system. In case you're not familiar with the term unit test, I'll briefly describe it here. A unit test is the piece of code that drives your main program code. (It's sometimes also referred to as test apps or test harness.) It's the testing code created by the developer that allows the developer to do "glass box," or "white box," testing to ensure the basic operations take place. For a complete definition of unit tests, see Chapter 25, "Unit Testing," of Steve McConnell's book Code Complete (Microsoft Press, 1993).

Including the unit tests in version control accomplishes two key objectives. First, you make the job of maintenance developers infinitely easier. So many times, the maintenance developer, who could easily turn out to be you, has to reinvent the wheel when upgrading or fixing the code. Doing so is not only a huge waste of effort but also a real morale killer. Second, you make general testing for QA teams trivial so that those teams can focus on the important testing areas, such as performance and scalability as well as fit and finish. One sign of a seasoned professional is that she always has her unit tests checked in.

Of course, checking in your unit tests means that you automatically have to keep them up to date with code changes. Yes, that's going to be additional work on your behalf. Nothing is worse than having a maintenance developer hunting you down and slapping you silly because the unit tests no longer work. Having outdated unit tests is worse than having no unit tests in the version control system.

If you look at the source code for this book, you'll notice that all my unit tests are included as part of the code. In fact, there's a separate build to automatically build all the unit tests for all the utilities or examples I include. I'm not about to recommend anything in this book that I don't do myself.

Some readers might be thinking that implementing the unit test discipline I advocate is going to take a lot more work. In reality, it's not going to take that much longer because most developers (I hope!) are already conducting unit tests. The big differences I'm suggesting are keeping those tests checked in and up to date as well as possibly generating a build script of some sort for them. The time you'll save by following the proper procedures will be huge. For example, for most of this book, I did my development on a machine with Microsoft Windows 2000 Server installed. To turn around and do immediate testing on my Microsoft Windows XP machines, I simply had to get the code from version control and do the build scripts. Many developers do a one-off unit test, so testing on other operating systems is difficult because they have no easy way to get the unit tests over and built. If everyone is making unit tests part of the code, you can shave many weeks off your schedules.

Controlling Changes

Tracking changes is vital; however, having a good bug tracking system in place doesn't mean that developers should be allowed to make wholesale changes to the master sources whenever they want. Such carte blanche would make all the tracking pointless. The idea is to control the changes during development, restricting certain types of changes to certain stages of the project so that you can have an idea of the state of the master sources on a day-to-day basis. The best scheme I've heard of for controlling changes comes from my friend Steve Munyan, and he calls it "Green, Yellow, and Red Times." In Green Time, anyone can check in anything to the master sources. The earliest parts of the project are usually fully in Green Time because at this point the team is working on new features.

Yellow Time is when the product is in a bug fix phase or nearing a code freeze. The only code changes allowed are for bug fixes—and only bug fixes. No new features or other changes are permitted. Before a developer can check in a bug fix, a technical lead or a development manager must approve it. The developer making the bug fix must describe the bug fix he's making and what it affects. In essence, this process is a mini code review for every single bug fix. The important item to remember when conducting that code review is to utilize the version control product's differencing utility to ensure exactly which changes occurred so that extraneous changes don't sneak in. On some teams I've been on, the product was in Yellow Time from day one because the team liked the code review aspects of this stage. We did loosen the approval requirements so that any other developer could approve changes. The interesting outcome was that because of the constant code reviews, the developers caught many bugs before they checked the code into the master sources.

Red Time occurs when you're in a code freeze or near a key milestone and all code changes require the product manager's approval. When I was a product manager (the person on the team responsible for the code as a whole), I even went to the extent of changing the permissions in the version control system so that the team had read-only access. I took this step mainly because I understood what the developers were thinking: "This is just a little change; it will fix this bug, and it won't hurt anything else." The developers' intentions were good, but that one little change could mean that the entire team would have to restart the test plan from the beginning.

The product manager must strictly enforce Red Time. If the product has a reproducible crash or data corruption, the decision to make the change is essentially automatic because you just do it. In most cases, however, deciding whether to fix a particular bug is less black and white. To help me decide how critical a bug fix was, I always asked the following questions with the company's needs in mind:

• How many people does this problem affect?

• Is the change in a core or a peripheral part of the product?

• If the change is made, what parts of the application will need to be retested?

The answers to these questions provided the criteria I needed to allow or to decide against the change. Let me put some concrete numbers behind this list and give you my general rules for the beta phases. If the is the bug is serious, which means that it's less than a crash or data corruption showstopper, and it's going to affect greater than 15 percent of our external testers, the bug is an automatic fix. If the bug will result in a change to a data file, I go ahead and fix it so that we don't have to change file formats later in the development process and so that beta testers can consequently get larger data sets for subsequent betas.

The Importance of Labeling

One of the most important commands that you can learn to use in your version control system is its label command. Microsoft Visual SourceSafe calls it a label, MKS Source Integrity calls it a checkpoint, and PVCS Version Manager calls it a version label. Different version control systems might refer to the label command in different ways, but whatever it's called, a label marks a particular set of master sources. A label allows you to retrieve a specific version of your master sources in the future. If you make a labeling mistake, you might never be able to retrieve the exact master sources used for a particular version. That could mean that you might not be able to discover why a particular version is crashing.

When deciding what to label, I've always followed these five hard-and-fast rules:

1. Label all internal milestones.

2. Label any transitions from Green, Yellow, or Red development times.

3. Label any build sent to someone outside the team.

4. Label any time you branch a development tree in the version control software.

5. Label after the daily build and smoke tests complete correctly. (Smoke tests are discussed in the "Smoke Tests" section later in this chapter.)

In all cases, I follow a scheme of "<Project Name> <Milestone/Reason> <Date>" so that the label names are descriptive.

The third labeling rule is one that many people forget. Your quality engineers are usually working with the milestone or daily build, so when they report a problem, they do so against a particular version of the master sources. Because developers can change code quickly, you want to make it simple for them to get back to the exact version of the files they need to reproduce the bug.

Common Debugging Question: What do we do if we're having trouble reproducing builds sent to others outside the team?

Every time you do a build for someone outside the team, you should make a complete copy of the project build directory to CD/DVD or tape. This copy will include all your source files, the intermediate files, the symbol files, and the final output. Also include the installation kit that you sent the customer. You might even want to consider copying the build tools. CD/DVDs and tapes are inexpensive insurance against future problems.

Even when I've done everything possible to preserve a particular build in version control, I've still experienced cases in which a rebuild produced a binary that differed from the original. By having the complete build tree archived, you can debug the user's problem with exactly the same binaries that you sent.

Bug Tracking Systems

In addition to tracking your bugs, the bug tracking system makes an excellent vehicle for jotting down reminders and keeping a to-do list, especially when you're in the process of developing code. Some developers like to keep notes and to-do lists in notebooks, but essential information often gets lost between random hexadecimal number streams from a debugging session and the pages on pages of doodling that you used to keep yourself awake in the last management status meeting. By putting these notes into the bug tracking system and assigning them to yourself, you consolidate them in one place, making them easier to find. Additionally, although you probably like to think that you "own" the code you work on, you really don't—it belongs to the team. With your to-do list in the bug tracking system, other team members who have to interface with your code can check your list to see what you have or haven't done. Another benefit of including to-do lists and notes in the bug tracking system is that fewer details fall through the cracks at the last minute as a result of you forgetting about a problem or another issue. I always find myself running the bug tracking system so that I can quickly jot down important notes and to-dos right when I think about them.

I like to reserve the lowest priority bug code in the system for notes and to-do lists. Flagging notes and to-do lists as lowest priority bugs makes it easier to keep them separate from the real bugs, but at the same time, you can quickly raise their priority if you need to. You should also structure your bug metrics reports so that they don't include the lowest priority bug code, because it will skew your results.

Don't be afraid to peruse the bug tracking data either. All the unvarnished truths about your products are there. When you're planning an update, run through the bug tracking system and find those modules or features that had the most bugs reported against them. Consider adding some time in your schedule to allow team members to go back and strengthen those sections.

When deploying your bug tracking system, make sure that everyone who needs it has it. At a minimum, everyone on the development team and the technical support team needs access to it. If your bug tracking system supports different levels of access, you might also want to think about allowing others, such as sales engineers (technical experts who are part of the sales organization and help the salespeople as needed when they're selling a complicated product) and marketing representatives, to have access as appropriate. For example, you might want to allow certain sales and marketing people to enter bugs and feature requests but not to view existing bugs. These two groups are generally out talking to customers more than your typical engineers are, and the customer feedback they can supply can be invaluable. Of course, this means you're going to have to give sales and marketing folks classes on how to fill out bug reports. They're more than happy to help, but you need to give them the guidance so that they can do it properly. Having these groups log their requests and problems in the same system that everyone else uses is efficient and practical. The idea is to have one central place where all problems and feature requests reside. If you store this information in multiple locations, such as in the product manager's e-mail inbox, in engineer paper notebooks, and in the bug tracking system, you're more likely to lose track of it.

Choosing the Right Systems for You

Numerous version control systems are available. Some might be easier to use than others or offer more features, but the real issue in choosing the best version control system comes down to your specific requirements. Obviously, if you're in a shop that has high-end requirements, such as multiple platform support, you're going to need to look at one of the more expensive systems or possibly an open source solution such as CVS. If you're a small team targeting just Microsoft Windows development, however, you can consider some of the less costly alternatives. Make sure you spend some time doing some hard evaluation of the system you're thinking about implementing, especially in trying to predict what you'll need in the future. You're going to be living with your version control system for a while, so make sure it will grow with you. And keep in mind that just as important as choosing the right version control system is using a version control system in the first place: any system is better than no system.

As for bug tracking systems, I've seen many people try to limp along with a homegrown system. Although doing a project with a homemade system is possible, I'd strongly recommend investing in a commercial product or utilizing an open source solution. The information in the bug tracking system is too vital to put into an application that you don't have the time to support and that can't grow to meet your needs six months or a year into the project. Additionally, developers avoid wasting time on internal tools and work instead on revenue-producing products.

The same criteria apply for choosing a bug tracking system as for choosing a version control system. Once, as a product manager, I decided on a bug tracking system without spending enough time looking at the most important part, reporting bugs. The product was easy enough to set up and use. Unfortunately, its reporting capabilities were so limited that we ended up transferring all our existing bugs over to another product right after we hit our first external code milestone. I was rather embarrassed for not having evaluated the product as thoroughly as I should have.

As I mentioned earlier in the chapter, you should definitely consider a bug tracking product that offers integration with a version control product. In the Windows marketplace, most version control systems support the Microsoft Source Code Control Interface (MSSCCI). If your bug tracking system supports MSSCCI as well, you can coordinate the bug fixes with particular file versions.

Some people have described code as the lifeblood of a development team. If that description is accurate, the version control and bug tracking systems are the arteries. They keep the lifeblood flowing and moving in the right direction. Don't develop without them.