How Your Product Should Flow

Never mistake activity for achievement.

Coach John Wooden, UCLA basketball legend

Recently, while I was at a popular application development site going through a build architect review, I noticed how extra busy everyone was. Everyone was running around like he was on the floor of the New York Stock Exchange trying to sell some worthless stock before the market closed. People barely had enough time to stop and talk to me about their top five build or SCM pain points. They didn't have time for chitchat because they were too preoccupied with putting out fires such as build breaks, administrating tools and permissions, and reacting to new bugs coming from their customers. Their explanation was that they did not have enough resources to do what the upper managers wanted them to do. This might have been partially true, but it was not the complete truth. They were equating this busy work as their job duties and why they got paid. This was later confirmed when I gave them my final trip report of how to improve their processes such that everything would be fixed and automated. The first question their build team asked was "If all of this is fixed and automated, then what will we do?" I was shocked. These guys were so used to being in reactive mode that they seemed to think that if they were not constantly putting out fires, their position was not needed.

The rest of this chapter outlines a smooth flow of how your product development should go. As Kent Beck, author of Test Driven Development and several Extreme Programming books, points out, flow is what the build team should encourage and try to achieve. The build team drives the product forward. I put together Figure 1.2 to show how this works at Microsoft because I don't think this concept is always clear. I don't think this concept is always clear, as this is the underlying philosophy of this book.

Figure 1.2. Software development flow.

Software Development Flow

The three boxes at the top of Figure 1.2 represent the respective teams listed. The members of each team meet to discuss the progress of its code development.

After the teams discuss the issues, they mark their priority in a bug database, or work item tracker. Sometimes at Microsoft we call everything (features, requirements, bugs, tasks, risks, wish list) a bug, but work item is more accurate.

Teams must enter every type of code implementation or necessary fix on the project into the work item tracker and assign it a tracking number.

Some Work Item Field Definitions

With the internal Microsoft work item tracker more than 46 fields are available in each item, although not all are used all the time. For Microsoft confidentiality reasons, I cannot include a graphic of our tracking tool here. However, the following are some of the fields that are included in a work item.

Setting work item priority and severity:

• Priority This field communicates overall importance and determines the order in which bugs should be attacked. A bug's priority takes severity and other project-related factors into account.

  • Pri 0 Fix before the build is released; drop everything you are doing and fix this immediately.

  • Pri 1 Fix by the next build.

  • Pri 2 Fix soon; specific timing should be based on the test/customer cost of the workaround.

  • Pri 3 Fix by the next project milestone.

  • Pri 4 Consider the fix by the upcoming release, but postponement is acceptable.

• Severity This communicates how damaging a bug is if or when it is encountered.

  • Sev 1 This involves an application crash, product instability, a major test blockage, a broken build, or a failed BVT.

  • Sev 2 The feature is unusable, a bug exists in a major feature and has a complex workaround, or test blockage is moderate.

  • Sev 3 A minor feature problem exists, or the feature problem has a simple workaround but small test impact.

  • Sev 4 Very minor problems exist, such as misspelled words, incorrect tab order in the UI, broken obscure features, and so on. Sev 4 has little or no test impact.

Following are other work item or bug field definitions:

• Status Active, Resolved, or Closed

• Substatus Fix Available

• Assigned To The most critical field, because this is the owner of the item

• FixBy The project due date for the bug fix

Each work item has two build fields:

• Build (1) The build number that the bug was found on

• Build (2) The build number that the bug was resolved on

Microsoft Sidenote: How Visual Studio Resolves and Closes Bugs

Testers close bugs.

Deep thought of the day.

I once was asked by a test manager to summarize everything I learned about builds in one sentence. I told him that "there are no free lunches, especially in the build lab, but there might be free beer." He told me that he was disappointed that I did not have anything deeper than that. He then said his motto was "Testers close bugs." I knew what he meant, so I said with tongue-in-cheek, "Wow, that's deep." I'm not sure if he took that as a compliment or just thought I was not very funny. Regardless, he did have a good point.

Let's break down the details of "a bug's life..."

When a developer fixes a bug on his machine, he marks the bug's substatus as Fix Available and keeps it assigned to himself. After he checks in the change to the team branch or tree, he resolves the bug (changing the status from Active to Resolved) and reassigns the bug to the original bug opener or a tester who owns that area of the product.

The original bug opener or tester then waits until an official build comes out that contains the bug fix. He then walks through the repro steps to ensure that the bug has truly been fixed. If it has, he closes the bug by changing the status from Resolved to Closed. If the issue still exists, the bug opener or tester reactivates the bug by resetting the status to Active and reassigning it to the developer. This continues until the bug is fixed or gets postponed for the next milestone or release.

WAR or Ship Meeting

Known as WAR, Central WAR, or Ship (the softer, more friendly Visual Studio Team System term), this meeting is focused on tracking and controlling the main product build. Its goal is to ship the product at a high quality according to its schedule by dealing with day-to-day project issues, test reports, and metric tracking.

Figure 1.3. WAR team.

The WAR team and everyone attending the WAR meeting must approve every work item before it can get built and shipped in the product. After the WAR team approves a work item, a field in the bug tracker gets set so that everyone on the build team knows that it's okay to accept this check-in into the main build lab.

If the WAR team does not approve the work item, the work item is reassigned to the person who opened it or to Active, which means that no specific person owns the bug, just a team. At this point, if the person who opened the bug thinks it should be fixed sooner than the people in the WAR meeting determine, it is his responsibility to push back with a solid business justification. If the person pushes back to the WAR team with a solid business justification and the WAR team still doesn't accept the change into the build, the work item is marked as Won't Fix or Postponed.

Upon the item's WAR team approval, the developer works with the build team to get his code changes into the next build. After the build team compiles and links all the source code, the code goes through the congeal process, which brings all the pieces of the project together. This includes files that don't need to be compiled, such as some HELP, DOC, HTML, and other files.

Then the post-build process starts (more on post-build in Chapter 14, "Ship It!"), which in some cases takes just as long or longer than the build process.

Microsoft Sidenote: How the Visual Studio Team Controls All Check-Ins and "Tell and Ask Mode"

The Visual Studio team controls check-ins in another way: the "tell and ask" process. Project managers use this process to slow the rate of code churn and force teams to deliberate about what work items or bugs are fixed or open. This is called triage.

Scott Guthrie is the product unit manager in Visual Studio. He explains triage in his blog:

During tell mode, teams within our division are still given discretion to fix any bugs they want they just need to be prepared to present and explain why they chose the ones they did to the central division ship room. This ends up ensuring a common bar across the division, slows the rate of fixes, and slowly brings up build quality. You might naturally wonder how not fixing bugs could possibly bring up build quality, since this obviously seems counterintuitive. Basically, the answer lies in the regression percentage I talked about earlier for check-ins. Even with a low regression number, you end up introducing new bugs in the product. (And when you have a division of over 1,000 developers, even a low percentage regression rate can mean lots of bugs introduced per week.) By slowing the rate of check-ins, you slow the number of regressions. And if you focus the attention on bad bugs and add [an] additional review process to make sure these fixes don't introduce regressions, the quality will go up significantly.

During ask mode, teams within our division then need to ask permission of our central ship room committee before making a check-in which adds additional brakes to slow the check-in rate. In addition, all bugs in ask mode must go through a full nightly automation run and buddy testing (which takes at least 12 hours) to further guard against introducing problems. Ask mode will also be the time when we'll drive our stresspassing numbers up to super-high levels, and we'll use the low rate of check-ins to find and fix pesky, hard-to-find stress failures.

You can read the entire entry at http://weblogs.asp.net/scottgu. I talk more about processes to control all check-ins into the source tree in Chapter 10, "Building Managed Code."

Release to Staging Servers

After the build is complete and has no errors, it is propagated to the daily build servers, where at least 15 to 20 builds are stored with all the sources and tools necessary to build. Milestone releases also are kept on the server. This is where the test team picks up the build. This is the "secret" to fast development and keeping your developers happy. I realize that most if not all SCC tools can retrieve sources of a certain build but sometimes those tools are clumsy or the labels on the trees are not accurate. So we came up with this staging server with massive amounts of diskspace available and stored our releases on it. It is a lot easier for the development and test teams to search that server than the SCC database.

From the staging servers, the build can go to production. This process is covered in Chapter 14.

Important Definitions

The following sections discuss terms that are specific to Visual Studio but that are used all over the Web and at various companies I have visited.

Solution Files

If you are new to Visual Studio .NET, you probably are not familiar with the term solution. A solution essentially represents everything you are currently working on. Visual Studio .NET uses solutions as containers for individual projects, which generate your system components (.NET assemblies). Solution files maintain project dependency information and are used primarily to control the build process.

Project

In the context of this book, projects are one of three types:

• General development projects The term project in its loosest sense refers to your team's current development effort.

• Visual Studio .NET projects Visual Studio .NET uses project files as containers for configuration settings that relate to the generation of individual assemblies.

• Visual SourceSafe (VSS) projects A project in a VSS database is a collection of files that are usually related logically. A VSS project is similar to an operating system folder, with added version control support.