Chapter 14. Ship It!
Philosophy: Chief Crazy Horse called to his men: "Ho-ka hey! It is a good day to fight! It is a good day to die! Strong hearts, brave hearts, to the front! Weak hearts and cowards to the rear."
From the book Crazy Horse and Custer, by Stephen E. Ambrose (Lt. Cmdr. Worf, the Klingon on the Starship Enterprise would have really like Chief Crazy Horse. They used this quote several times on the show "Qapla!" [translated die well/success]).
I have always liked this quote and felt that the phrase, "It is a good day to die!" is taken out of context most of the time. So I wanted to make sure that it is presented here, in its entirety and intentions, to make it clear that it is not some kind of suicidal statement but a brave call. In reference to this chapter, this is the type of mentality that you need when getting ready to release your product. Another term that Microsoft has used in the past is death march not a pretty term, but descriptive. Microsoft doesn't use the term much anymore because people came on board who didn't like the term or softened it up a bit to crunch time or ship mode.
Several milestones and stages are involved in developing a product, including brainstorming features, coming up with the specifications (or specs), writing and testing code, releasing betas, and finally shipping it. This last stage is the focus of this chapter. More specifically, it's about what should happen just prior to shipping and after the product is out the door.
Microsoft Sidenote: Jim McCarthy's Rule 21
I quoted Jim McCarthy in the first chapter and reference him again here because he is one of the most entertaining speakers I have ever seen. Although Jim left Microsoft in 1996, his "21 Rules of Thumb for Shipping Great Software on Time" memo is still floating around and quoted. It's a true classic that includes gems such as "Don't flip the Bozo bit" (basically, don't jump to conclusions on how smart or dumb someone is), "Lucid ignorance" (know what you don't know one of my favorites), "If you build it, it will ship" "conversely, if you don't, it won't" (daily builds are the answer).
What is included here is what I think he elegantly put as Rule 21, "Get the team into ship mode."
There is a moment on every development project when it is ideal for a team to enter ship mode. Ship mode is a high-performance period characterized by efficiency and determination. It is a period of flow. Before a team can enter ship mode, several prerequisites must be satisfied.
Shipment must be the next milestone.
Everybody (or nearly everybody) must believe that achieving the milestone is possible.
All members of the team must understand precisely what they must do prior to shipping. All unknowns are factored out.
Management must lead the team to ship mode by entering ship mode first. That is, superfluous management hoo-ha is eliminated, the manager's awareness of detail climbs, fire-drills, and other deprioritizing activities are eliminated entirely, and tremendous focus is brought to bear.
The team must desire to ship. Generally, a complete awareness of the effect of shipping (or not shipping) will create desire.
The team becomes especially vigilant about thinking things through and looking for traps. Check-ins are made with extra precaution. Stabilization of the product is the principle goal. All development is complete but for bug fixing.
The endgame, the last stage of ship mode, is different yet again. It is conceptually a very simple exercise. There is a list of activities. When every activity on the list is complete, you ship. Though the list might have hundreds or thousands of items, it is still just a list. There is no time for any effort that does not contribute toward completing the items on the list. Everybody is expected to complete their items as promised. As unanticipated items arise, after appropriate resistance, they are put on the list.
A daily meeting should be established, with final decision makers in attendance. Agenda is ad hoc, assembled at the beginning of each meeting. No item is postponed that can be handled now. The team is aware that all issues can be brought to this meeting for expeditious remedy. Management is involved, leading the team toward their goal.
The goal is an acceptable quality level at ship time. Only showstopper bugs should be addressed at all. Showstoppers are bugs that will either effect more than a handful of users or will cause unacceptably serious errors. Cosmetic changes, performance enhancements, new functions are not appropriate changes. The purpose of beta feedback during this period is to prove there are no showstoppers, provide advance warning of unanticipated market reaction, and provide input to the next release.
Understand the range of quality that is acceptable to your customers. How many low-priority bugs did your product ship with last time? Was it a problem? Are the customers better off with this product including this bug? Since destabilizing the software is more of a problem than most bugs, be very careful about which bugs you fix. This is why we have "ReadMe" s and bug lists.
Ship mode is basically a succession of daily milestones climaxing with the product's shipment.
If you would like to see or hear more of Jim, he and his wife Michele formed their own consulting firm after they left Microsoft in 1996. Or you can purchase either of their books: Software for Your Head, or my favorite, Dynamics of Software Development.
Okay, so now that you are in ship mode, what should everyone be working on at this point? The following list is typical but not always the case:
Most of the developers have moved on to the next version or other projects, with the exception of any developers who have a showstopper or Severity A, Priority 1 bug assigned to them.
The Central Build Team should be taking few code check-ins on the current product source tree because it is close to shipping, and most of the critical bugs should have been already been found.
The testing or the quality assurance (Q/A) group is scrambling trying to find critical bugs.
Marketing or upper management is putting a lot of pressure on everyone to make a somewhat random date that they decided the product would ship.
Activities cited in the previous list are a healthy sign of the successful progress of shipping the product. If you follow the suggestions that have been given up to this chapter, you will find yourself in this situation and be capable of going the distance and getting the product out the door. I get concerned when the previous list is convoluted with problems such as the following:
Developers are not able to work on future versions because the source trees are not properly set up.
The Central Build Team has too many build breaks to get a build out every day.
Testing or Q/A does not have an idea of what code they really are testing.
Upper management pulls rank and just ships the product when it really isn't ready.
Because I've spoken about these points throughout this book, let's address how the build team can keep the work flowing after or just prior to shipping.
Microsoft Sidenote: Small Business Server Tree
Figure 14.1 is an actual diagram I used when I was on the small business server (SBS) team. A developer (in this example, Jony) came to me (the build team) to let me know that he was working on the next release of SBS and asked how he should proceed.
Figure 14.1. Small business server tree.
Jony was one of three developers working on this project. The following text is taken from an e-mail that explained to the SBS team how the source trees were to be set up as they moved forward and to allow Jony to work on the new code with as little inconvenience as possible. It worked well, and the product has been successful for Microsoft.
Because at this time, we have already shipped the U.S. 4.0a SBS product, any new check-ins from now until we ship all the international versions will be considered QFE check-ins to 4.0a U.S. (Note: We need to document all checkins.)
After we ship the international versions of 4.0a, this tree will become the NEW QFE tree because it includes all of the 4.0 source.
We will save the 4.0 source on tape but kill the tree and use the tree for 4.5. I really doubt there will be any 4.0 QFE fixes that we can't get out of the 4.0a QFE tree.
When we met last, Jony was going to keep a 4.5 tree in his office. If he has been doing this, we need to merge the 4.5 check-ins into the new tree (\\ orville\razzle) after getting that tree in sync with the final 4.0a ship sources.
Let's meet again to discuss this and make sure everyone is in agreement.
This sidenote is just an example of how simple setting up source trees and communicating it to everyone can be. The example here really only applies to small teams. Larger teams need to provide more detail and formal announcements.
Because there should be a lot of down time in the build lab as the check-ins slow down, the build lab should be working on a new build process. The build lab should roll out this process just after the final bits of the product are sent to manufacturing or released to the Web. Part of the process can include the following:
New tools and how to use them
Explanation of the source tree restructuring
Build schedule for the next month
Any changes in the check-in policy
The build team can help accommodate the testers and upper management by being as efficient as possible with communications and delivery. Many of these responsibilities are outside the scope of the build team, so this is all I will say about these two points.
After the team has completed all the daily milestones and everyone has given the product his blessing or approval, it is time to get it out the door. In the old days at Microsoft, this was a manual process, and we had our own manufacturing plant. Nowadays, the release is outsourced right after the final verification stage, which is the next topic.