Chapter 3. Daily, Not Nightly, Builds

Philosophy: The build team drives the product forward.

Vincent Maraia

It's amazing how many articles and books I have read that recommend nightly builds. Unfortunately, they mean this literally. Whenever I talk to different software companies or Microsoft product groups that do nightly builds, I always ask, "What is the success rate of your nightly build?" Most of the time, I hear "10 to 20 percent success with zero errors." I figure the people who tell me that they have 80 to 100 percent success rates are either lying or compiling very little code every night.

I understand that the beautiful vision of a nightly build is that the build will be ready to go in the morning, all the first and second stage tests will be run, and if you have a really efficient process, the build will be deployed to the developer and tester's boxes. As a result, everyone can crank away on finding and fixing new bugs and getting the new code checked in as fast they can get out of bed and connect to the network.

Well, this is not reality when it comes to software builds. We tried the nightly builds at Microsoft in various groups. We found that you end up having some build hero up late at night or early in the morning fixing a build break or two. This is usually the only way that everyone can have their doughnut in the morning as they download the newly released nightly build. But for some reason, people keep trying it again.

Microsoft Sidenote: When Nightly Builds Do Work

The daily build process took place prior to the NT build teams switching to the VBL process, discussed in Chapter 2, "Source Tree Configuration for Multiple Sites and Parallel (Multi-Version) Development Work." The VS and NT teams both start builds in the evening in their main lab because build breaks are so rare in the golden tree. This is a result of everything going through some kind of pre-build test during the integration process.

If you have this type of setup, you can still do a daytime daily build in your VBLs and kick off a nightly build on your golden tree. But don't do nightly builds unless you can guarantee a 95 percent or higher build success rate.

Nightly builds actually promote bad behavior and carelessness for developers who check in code. What usually happens is that people get used to the fact that the build breaks almost every night. The developers count on the Central Build Team to fix the breaks in the morning, which buys them some buffer time to try and get last-minute changes in. I recommend running a build at a consistent time during the day when developers are around so that they can fix their build breaks before going home. When the product gets close to the shipping date, you should be building on the weekends, too.

As part of the daily build process, you should publish a regular build schedule. This should include the last check-in time, the time that the build will be completed, and the time that initial build verification tests (BVTs) will be completed. Here is roughly how this would look:

9 to 10 AM	Decision makers meet at WAR or Ship meeting
11 to 2 PM	Check-in window open
2 to 5 PM	Build product (development teams triage their bugs in private meetings)
5 to 6 PM	Release product to test shares
6 PM to 9 AM	Run automated tests at night

When any part of this process is broken, the person who checked in the defective module that broke the build should be published via build intranet page or e-mail. The build intranet page should be a collection point for all the relevant documents, policies, schedules, links, and other general information, such as new hire information. Everyone on the product team should be able to reference this site, and most will make it their default start page when they bring up IE. Years ago, we used a page similar to the one in Figure 3.1 in the Windows NT group.

Figure 3.1. Sample build intranet page.

Microsoft Sidenote: Famous Religious Poster

We used to have a poster in the Windows NT Build Lab that read: Remember the 11th Commandment:

"THOU SHALL NOT BREAK THE BUILD"

Microsoft Sidenote: When Are We Going to Ship?

If you really want to know when your product is going to ship, just ask your Central Build Team or someone close to the daily build. The NT group used to have a whiteboard full of predicted shipping build numbers by different people in the project. The people who always came closest to the actual build number that shipped were either the builders or the operating system core developers.

Because we were the closest people to the daily grind of accepting and building check-ins, we were more scientific about calculating the ship date. The core developers were good because they seemed to always be in the build lab and used the method described in the next paragraph.

It is really an easy math problem. Say that you are taking 60 check-ins per day, but 30 new bugs are being opened. The net gain is 30 closed bugs per day. If you have 100 bugs in the database, you should ship your product in 4 to 5 days as long as no new bugs are opened on the last day.

This was the simple equation we used to guess ship dates. We didn't use the crazy estimates we got from product or program managers who used a Gantt chart loaded with guesses from development managers who were in turn getting overambitious guesses from their development teams about how fast they could write code.

Figure 3.2. Moses.

The warning here is that if you break the build you will have hell to pay.