Chapter 12. Build Verification Tests and Smoke Tests

Philosophy: Eating our own dog food.

Paul Maritz, former vice president of systems at Microsoft

What does this dog food philosophy mean? As explained in the earlier chapters, we at Microsoft test our own software on our developer, IT, and in some cases home systems before we ship a product out the door. In other words, we eat our own dog food. Why is this relevant? The following sidenote illustrates why.

Microsoft Sidenote: How BVTs Evolved

In the early days of NT, whenever a build was released, testers would immediately pick up the build and start pounding on it with their standard and extensive suite of tests. Because NT is an operating system and does a lot of kernel-level calls, it always pushes the technology envelope. Because we "ate our own dog food" occasionally, a build that had a violent bug in it would take down several servers in our IT department. If the servers that went down were e-mail or database servers, this bug could be very troublesome and costly to the company. To keep this from happening again, we required that a certain quality level be achieved before we released the software for "dog food testing." Build verification tests (BVTs) were one way of achieving this. BVTs were also helpful in determining the quality of the build, because if the build could not pass the first level of testing the BVTs you knew that the quality of the released bits was low.

It's possible to work on a Web application or a small standalone application that doesn't take your whole company down if there's a critical bug, but don't let that stop you from implementing BVTs and smoke tests. Every software application needs them, and they should always be the first round of tests that are run on your built binaries.

Some groups have teams dedicated to running and reporting the results of BVTs, whereas others assign this task to the build team. This chapter covers how to set up BVTs and smoke tests and what the difference is between the two. One of the questions I am often asked is, "What should we use for BVTs or smoke tests?" The correct answer? It depends, and I am really not the person to answer that question. The correct people to ask are your testing personnel, because they know which critical pieces of the product need the most testing.

The way that we gathered BVTs to run in the build lab was to ask the testers to give us a certain number of tests that ran for a fixed amount of time, say three hours. We did this for every component of the product.

Every company should run some standard tests that not only try out the application, but also try out the integrity of the build. I mention those tests later. I also mention a good "Testing Guide" (complete guide in Appendix C) compiled by Josh Ledgard, a program manager in the developer division at Microsoft. It is a great starting point for building a test plan.

I would like to define two terms that are used loosely to label a build once tests have been run on it. Feel free to customize the definitions to what you and your development/test team would like:

• Self-Test Builds This is a build that passes all unit tests and BVTs. The binaries are not optimized and have minimal testing done on them. These builds are usually picked up by developers or testers that are awaiting a build release so they can verify their last check-ins.

• Self-Host Builds This build is also known as a "dog food build." By self-host, we mean that you not only install this build on any machine you have, but you also use it on a day-to-day basis as if it were a released product. A self-host build requires a certain acceptance level of tests passed, which include unit, BVT, regression, and stress. These builds are a higher quality than the Self-Test builds; there is even a tracking field in our bug database that states whether the bug needs to be fixed so the build can be marked Self-Host (e.g. Fixbyrel = Self Host).

Another term I would like to define is unit test. A unit test is a test that checks the functionality of a particular code module or component. Usually, unit tests make good BVTs or smoke tests.

Let's look further into smoke tests and BVTs, which are really a subset of smoke tests.