Smoke Test

The term smoke test originated in the computer hardware industry. The term derived from the practice of powering up the equipment after a piece of hardware or a hardware component was changed or repaired. If the component didn't smoke, it passed the test. After all, chips run on smoke, right? If you let the smoke out, the chips don't work anymore. (That was a joke. Get it?)

In software, the term smoke test describes the process of validating code changes before checking them into the source tree. Smoke tests are focused tests that ensure that changes in the code function as expected and don't destabilize an entire build. If your smoke tests are good, they keep your source trees from catching on fire. (Another joke. Sorry!)

Note

Sometimes, smoke tests are the same as unit tests, but that's another chapter in another book. I don't want to stray off the subject of what tests the build team are concerned about. For more info on testing in general, refer to James A. Whittaker's book How to Break Software: A Practical Guide to Testing.

It's important to state up front that the build team does not own the maintenance of smoke or BVTs, but they own the execution and reporting of the tests and results. The ownership should follow under the development/test/QA group. I will provide suggestions on how these tests are written and designed because I always get asked this question by customers and have gathered some good tips over the years. Since builders are the priests responsible for administering the sacraments and policies of the build religion, it is appropriate for them to keep these guidelines in a place where everyone can easily find them the build intranet page.

The following are some steps you should follow when designing and writing smoke tests:

1.	Collaborate with the developer Collaborate with the developer to understand the following: What has changed in the code? How does the change affect the functionality? How does the change affect the interdependencies of various components?
2.	Conduct a code review prior to smoke testing Prior to conducting a smoke test, conduct a code review that focuses on any changes in the code. Code reviews are the most effective and efficient method to validate code quality and prevent code defects and faults of commission. Smoke tests ensure that the primary critical or weak area identified either by code review or risk assessment is primarily validated because the testing cannot continue if it fails.
3.	Install private binaries on a clean debug build Because a smoke test must focus on validating only the functional changes in updated binaries, run the test on a clean test environment by using the debug binaries for the files being tested.
4.	Don't use mismatched binaries Testing with mismatched binaries is a common mistake in smoke testing. To avoid this mistake, when there is a dependency between two or more updated binaries, include all the updated binaries in the test build. Otherwise, the results of the test could be invalid.
5.	Don't perform exhaustive tests The purpose of smoke testing is not to ensure that the binary is 100 percent error free; that would require too much time. Perform smoke tests to validate the build at a high level. You want to ensure that changes in a binary don't destabilize the general build or cause catastrophic errors in functionality.

Running your tests under debugger is important for debugging hard-to-reproduce issues. You should set up the debugger and make sure that you have the proper symbols before starting the smoke test (see Appendix D, "Debug Symbols," for more information on symbols).

Ideally, you would run your smoke test on debug binaries. Make sure that the developer gives you symbols for the binaries when he gives the binaries to you if this is a private build from his development machine. This is good practice if the test is run before he checks his code in.

A common pitfall of smoke testing is mismatched binaries. This occurs when a dependency exists between two or more binaries and you are not given updated versions of all the binaries. Often, you get registration errors, or the application completely fails to load. If you suspect you have mismatched binaries, ask the developer to build you a complete set of binaries.

Make sure that you enter any issues you caught and didn't fix during the smoke test process into your bug tracking database so that these issues do not fall through the cracks.

Create a record of all smoke tests. Testers must sign off before closing the smoke test record. A good practice is to enter a summary of the testing done both as a record and so that the developer understands exactly what you did and did not test.

Smoke tests are sometimes incorrectly referred to as BVTs. That's not to say that you might have the same test run as a smoke and a BVT, but it is rare that all of your BVTs are the same as the smoke tests in your group.

Let's go on to what BVTs are and explore some tips on how to set them up.