Build Verification Tests


BVTs are automated suites of tests designed to validate the integrity of each new build and the basic functionality of the build before it is released for more in-depth testing. A BVT is not a comprehensive test; it is a type of test that is designed to do the following:

  • Validate the integrity and testability of each new build.

  • Ensure basic functionality for continued in-depth testing.

  • Test critical functionality and the highest priority use cases or user scenarios.

BVTs must be able to determine if the general quality of the build is sufficient for self-hosting, or if the build is unstable and should be used only in testing environments. Unit tests are sometimes used for BVTs, but only if they are critical to the execution of the program.

Define the Scope or Span of the BVTs

It is important to set limits of what is tested and how long BVTs should run. This limit is determined by the amount of time you have between the build completing and when the reports of the BVTs are due. As a general rule (and what we have found to be a good guideline), three hours should be sufficient for testing. In order to determine what tests need to be included in a BVT suite, you need to determine the high-risk areas of your product.

Consider the following factors when assessing priority and high-risk areas:

  • Probability of failure How likely is failure?

  • Severity of failure What are the consequences of failure?

  • Visibility of failure Will the user see a failure?

Now that you have identified the high-risk areas and some tests that should be included in a BVT suite, it is time to start setting up the boundaries of the BVT suite.

When setting up BVTs, the following recommendations should be followed:

  • Establish a quality bar BVTs validate the quality of the build. Their purpose is to quickly test and assess blocking issues that would make the build unusable before further testing continues. BVTs should determine the quality and stability of the build for self-hosting by everyone on the team, or whether the build should be limited only to test environments.

  • Define a process for immediate problem resolution Consider bugs that are identified in the BVTs as top priority. In the case of a BVT failure that makes the build unusable for further testing, identify the bug that caused the failure as a hotfix bug. Mandate quick turnaround time for these.

  • Automate the BVT test suite Because a successful result from the BVTs signals the start of more comprehensive testing, it is important to quickly get the results of the BVTs. The same set of tests also needs to ensure the baseline functionality for further testing, so consider automating the BVT suite if it is fiscally possible.

  • Limit the duration of the BVTs BVTs are not extensive test suites. The purpose of a BVT is simply to determine the validity of the build and the stability of high-priority functional areas. Don't attempt to run entire test suites during the BVTs because this can be time-consuming. The more time a builder spends running BVTs, the less time the build is exposed to a greater number of testers.

  • Don't run an entire BVT suite on partial builds The BVT suite is inappropriate to validate the quality of partial builds. Developers should perform unit tests to validate code fixes or changes. In addition, developers should run subsets of the BVT suite on their private builds prior to checking in code for the next build to ensure that the new code does not break the build or the BVT test pass.

  • Run the BVT in a controlled environment Control the environment for your BVTs by taking the following precautions:

    • Install each new build on a "clean" test environment to keep the focus of the test on the build and not on external influences.

    • Ensure that no heavy prerequisites are required for the setup of the BVT environment or framework.

    • Make the BVT suite portable so that it can run on multiple language versions of the same product.

  • Track BVT results The BVT suite should provide a baseline measurement of build stability for comparative analysis:

    • Compare results between test passes to provide information on potential problem areas.

    • Track the results of the BVT pass to provide quality trend information.

    • Use BVT metrics to establish release-to-test and release-for-self-host quality criteria.

    • Create a link from the build intranet page to the BVT results.

  • Review the BVT suite periodically Update the BVT suite as necessary to validate the integrity of the build and ensure its essential functionality. Update the suite by removing tests that are no longer applicable. Only incorporate tests in the BVT suite that help to accomplish its two primary objectives:

    • Evaluating build integrity.

    • Verifying basic functionality.

  • Don't constantly change the parameters of the BVT suite The BVT test suite should provide a baseline measurement on build integrity and basic functional quality. Although features might change, don't modify the BVT suite unless you need to modify it to more effectively validate either the integrity of the build or changes in high-priority functionality. Changing the parameters or variables in the BVT suite skews the baseline metrics and other measurements, such as the quality bar.

  • Create a service level agreement (SLA) with program management and development teams Collaborate with program management and development teams to agree about the BVT process, defect resolution and timelines, the quality bar, results reporting, and ownership. Use SLAs to establish and record the BVT process and responsibilities.

Since you now have some good recommendations on how to set up your BVT suite, you should make sure that your BVTs are automated tests so they will be less prone to human interaction errors. Also, you should not release a build until all of the BVTs have passed. A good way to do this is to hide the release shares so that overzealous testers or developers don't pick up the build too early.

You really need to customize BVTs to the product. The testing team is always in the best position to determine what tests should be in the BVTs. Some basic tests apply to the actual build process itself. These tests can be broken into a subgroup of BVTs that are focused on the build (e.g. Build BVTs). The following are two tools that focus on the build itself that you should use after every build:

  • Visual File Information tool (vfi.exe) Use this tool to validate the build and the integrity of all the files in each build. You can download this tool at http://www.microsoft.com/downloads. Search for "Windows Resource Kit tools." Check each file for the following:

    • Correct version information

    • Correct time/date stamp

    • Appropriate file flags

    • Correct cyclic redundancy check (CRC) keys

    • Correct language information

    • Potential ISO 9660 filenaming issues

    • Viruses

  • VerCheck tool (vercheck.exe) Use this tool to validate installed files and registry keys after setup of each new build on a clean test environment. You also can download this tool at http://www.microsoft.com/downloads. Search for "Windows Resource Kit tools."

VerCheck also lists all files matching a given pattern and reads out the internal description, version number, and other properties. The entire list can be copied, stored, or printed for a later comparison. Each file can be deleted if no longer needed.

Several other valuable tools are available in the resource kit (for free!) that you might want to use for testing. These are just the two basic tools that need to be included in your BVT suite.

Microsoft Sidenote: Tips from an Old Test Manager

Since we have been focused on testing in this chapter, I thought it would be appropriate to include this classic list from a 16-year Microsoft test manager. You may want to make sure your Q/A or test team reads this. Developers will have an appreciation for it as well. In her own words:

For most of my career, I've been a test manager. I've come to realize that the job of a test manager is really quite simple and consists of three activities:

  1. Say "no" to development at least once a day.

  2. On a regular basis, complain that the project is off track, the schedule is ludicrous, and the quality is terrible only to be told to "lighten up" by the program manager.

  3. Find the killer metric to measure the product.

What I've learned that I'd like to pass along (the condensed version):

  1. If [people] really want you to do something, they'll ask at least twice. The first time they ask is just to see if what they're asking for sounds good to them. If you do what they ask when they've only asked once, you're not playing the game right.

  2. The bug resolution fixed is more a wish than a statement of fact.

  3. You can still be friends with your developer even after threatening him/ her with a baseball bat.

  4. A schedule is not a sausage casing designed to be stuffed to the breaking point.

  5. 95 percent of reorgs have no impact on the people actually doing the work. The other 5 percent of reorgs mean you'll be looking for a new job. The trick is to know which one is about to happen.

  6. It's fun to have one's mistakes made into a Dilbert cartoon.

  7. If no one else knows how to do what you do, leaving your group is going to be tough.

  8. A spec is like a fairy tale. Only the naïve and childlike believe it.

  9. The first time a question is asked, it's fine to say, "I don't know." When the same question is asked again a week later, you'd better have an answer.

  10. Metrics are dangerous. No matter how carefully you caveat them in the original mail, they are interpreted to mean something completely different.


BVT States

It's important to talk about different states in which BVTs can exist. There are three basic states:

  • An Active BVT is one that stops a self-host build from being released if it fails. For the most part, when you see a reference to a BVT, it refers to an Active BVT.

  • An Inactive BVT is one that does not stop a self-host build from being released if it fails. Such tests are usually in this state because they are new and have not yet passed against a build. Sometimes they are in this state because they contain errors in the test script or because they show a bug that does not have a reproducible scenario.

  • A Disabled BVT is one that is never run. Typically, BVTs are disabled when they are still under development or the test is made obsolete for some reason.

Keep in mind that BVTs are meant to ensure that major functionality of a shared feature or an application is not disabled to the extent that you cannot test the functionality. Keep this in mind when you're creating a BVT. If a BVT is run and fails but does not result in a BVT-blocking or self-host designation, it probably shouldn't be a BVT.

Did I confuse you about the differences between BVTs and Smoke Tests? If I did, here is a basic list of the differences:

  • BVTs are a subset of smoke tests.

  • For some components, unit tests are used for BVTs and smoke tests.

  • The build team or a BVT team runs BVTs. Testers or developers run smoke tests.

  • Smoke tests are used for new functionality, whereas BVTs are usually reserved for core functions of the code that are more critical to the stability of the product.

  • BVTs have a limited time frame in which to run. Smoke tests should be allowed to work without time limits.

Microsoft Sidenote: Testing Guide (See Appendix C)

Another very popular question that I get from customers is: Do you have any resources on testing? Well, I do. There is a Test Guideline that has been floating around the developer division (devdiv) the people that bring you Visual Studio that Josh Ledgard has pulled together. I have reprinted the guide in Appendix C. You should not consider it to be a compressive test matrix, but it is a good starting point for a test plan. As with anything in this book, if you see something we're missing or you see any mistakes, please let me know and I will add to this "testing snowball."




The Build Master(c) Microsoft's Software Configuration Management Best Practices
The Build Master: Microsofts Software Configuration Management Best Practices
ISBN: 0321332059
EAN: 2147483647
Year: 2006
Pages: 186

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net