Section 12.5. Testing and Quality Assurance

12.5. Testing and Quality Assurance

Testing is another major component of any software development process, and any successful project has some sort of policy in place for assuring that a sufficient amount of testing is done. Although most of the actual testing is beyond the purview of Subversion, a good set of policies for project organization within the repository can lend strong support to your quality assurance process.

Some of the policies that I discuss in this section were already touched upon in the previous sections of the chapter, but here they are brought together into a coherent structure. I note this not to assure you that I didn't forget that I already mentioned them, but rather to stress the total interdependency of the various policies that make up a total software development process. The policies that you ultimately make a part of your strategy for using Subversion in your software development process are not a set of unrelated policies tacked together. Instead, your interaction with Subversion should be considered for how it fits into the whole of your software development process.

12.5.1. The Parts of Testing

There are a number of different types of testing that make up the full gamut of testing that a project undergoes during its development. Not every project uses all of the testing methods that I talk about in this section, but I find it unlikely that any successful project has ever avoided all of these methods (as much as some have tried). For each of these different areas of testing, I will talk about what they are, and what sort of policies you might adopt to integrate Subversion into the process.

Individual Developer Testing

In most cases, developers do their own personal testing of the code that they write, before integrating it with the main public development branch of a project. Even in projects where it is difficult or impossible for an individual developer to personally test her code against the full project before committing, most developers will do some sort of private testing to ensure to the best of their ability that the code does what it is supposed to do. Often, this involves local test harnesses or throwaway unit tests.

For the most part, Subversion doesn't play a particularly big role in individual developer testing. Most such testing is done on the developer's personal workstation, in her working copy, without anything getting committed to the repository until the testing is finished. However, there are a few policies that you can use to support individual developer testing.

• Give developers a place to version their personal test code. When developers write local test code, it often consists of throwaway tests that either get erased or tossed in a random directory on the developer's hard drive after they've been used, or they continue to sit in the working copy as unversioned files. If developers have a place in the Subversion repository dedicated to storing those test programs, they will be more likely to keep them around and reuse them in the future.

• Use task branches for developer work. By using task branches, the individual developer's work is segregated from the rest of the project until it's tested and ready to be merged back into the main branch. That makes it much easier for the developer to perform incremental local testing on the code as it's written, without interfering with the rest of the project. This is especially useful if the developer can't do local testing against the full project.

• Allow developers to create private branches, for testing multiple potential solutions to a problem. In the case of most nontrivial development tasks, there are many different ways that a problem can be solved, and it's not always clear to the developer what the best method is. If you allow the developer to make private branches for testing different potential solutions, it is easy to write two versions of the solution and then run tests that compare the two to see which is better.

Automated Unit and Regression Testing

Automated tests are a big part of many software development processes. They generally make up a comprehensive test suite that exercises a project in order to make sure it's doing what the specification says it should, and as changes are made to the project, the test suite also helps to ensure that previously working components of the project continue to work. How often these tests are run depends on an individual project (and is often dependent on how long the tests take to run). When integrating your automated testing process into Subversion, the biggest questions to answer are "Which branches will the automated tests be run on?" and "What role, if any, will Subversion play in triggering those tests?"

One way to trigger automated tests is to have them run whenever someone commits something to the repository, or even to a specific branch. The commit can trigger a hook script that runs the test suite and reports any errors to the author of the commit and anyone else appropriate. This could work well as a policy if you have a main development branch and use task branches for feature development. Whenever a feature is deemed complete enough for the main branch, the task branch can be merged in and committed. The automated test suite could then run regression tests on the full main branch to make sure everything still works. Or, if you want an extra layer of redundancy, you could have an intermediary branch. When a task branch is finished, it can be merged into the testing branch. If all of the regression tests pass, the testing branch could then be merged (or copied) over to the main development branch.

The QA Team

Some projects (especially large ones) make use of a dedicated quality assurance team whose sole job is to rigorously test the work of the project's developers. As I have touched on previously at several points, there is a lot that you can do organizationally with your repository to support the interaction between a development team and a testing team. The policies that you set in regard to support for testing in your Subversion repository not only can support your QA process, they can help to shape it.

One of the areas where you will find a wide range of policy choices for supporting QA is in the use of a branching/merging structure that supports the transfer of data from developers to testers in an organized manner. I talked previously about what there is to know on this subject. If you would like to review my thoughts on the subject, you can reread Section 12.1, "Effective Branching and Tagging."

Another place where you can use Subversion to support your QA testing process is through the use of properties. For example, you could set a property, qa:tested, to label the current testing state of a file or directory. Or, you could use a property to allow the tester to "sign off" on a section of code. You could even set a property to a digitally signed md5 sum of the repository revision on which the tester is signing off, if there are safety critical issues for which someone is required to take legal responsibility.^[1]

^[1] I am not a lawyer, and make no claim as to how well this would stand up legally. If you are doing something critical to require that level of liability, I strongly suggest speaking with a real attorney.

Beta Testing

Beta testing is the point in a project's life cycle when it is sent out to real users to allow them to stress test the product under real (or at least semi-real) working conditions. In many ways, beta versions are handled just like a final release. Generally, unless it is an open source project, the release is binary only, and the beta testers have no knowledge of the inner workings of the application, or even any knowledge of programming. However, since it is still not a final release of the project, you are usually expecting a large number of bug reports to come back as a part of the beta test process. How these bug reports are supplied to you is, of course, outside the purview of Subversion, but it can be helpful for you to have some policies for what you do to the repository in response to the bug reports.

When you release a beta, you will almost certainly want to create a tag of the revision of the project that made up the beta release. That tag can then be used as a basis for creating branches dedicated to fixing issues raised in bug reports. You can either create task branches for each reported bug or you can create a single beta branch for fixing the bugs and then have each individual developer create branches as he sees fit.

Shipping the Final Product

It may seem a little counterintuitive to have a section on shipping the final product in a section on testing the project during development, but the fact is that shipping the product is really nothing more than another step in the product's development path. Except in very rare cases, no product ships without flaws, and flaws mean that bug reports will continue to come in even after the product is in full release. To that extent, a full product release is really no different from a beta release in the eyes of Subversion, and the policies you set in place for handling beta tests are likely the same policies that you will want to have in place for maintaining the full release of the project.