Regression Testing

Run all the executable acceptance tests as a regression test on every new build in your test environment. The programmers should follow this practice as well. This will probably be at least once per day. If the tests take a long time to run, brainstorm with the programmers on how they could be made to run faster, or split them up so the most critical ones run in the first batch. Programmers need the feedback of the tests to confidently add and refactor code.

Once an acceptance test passes, it should never be allowed to fail. The only exception to this would be if the customer changed her mind so as to make the test irrelevant. If she just changes her mind enough to make the test incorrect, the test must be refactored, along with the code that implements the change. Only when the change renders the test completely unnecessary should it be allowed to fail, at which point it no longer matters and can be discarded altogether.

If you use one system to integrate the latest code and perform unit testing and another to run acceptance and regression testing, make sure you can run tests and report the results equally well in either environment. You'll already have this if you're using the JUnit (or similar) framework and direct calls to interface to the code.

However, if you're running the tests through the user interface with a test tool, minor differences between the systems can cause tests built to run in one environment to fail in the other. Watch out for programmers writing tests that work only in their development environment, and beware of writing tests yourself that work only in the acceptance test environment. Pairing with programmers when working on automation task will help avoid this and give you the critical ability to run tests as needed in both your integration and system test environments.