Running Tests

Watch for falling objects! Don't let the developers throw code over the wall at the end of the coding phase. You established a relationship with the developers and architects during the system design. Brainstorm with them about how they can deliver bits and pieces to you to test early. Pair with them to do integration testing. Budget time in your schedule for starting testing as early in the cycle as you can.

The advantages of testing early in the cycle will be apparent to everyone. As long as you have control over your own test environment, so changes by programmers can't catch you unawares and potentially slow down your testing, you can test parts of the software that aren't ready for prime time and give the programmers valuable feedback. Programmers know defects are lurking in their code. It's helpful for them to find out if the major features work without blowing up or that the architecture performs as expected under heavy loads.

You might agree to just do "smoke" testing at first and give feedback without opening defect reports. You're not in a competition to see who can find the most bugs; you're trying to ensure that when the programmers think they're finished, they really are.

Do whatever you can to encourage the programmers to write unit tests. The advantages of unit tests and test-first programming, as described in Extreme Programming Explained, apply no matter what methodology you're developing with. Automated unit testing has become a widely accepted practice. It will help the programmers gain courage and go faster. Effective unit testing means that when you start doing functional or system testing, you aren't spending your time finding defects, especially regression bugs, that could have been ferreted out at build time by the unit tests. All software development projects, not just those using XP, will suffer adverse consequences if they fail to write and use effective automated unit tests.

Automated unit testing, especially the concept of test-first design, can be a challenge for the uninitiated. If programmers haven't written automated unit tests before, it's tough to get them to start. You need management on your side for this one. Offer to help the programmers figure out the best way to unit test. Introduce them to XUnit if they're not familiar with it. If you have to, bring in an expert test-first programmer to show how it's done. Document the results of automated unit testing. You'll see a drop in the number of defects you find in post-development testing. The defects your acceptance tests find will be a different type, not isolated to one piece of code but produced by a combination of factors.

Ask the business experts to do acceptance testing. They can use the test cases they helped you produce that they identified as crucial to acceptance, or they can just do their own thing. Either way, you'll get valuable feedback, and they'll have a level of confidence and comfort that they're in charge of their own destiny that what gets put into production isn't something totally different from what they requested.

You and the business experts can start planning what changes will be needed as soon as changes are allowed usually after the production release. You could be really subversive and show the business experts' desired changes to a programmer and get an estimate of how much time will be needed to implement them. Who knows if the programmers have written automated unit tests to give themselves more courage and confidence to make changes, they might even be willing to make the changes before the production release. You don't mind you have a lightweight, flexible test design, and you can easily modify your automated test scripts.