Chapter 10. High-Level Acceptance Test Estimates

During release planning, programmers estimate the relative effort and technical risk for each story, using some kind of point system. At the start of each iteration, customers choose stories whose points add up to no more than the total number the team has said they can complete for that iteration. The units used to estimate stories aren't important; what's important is the consistency of the estimates, so it's clear how much relative effort each story requires. If a story turns out to take ten times the projected effort, you're going to have one messed-up iteration. The estimates don't have to be exactly right, but it helps if they're in the ballpark. Estimating the effort to complete high-level acceptance testing for each story will help make the story estimates accurate enough for useful iteration planning.

Acceptance testing is going to take time, regardless of whether testers, programmers, the customer, or some automated tool is doing it. Programmers typically include unit test time when they estimate a story, but they don't always think too much about acceptance testing requirements, which may be allocated as a fixed block of "a day or two" at the end of an iteration. They also may forget to allocate time for fixing defects and retesting.

When we planned our road trip, we built in time for everything that marks a successful trip: coping with unexpected delays, visiting some interesting sights, and making regular rest stops, so we can remain alert. On our XP test drive, we want to build in enough time for a safe and sane journey.

Here's why we want to consider acceptance tests when we're estimating stories during release planning. The time to develop and unit test a story is highly dependent on the story. This is true for acceptance testing time as well. However, they don't necessarily track together. Some things are a lot harder to code than to test; some are the other way around. The only reliable way to make sure you avoid a big problem near the end of an iteration (or one that bleeds into the next iteration) is to have some estimate during release and iteration planning of the acceptance-test time required for each story. Then, as you pack the iteration bin full of stories, you can make sure the acceptance test time fits as well.

The high-level acceptance tests you've already defined pay off in more ways than one. They give more detail about the requirements for the story, which helps the programmers estimate development time more accurately. These high-level test definitions also help you estimate the time required for acceptance testing.

If you think too much about how to account for acceptance testing time, it can get confusing. It seems that development time is the main driver of the iteration, since you can't execute acceptance tests before the code is ready and the time spent preparing for acceptance testing overlaps development time.

Sequencing problems may also arise. You may not be able to do some of the preparation until another story in the iteration is completed. That part of the preparation becomes an obstacle if the needed story is not completed enough in advance. Certain tests may only run in a limited window. For instance, a 6,000-concurrent-user test may need access to a production database that's available only late at night or on weekends. And what about when a test fails and you have to rerun it? How do you account for that?

Well, we've found it best not to worry about any of this, especially at this point in the process. All the dependencies, overlaps, and contingencies would be really difficult to figure exactly. If you did, you'd be wrong anyway, because everything would change by the time you actually got there.

The overlap between acceptance-test tasks and development tasks is no different from the overlap between different development tasks. Just estimate the time the testing tasks will take in the same units of measurement as the development tasks. As long as acceptance-test estimates get into the planning process, the team's velocity will account for them, and the fact that some tasks overlap or appear to be dependent on others won't matter.