9.1 Structured Expert Judgment
9.1 Structured Expert Judgment
Individual expert judgment does not have to be informal or intuitive. Researchers have found significant accuracy differences between "intuitive expert judgment," which tends to be inaccurate (Lederer and Prasad 1992) and "structured expert judgment," which can produce estimates that are about as accurate as model-based estimates (Jørgensen 2002).
Who Creates the Estimates?
For the estimation of specific tasks—such as the time needed to code and debug a particular feature or to create a specific set of test cases—the people who will actually do the work will create the most accurate estimates. Estimates prepared by people who aren't doing the work are less accurate (Lederer and Prasad 1992). In addition, separate estimators are more likely to underestimate than estimator-developers are (Lederer and Prasad 1992).
| Tip #43 | To create the task-level estimates, have the people who will actually do the work create the estimates. |
This guideline is for task-level estimates. If your project is still in the wide part of the Cone of Uncertainty (that is, specific tasks haven't yet been identified or assigned to individuals), the estimate should be created by an expert estimator or by the most expert development, quality assurance, and documentation staff available.
Granularity
One of the best ways to improve the accuracy of task-level estimates is to separate large tasks into smaller tasks. When creating estimates, developers, testers, and managers tend to concentrate on the tasks that they understand and deemphasize tasks that are unfamiliar to them. The common result is that a 1-line entry on the schedule, such as "data conversion," which was supposed to take 2 weeks, instead takes 2 months because no one investigated what was actually involved.
When estimating at the task level, decompose estimates into tasks that will require no more than about 2 days of effort. Tasks larger than that will contain too many places that unexpected work can hide. Ending up with estimates that are at the 1/4 day, 1/2 day, or full day of granularity is appropriate.
Use of Ranges
If you ask a developer to estimate a set of features, the developer will often come back with an estimate that looks like Table 9-1.
| Feature | Estimated Days to Complete |
|---|---|
| Feature 1 | 1.5 |
| Feature 2 | 1.5 |
| Feature 3 | 2.0 |
| Feature 4 | 0.5 |
| Feature 5 | 0.5 |
| Feature 6 | 0.25 |
| Feature 7 | 2.0 |
| Feature 8 | 1.0 |
| Feature 9 | 0.75 |
| Feature 10 | 1.25 |
| TOTAL | 11.25 |
If you then ask the same developer to reestimate each feature's best case and worst case, the developer will often return with estimates similar to those in Table 9-2.
| Feature | Estimated Days to Complete | |
|---|---|---|
| Best Case | Worst Case | |
| Feature 1 | 1.25 | 2.0 |
| Feature 2 | 1.5 | 2.5 |
| Feature 3 | 2.0 | 3.0 |
| Feature 4 | 0.75 | 2.0 |
| Feature 5 | 0.5 | 1.25 |
| Feature 6 | 0.25 | 0.5 |
| Feature 7 | 1.5 | 2.5 |
| Feature 8 | 1.0 | 1.5 |
| Feature 9 | 0.5 | 1.0 |
| Feature 10 | 1.25 | 2.0 |
| TOTAL | 10.5 [1] | 18.25 |
|
[1]Some statistical anomalies arise when you simply total the Best Case estimates and the Worst Case estimates. Chapter 10 discusses these in detail. | ||
Open table as spreadsheet When you compare the original single-point estimates to the Best Case and Worst Case estimates, you see that the 11.25 total of the single-point estimates is much closer to the Best Case estimate of 10.5 days than to the Worst Case total of 18.25 days.
If you examine the estimate for Feature 4, you'll also notice that both the Best Case and the Worst Case estimates are higher than the original single-point estimate. Thinking through the worst case result sometimes exposes additional work that must be done even in the best case, which can raise the nominal estimate. In thinking through the worst case, I like to ask developers how long the task would take if everything went wrong. People's worst cases are often optimistic worst cases rather than true worst cases.
If you're a manager or a lead, have your developers create a set of single-point estimates. Hide those estimates from them. Then have the developers create a set of Best Case and Worst Case estimates. Have them compare their Best Case and Worst Case estimates to their original single-point estimates. This is often an eye-opening experience.
This exercise yields two benefits. First, it raises awareness that single-point estimates tend to be akin to Best Case estimates. Second, going through the process of writing down Best Case and Worst Case estimates a few times begins to engrain the habit of thinking through the worst case outcome when estimating. Once you get into the habit of considering both best case and worst case outcomes, you'll get better at factoring the full range of possible outcomes into your single-point task estimates, regardless of whether you actually write down the best and worst cases.
| Tip #44 | Create both Best Case and Worst Case estimates to stimulate thinking about the full range of possible outcomes. |
Formulas
Creating the Best Case and the Worst Case estimates is just the first step. You're still left with the question of which estimate to use. Or maybe you should use the mathematical midpoint instead? The answer is none of the above. In many cases, the Worst Case is much worse than what's called the Expected Case. Taking the midpoints of the ranges could result in an unnecessarily high estimate.
A technique called the Program Evaluation and Review Technique (PERT) allows you to compute an Expected Case that might not be exactly in the middle of the range from best case to worst case (Putnam and Myers 1992, Stutzke 2005). To use PERT, you add an additional Most Likely Case to your set of cases. You can estimate the Most Likely Case using expert judgment. You then calculate the Expected Case using this formula:
| (#1) |  |
This formula accounts for the full width of the range as well as the position of the Most Likely Case within the range. Table 9-3 shows the estimates from Table 9-2 with the addition of Most Likely Case and Expected Case. As you can see from the table, the overall estimate of 13.62 is closer to the lower end of the range than the midpoint of 14.4 would have been.
| Estimated Days to Complete | ||||
|---|---|---|---|---|
| Feature | Best Case | Most Likely Case | Worst Case | Expected Case |
| Feature 1 | 1.25 | 1.5 | 2.0 | 1.54 |
| Feature 2 | 1.5 | 1.75 | 2.5 | 1.83 |
| Feature 3 | 2.0 | 2.25 | 3.0 | 2.33 |
| Feature 4 | 0.75 | 1 | 2.0 | 1.13 |
| Feature 5 | 0.5 | 0.75 | 1.25 | 0.79 |
| Feature 6 | 0.25 | 0.5 | 0.5 | 0.46 |
| Feature 7 | 1.5 | 2 | 2.5 | 2.00 |
| Feature 8 | 1.0 | 1.25 | 1.5 | 1.25 |
| Feature 9 | 0.5 | 0.75 | 1.0 | 0.75 |
| Feature 10 | 1.25 | 1.5 | 2.0 | 1.54 |
| TOTAL | 10.5 | 13.25 | 18.25 | 13.62 |
As discussed in Chapter 4, "Where Does Estimation Error Come From?" people's "most likely" estimates tend to be optimistic, which can yield optimistic overall estimates when using this approach. Some estimation experts suggest altering the basic PERT formula to account for a downward bias in the estimates (Stutzke 2005). Here's the altered formula:
| (#2) |  |
This is a reasonable short-term solution to the problem. The long-term solution to this problem is to work with people to make their Most Likely Case estimates more accurate.
Checklists
Even experts occasionally forget to consider everything they should. Studies of forecasting in a variety of disciplines have found that simple checklists help improve accuracy by reminding people of considerations they might otherwise forget (Park 1996, Harvey 2001, Jørgensen 2002). Table 9-4 presents a checklist you might use to improve the accuracy of your estimates.
|
To avoid omitting work from your estimates, you might also review the lists of overlooked activities in Section 4.5, "Omitted Activities."
| Tip #45 | Use an estimation checklist to improve your individual estimates. Develop and maintain your own personal checklist to improve your estimation accuracy. |
EAN: 2147483647
Pages: 212