13.2 Wideband Delphi

13.2 Wideband Delphi

Wideband Delphi is a structured group-estimation technique. The original Delphi technique was developed by the Rand Corporation in the late 1940s for use in predicting trends in technology (Boehm 1981). The name Delphi comes from the ancient Greek oracle at Delphi. The basic Delphi technique called for several experts to create independent estimates and then to meet for as long as necessary to converge on, or at least agree upon, a single estimate.

An initial study on the use of Delphi for software estimation found that the basic Delphi technique was no more accurate than a less structured group meeting. Barry Boehm and his colleagues concluded that the generic Delphi meetings were subject to too much political pressure and were also likely to be dominated by the more assertive estimators in the group. Consequently, Boehm and his colleagues extended the basic Delphi technique into what has become known as Wideband Delphi. Table 13-1 describes the basic procedure.

1. The Delphi coordinator presents each estimator with the specification and an estimation form

2. Estimators prepare initial estimates individually. (Optionally, this step can be preformed after step 3.)

3. The coordinator calls a group meeting in which the estimators discuss estimation issues related to the project at hand. If the group agrees on a single estimate without much discussion, the coordinator assigns someone to play devil's advocate.

4. Estimators give their individual estimates to the coordinator anonymously.

5. The coordinator prepares a summary of the estimates on an iteration form (shown in Figure 13-2) and presents the iteration form to the estimators so that they can see how their estimates compare with other estimators' estimates.

6. The coordinator has estimators meet to discuss variations in their estimates.

7. Estimators vote anonymously on whether they want to accept the average estimate. If any of the estimators votes "no," they return to step 3.

8. The final estimate is the single-point estimate stemming from the Delphi exercise. Or, the final estimate is the range created through the Delphi discussion and the single-point Delphi estimate is the expected case.

Source: Adapted from Software Engineering Economics (Boehm 1981).

Figure 13-2: A Wideband Delphi estimating form.

Steps 3 through 7 can be performed either in person, in a group-meeting setting, or electronically via e-mail or chat software. Performing the steps electronically can help preserve anonymity. Iterations of steps 3 through 7 can be performed immediately or they can be performed in batch mode, depending on the time-criticality of the estimate and the availability of the estimators.

The estimating form shown in Figure 13-2 can be a paper form or it can be drawn by the coordinator on a whiteboard. The form shown has a range of 0 to 20 staff months. The range you initially show on the form should be at least triple the range you expect the estimators to come up with so that the estimators don't feel constrained to a predefined range.

The coordinator should take care to prevent people with dominant personalities from unduly influencing the estimate. Software developers aren't known for their assertive personalities, and the most reserved person will sometimes have the best insights into the work being estimated.

It's also useful to show all the rounds of estimates on the same scale so that the estimators can observe how their estimates are converging (or, in some cases, diverging). Figure 13-3 gives an example of this.

Figure 13-3: A Wideband Delphi estimating form after three rounds of estimates.

In this case, after Round 3, the group might decide to settle on a range of 12 to 14 staff months with an expected value of 13 staff months.

Effectiveness of Wideband Delphi

I've collected data on the use of Wideband Delphi with a very difficult estimation problem. For the first 25 groups I worked with, Figure 13-4 shows the error rate from a simple averaging of their initial estimates compared to the error rate from Wideband Delphi estimating.

Figure 13-4: Estimation accuracy of simple averaging compared to Wideband Delphi estimation. Wideband Delphi reduces estimation error in about two-thirds of cases.

My experience with Wideband Delphi suggests that it cuts estimation error by an average of approximately 40% compared to the initial group average. Of the groups in my study, about two-thirds produced a more accurate answer by using Wideband Delphi than by simply averaging their individual estimates.

Of the 10 groups I've worked with that produced the worst initial estimates (shown in Figure 13-5), Wideband Delphi improved estimation accuracy in 8 out of 10 cases, with an average error reduction of about 60%.

Figure 13-5: Wideband Delphi when applied to terrible initial estimates. In this data set, Wideband Delphi improved results in 8 out of 10 cases.

From this data, I conclude that Wideband Delphi improves accuracy in most cases, and it is especially useful in avoiding wildly erroneous results.

"The Truth Is Out There"

Implicit in estimation techniques that rely on averaging individual estimates is the idea that the correct answer lies somewhere in the range between the lowest estimate and the highest. In my Wideband Delphi data, however, 20% of the groups' initial estimation ranges do not include the correct answer. This means that averaging their initial estimates cannot possibly produce an accurate result.

Perhaps the most interesting phenomenon associated with Wideband Delphi is that one-third of the groups whose initial range does not include the correct answer ultimately settle on an estimate that is outside their initial range and closer to the correct answer. In other words, for these groups, the Wideband Delphi estimate turns out to be better than the best individual estimate. Figure 13-6 illustrates this dynamic. Notice that none of the groups settled on a final estimate that was worse than the worst individual estimate.

Figure 13-6: In about one-third of cases, Wideband Delphi helps groups that don't initially include the correct answer to move outside their initial range and closer to the correct answer.

When to Use Wideband Delphi

In the difficult group estimation exercise I've discussed in this chapter, Wideband Delphi reduced the average estimation error from 290% to 170%. Errors of 290% and 170% are very high, characteristic of estimates created in the wide part of the Cone of Uncertainty. Still, reducing error by 40% is valuable, whether the reduction is from 290% to 170% or from 50% to 30%.

Although my data seems to endorse the use of Wideband Delphi, industry studies on the question of how to combine estimates created by different estimators have been mixed. Some studies have found that group-based approaches to combining estimates work best, and others have found that simple averaging works best (Jørgensen 2002).

Because Wideband Delphi requires a meeting, it burns a lot of staff time, making it an expensive way to estimate. It is not appropriate for detailed task estimates.

Wideband Delphi is useful if you're estimating work in a new business area, work in a new technology, or work for a brand-new kind of software. It is useful for creating "order of magnitude" estimates at product definition or software concept time, before you've pinned down many of the requirements. It's also useful if a project will draw heavily from diverse specialties, such as a combined need for uncommon usability, algorithmic complexity, exceptional performance, intricate business rules, and so on. It also tends to sharpen the definition of the scope of work, and it's useful for flushing out estimation assumptions. In short, Wideband Delphi is most useful for estimating single, focused items that require input from numerous disciplines in the very wide part of the Cone of Uncertainty. In these uncertain situations, Wideband Delphi can be invaluable.

Tip #63 

Use Wideband Delphi for early-in-the-project estimates, for unfamiliar systems, and when several diverse disciplines will be involved in the project itself.