Section 3.2. Wideband Delphi Estimation

3.2. Wideband Delphi Estimation

The Wideband Delphi estimation method was developed in the 1940s at the Rand Corporation as a forecasting tool. It has since been adapted across many industries to estimate many kinds of tasks, ranging from statistical data collection results to sales and marketing forecasts. It has proven to be a very effective estimation tool, and it lends itself well to software projects.^[*]

^[*] The repeatable process for Wideband Delphi was developed in the 1990s by Mary Sakry and Neil Potter of The Process Group . Potter and Sakry offer a software estimation workshop based on their process. Information on their training products can be found at http://www.processgroup.com.

The Wideband Delphi estimation process is especially useful to a project manager because it produces several important elements of the project plan. The most important product is the set of estimates upon which the project schedule is built. In addition, the project team creates a work breakdown structure (WBS), which is a critical element of the plan. The team also generates a list of assumptions, which can be added to the vision and scope document.

The discussion among the team during both the kickoff meeting and the estimation session is another important product of the Delphi process. This discussion typically uncovers many important (but previously unrecognized) project priorities, assumptions, and tasks. The team is much more familiar with the work they are about to undertake after they complete the Wideband Delphi process.

Wideband Delphi works because it requires the entire team to correct one another in a way that helps avoid errors and poor estimation. While software estimation is certainly a skill that improves with experience, the most common problem with estimates is simply that the person making the estimate does not fully understand what it is that he is estimating. He may be an experienced software engineer, but if he has not fully explored all of the assumptions behind the estimate, the estimate will be incorrect. Delphi addresses this problem through the discussion of assumptions and the generation of consensus among the estimation team members.

Note: The Wideband Delphi process described here depends on a vision and scope document. It is possible to estimate a project without a vision and scope document, instead relying solely on the project team's understanding of the organization's needs. However, if there is no vision and scope document for the project, most project managers will find that writing one will improve the project far more than trying to estimate without it. (Chapter 2 describes how to develop a vision and scope document as part of the software project planning activities.)

3.2.1. The Delphi Process

To use Wideband Delphi, the project manager selects a moderator and an estimation team with three to seven members. The Delphi process consists of two meetings run by the moderator. The first meeting is the kickoff meeting, during which the estimation team creates a WBS and discusses assumptions. After the meeting, each team member creates an effort estimate for each task. The second meeting is the estimation session, in which the team revises the estimates as a group and achieves consensus. After the estimation session, the project manager summarizes the results and reviews them with the team, at which point they are ready to be used as the basis for planning the software project. The script in Table 3-1 describes the Wideband Delphi process.

Table 3-1. Wideband Delphi script
Name	Wideband Delphi script
Purpose	A project team generates estimates and a work breakdown structure.
Summary	A repeatable process for estimation. Using it, a project team can generate a consensus on estimates for the completion of the project.
Work Products	Input Vision and scope document, or other documentation that defines the scope of the work product being estimated Output Work breakdown structure (WBS) Effort estimates for each of the tasks in the WBS
Entry Criteria	The following criteria should be met in order for the Delphi process to be effective: The vision and scope document (or other documentation that defines the scope of the work product being estimated) has been agreed to by the stakeholders, users, managers, and engineering team. If no vision and scope document is available, there must be enough supporting documentation for the team to understand the work product. The kickoff meeting and estimation session have been scheduled (each at least two hours). The project manager and the moderator agree on the goal of the estimation session by identifying the scope of the work to be estimated.
Basic Course of Events	Choosing the team. The project manager selects the estimation team and a moderator. The team should consist of three to seven project team members. The team should include representatives from every engineering group that will be involved in the development of the work product being estimated. Kickoff meeting. The moderator prepares the team and leads a discussion to brainstorm assumptions, generate a WBS, and decide on the units of estimation. Individual preparation. After the kickoff meeting, each team member individually generates the initial estimates for each task in the WBS, documenting any changes to the WBS and missing assumptions. Estimation session. The moderator leads the team through a series of iterative steps to gain consensus on the estimates. At the start of the iteration, the moderator charts the estimates on the whiteboard so the estimators can see the range of estimates. The team resolves issues and revises estimates without revealing specific numbers. The cycle repeats until either no estimator wants to change his or her estimate, the estimators agree that the range is acceptable, or two hours have elapsed. Assembling tasks. The project manager works with the team to collect the estimates from the team members at the end of the meeting and compiles the final task list, estimates, and assumptions. Reviewing results. The project manager reviews the final task list with the estimation team.
Alternative Paths	During Step 1, if the team determines that there is not enough information known about the project to perform an estimate, the script ends. Before the script can be started again, the project manager must document the missing information by creating or modifying the vision and scope document (see Chapter 2). During either Step 1 or 3, if the team determines that there are outstanding issues that must be resolved before the estimate can be made, they agree upon a plan to resolve the issues and the script ends.
Exit Criteria	The script ends after the team has either generated a set of estimates or has agreed upon a plan to resolve the outstanding issues.

3.2.1.1. Choosing the team

Picking a qualified team is an important part of generating accurate estimates. Each team member must be willing to make an effort to estimate each task honestly, and should be comfortable working with the rest of the team. Estimation sessions can get heated; a team that already has friction will find that it runs into many disagreements that are difficult to resolve. The free flow of information is essential, and the project manager should choose a group of people who work well together. The estimators should all be knowledgeable enough about the organization's needs and past engineering projects (preferably similar to the one being estimated) to make educated estimates.

The moderator should be familiar with the Delphi process, but should not have a stake in the outcome of the session, if possible. Project managers are sometimes tempted to fill the moderator role, but this should be avoided (if at all possible) because the project manager should ideally be part of the estimation team. This is because the PM needs to take an active role in the discussion of the assumptions. She usually has a perspective on the project priorities that some of the engineers, stakeholders, and users do not see at first.

The role of the moderator is to listen to the discussion, ask open-ended questions, challenge the team to address issues, and ensure that everyone on the team is contributing. The moderator may estimate, but if he does, it is important that he remain unbiased by the team's estimates. A well-chosen team will allow the moderator to sit out on the estimation tasks and remain neutral and open-minded during the discussion.

The project manager should choose the team, and it should include people that she is comfortable working with. The team should include representatives from as many areas of the development team as possible: managers, developers, designers, architects, QA engineers, requirements analysts, technical writers, etc. Most importantly, each of the team members should have a stake in the plan, meaning that his goal is to establish a plan which he can agree to and live with. This allows the Delphi process to serve as an important tool for gaining the engineering team's support for the project plan, giving all involved a feeling of ownership of the estimates on which it is based.

Finally, one or more observersselected stakeholders, users, and managersshould be encouraged to attend the meeting. The reason that the observers are important is that they often do not understand the engineering process and what goes into building the software. Including observers is an effective way to encourage mutual trust between the team and the nontechnical people in the organization. While the observers do not directly contribute to the numerical estimates, encouraging their involvement in the meetings will increase their feeling of ownership of the final estimates that are generated by the team. When the non-engineers participate in the discussion of assumptions and see how the team arrives at estimates, they walk away with a much greater understanding of how the engineers do their work. What's more, the assumptions are almost always discussed on a level that can generally be understood by most of the nontechnical observers. Since these assumptions usually end up focused on the most problematic areas of development, the observers leave the meetings with a much clearer picture of exactly how the software will be developed.

3.2.1.2. Kickoff meeting

The goal of the kickoff meeting is to prepare the team for the estimation session. When the kickoff meeting is scheduled, each team member is given the vision and scope document and any other documentation that will help her understand the project she is estimating. The team members should read all of the material before attending the meeting.

In addition, a goal statement for the estimation session should be agreed upon by the project manager and the moderator and distributed to the team before the session. This statement should be no more than a few sentences that describe the scope of the work that is to be estimated ("Generate estimates for programming and testing the first phase of Project X").

The moderator leads the meeting, which consists of the following activities:

The moderator explains the Wideband Delphi method to any new estimators.
If any team member has not yet read the vision and scope document and supporting documentation, the moderator reviews it with the team. (If this happens, the meeting should be expected to take an extra half-hour to hour.)
The moderator reviews the goal of the estimation session with the team, and checks that each team member is sufficiently knowledgeable to contribute.
The team discusses the product being developed and brainstorms any assumptions.
The team generates a task list consisting of 1020 major tasks. These tasks represent the top level of the work breakdown structureadditional detail can be generated later and/or discussed in the assumptions. This high-level task list is the basis for the estimates that are going to be created.
The team agrees on the units of estimation (days, weeks, pages, etc.).

The team must agree on the goal of the project estimation session before proceeding with the rest of the estimation process. In most cases, the goal is straightforward; however, it is possible that the team members will disagree on it. Disagreement could focus on missing requirements, on which programs or tasks are to be included, on whether or not to estimate user documentation or support requirements, on the size of the user base being supported, or other basic scope issues.

After the assumptions are discussed, the moderator leads a brainstorming session to generate the WBS. The team breaks the project down into between 10 and 20 tasks, representing all of the project activities that must be performed. Once the team is comfortable with the WBS and the assumptions, it will feel much more knowledgeable about the context in which it will be developing the software. This, in turn, will make everyone more comfortable with the team's estimates.

3.2.1.3. Individual preparation

After the kickoff meeting, the moderator writes down all of the assumptions and tasks that were generated by the team during the kickoff meeting and distributes them to the estimation team. Each team member independently generates a set of preparation results, a document which contains an estimate for each of the tasks, any assumptions that the team member made in order to create the estimates, and any additional tasks that should be included in the WBS but that the team missed during the kickoff meeting. (Figure 3-1 shows the format of the individual preparation results.) Each team member builds preparation results by first filling in the tasks, and then estimating the effort for each task. An estimate for each task should be added to the "Tasks to achieve goal" section of the preparation results; the "Time" column should contain the estimate for each task.

Figure 3-1. Individual preparation results

Each estimate should be made in terms of effort, not calendar time. This means that if the unit of estimation is "days," then the estimate should be for the total number of person-days spent. For example, if a task will require one person to work for 10 days and a second person to work for 6, the estimate should be 16 person-days (or 3.2 person-weeks, assuming a 5-day week). If both people are working at the same time so that their effort overlaps entirely, the calendar time required to do this task is 10 days.

Usually, effort does not overlap perfectly like this; this kind of parallel effort will be factored in later, when the project schedule is created (see Chapter 4). However, one important factor in creating the schedule is taking into account necessary delays in which no work will be done. For example, the team may need to wait for a server to be built or a software licensing agreement to be reached; estimates for any known waiting time can also be added to the preparation results.

Any effort related to project overhead should not be taken into account. This includes things like status meetings, reports, vacation, etc. A separate estimation session can be held for overhead. Any time an estimator identifies a project overhead task, it should be added to the "Project overhead tasks" section of the preparation results. Similarly, estimators may run across potential delays, because certain tasks can't start until after specific dates. These should be added to the "Calendar waiting time" section, and not taken into account while making estimates. (Often, a separate Delphi session will be held specifically to estimate waiting time or overhead tasks. This is the purpose of the checkboxes at the top of the estimation form in Figure 3-2.)

Figure 3-2. Filled-in estimation form ( ©The Process Group, copied with permission)

While estimating each task, most people realize that they must make additional assumptions in order to estimate tasks. These should be recorded in the "Assumptions" section of the preparation results. They may also discover additional tasks that were not found during the kickoff meetingthese missing tasks should be added to the "Task list" section and estimated along with the rest of the WBS tasks. The final result should be a complete task list, including any additional tasks found, waiting time, and overhead tasks, with an estimate attached to each task. Each team member should bring the results to the estimation session .

3.2.1.4. Estimation session

The estimation session starts with each estimator filling out an estimation form. Blank estimation forms should be handed out to meeting participants, who fill in the tasks and their initial estimates from their individual preparations. During the estimation session, the team members will use these estimation forms to modify their estimates. After the estimation session, they will serve as a record of each team member's estimates for the individual tasks, which the project manager uses when compiling the results. (Figure 3-2 shows an example of a typical filled-in estimation form.)

Before the team members fill in their forms, the moderator should lead a brief discussion of any additional tasks that were discovered during the individual preparation phase. Each task that the team agrees to add to the WBS should be added to the form; the team will generate estimates for that task later in the meeting. If the team decides that the task should not be included after all, the person who introduced it should make sure that the effort he estimated for that task is taken into account.

At this point, the participants fill out the estimation forms. The estimation form contains one row for each task being estimated. If there are more tasks than rows on the form, more than one form can be used for the session. Each participant starts with a blank form and writes the name of each task on consecutive rows. The estimate for that task should be written into the "Estimate" box next to the task. The estimate boxes are then added up and the total written into the "Total" box at the bottom of the Estimate column.

The moderator then leads the team through the estimation session:

The moderator collects all of the estimate forms. The estimates are tabulated on a whiteboard by plotting the totals on a line (see Figure 3-3). The forms are returned to the estimators.

Figure 3-3. Initial estimates
Each estimator reads out clarifications and changes to the task list written on the estimation form. Any new or changed tasks, discovered assumptions, or questions are raised. Specific estimate times are not discussed.
The team resolves any issues or disagreements that are brought up. Since individual estimate times are not discussed, these disagreements are usually about the tasks themselves, and are often resolved by adding assumptions. When an issue is resolved, team members should write clarifications and changes to the task list on their estimation forms. This usually takes about 40 minutes for the first round, and 20 minutes for the following rounds.
The estimators all revise their individual estimates by filling in the next "Delta" column on their forms. (Using a delta allows the estimators to write "+4" or "-3" to add 4 or remove 3 from the estimate. They write the new total at the bottom of the sheet.)

This cycle repeats until either all estimators agree that the range is acceptable, the estimators feel they do not need to change their estimates, or two hours have elapsed.

Each round brings the estimates closer to convergence. Figure 3-4 shows what the typical results of an estimation session will look like. It may seem magical to the teamthrough a straightforward discussion of assumptions and task definition, the team arrives at a consensus that is visible on the whiteboard! This consensus is a result of clarifying these potential ambiguities. Usually the first round brings a long discussion of missed assumptions and changes to the task definition. In the later rounds, the moderator will play an important role in directing the conversation toward where it will be most effective. The moderator should look for the tasks that have the largest spread between the highest and lowest estimates. He should lead a frank discussion of the breakdown or details of that task, as well as places where the team might be over- or underestimating the effort for that task.

Figure 3-4. Converging estimate results

Disagreements often arise, but they are easier to deal with in this setting than later on, when the project is in progress. One effective way to resolve these disagreements is to talk about both sides of the issue, and then agree on an assumption that takes one of those sides. It is easier to make this decision at this stage because the assumption is not permanent; the team can very easily go back and change that assumption, if necessary. But writing down the assumption allows the team to show management that if this assumption turns out to be incorrect, the estimate may no longer be accurate. This way, even though the estimate is not perfect, the team understands why that is the case. If the team has confidence that they did a complete job with the assumptions, they will have more confidence in the value of the estimate.

After the conclusion of the estimation cycle, the moderator leads a discussion on how the session went. The team suggests ways to improve. The moderator notes their feedback, to include it in the final estimation report.

3.2.1.5. Assemble tasks

After the estimation meeting is finished, the project manager works with the moderator to gather all of the results from the individual preparation and the estimation session. The project manager removes redundancies and resolves remaining estimate differences to generate a final task list, with effort estimates attached to each task. The assumptions are then summarized and added to the list.

The final task list should be in the same format as the individual preparation results (see Figure 3-1). In addition, the project manager should create a spreadsheet that lists the final estimates that each person came up with. The spreadsheet should indicate the best-case and worst-case scenarios, and it should indicate any place that further discussion will be required. Any task with an especially wide discrepancy should be marked for further discussion. Figure 3-5 shows an example of a spreadsheet that summarizes the results. It contains a column for each estimator, as well as columns for the best-case estimate (using the lowest estimates, which assume that everything has gone well), worst-case estimate (using the highest estimates, which assume that the project hit many roadblocks), and average estimate. (When the project manager calculates these numbers, it's often a good idea to remove the highest and lowest estimates in order to eliminate outliers.)

Figure 3-5. Summarized results of estimation

The assumptions should also be put into a final form, suitable for inclusion in the "Assumptions" section of the vision and scope document. (If the author of the vision and scope document is not the project manager, then the project manager should turn the assumptions over to the author and help incorporate them into the document.) If the vision and scope document was already inspected and approved, it should be updated and then inspected again prior to being used as the basis for a project plan.

Sometimes there may be a team member who disagrees with the team's assessment estimate for a task, and continues to disagree with it over the course of the project. The project manager must be careful in this situation, especially if the disagreement is over a task that will be assigned to that team member. An important part of the Delphi process is that it generates consensus among the team about the final schedule. That consensus can be much harder to achieve if any of the team members feels like her objections are being ignored. One way for the project manager to make sure that this does not happen is to call an additional meeting to discuss the outlying estimates. During this meeting, the project manager must take the time to listen to everything that the person who differs from the rest of the team has to say. The final decision about the effort still lies with the project manager; however, if there is a genuine disagreement, then the person who disagrees with the rest of the team will at least feel like his objections were heard and evaluated on their merits, rather than just rejected outright.

3.2.1.6. Review results

Once the results are ready, the project manager calls a final meeting to review the estimation results with the team. The goal of the meeting is to determine whether the results of the session are sufficient for further planning. The team should determine whether the estimates make sense and if the range is acceptable. They should also examine the final task list to verify that it's complete. There may be an area that needs to be refined: for example, a task might need to be broken down into subtasks. In this case, the team may agree to hold another estimation session to break down those tasks into their own subtasks and estimate each of them. This is also a good way to handle any tasks that have a wide discrepancy between the best-case and worst-case scenarios.