8.5 Calibration with Industry Average Data

8.5 Calibration with Industry Average Data

If you don't have your own historical data, you have little choice but to use industry-average data, which is adequate but no better. As Table 5-2 illustrated, the productivity rates for different organizations within the same industries typically vary by a factor of 10. If you use the average productivity for your industry, you won't be accounting for the possibility that your organization might be at the top end of the productivity range or at the bottom.

Figure 8-1 shows an example of an estimate created using industry-average data. Each point in the graph represents a possible project outcome created using a statistical technique known as a Monte Carlo simulation. The solid black lines represent the median effort and schedule found during the simulation. The dashed black lines represent the 25th and 75th percentiles for effort and schedule.

Figure 8-1: An example of estimated outcomes for an estimate calibrated using industry-average data. Total variation in the effort estimates is about a factor of 10 (from about 25 staff months to about 250 staff months).

Figure 8-2 shows an example of a comparable estimate calibrated using historical data from one of my clients.

Figure 8-2: An estimate calibrated using historical productivity data. The effort estimates vary by only about a factor of 4 (from about 30 staff months to about 120 staff months).

The sizes and nominal productivity rates of the two projects are identical, but the amount of variability in the two estimates is dramatically different. Because the industry-average estimate must account for factor-of-10 differences in productivity, the standard deviation on the effort estimate created using industry-average data is about 100%! If you wanted to give your boss an estimate that ranged from 25% confident to 75% confident using industry-average data, in this case you'd need to quote a range of 50 to 160 staff months—a factor of 3 difference!

If you could use historical data instead of industry-average data, you could quote a range of 70 to 95 staff months—a factor of only 1.4 from the top end of the range to the bottom. The standard deviation in the estimate created using historical data is only about 25%.

A review of studies on estimation accuracy found that in studies in which estimation models were not calibrated to the estimation environment, expert estimates were more accurate than the models. But the studies that used models calibrated with historical data found that the models were as good as or better than expert estimates (Jørgensen 2002).

Tip #41 

Use project data or historical data rather than industry-average data to calibrate your estimates whenever possible. In addition to making your estimates more accurate, historical data will reduce variability in your estimate arising from uncertainty in the productivity assumptions.