8.1 Improved Accuracy and Other Benefits of Historical Data

The most important reason to use historical data from your own organization is that it improves estimation accuracy. The use of historical data, or "documented facts," is negatively correlated with cost and schedule overruns—that is, projects that have been estimated using historical data tend not to have overruns (Lederer and Prasad 1992).

The following sections discuss some of the reasons that historical data improves accuracy.

Accounts for Organizational Influences

First and foremost, use of historical data accounts for a raft of organizational influences that affect project outcomes. For very small projects, individual capabilities dictate the project outcome. As project size increases, talented individuals still matter, but their efforts are either supported or undermined by organizational influences. For medium and large projects, organizational characteristics start to matter as much as or more than individual capabilities.

Here are some of the organizational influences that affect project outcomes:

How complex is the software, what is the execution time constraint, what reliability is required, how much documentation is required, how precedented is the application—that is, how does the project stack up against the Cocomo II factors related to the kind of software being developed (as discussed in Chapter 5, "Estimate Influences")?
Can the organization commit to stable requirements, or must the project team deal with volatile requirements throughout the project?
Is the project manager free to remove a problem team member from the project, or do the organization's Human Resources policies make it difficult or impossible to remove a problem employee?
Is the team free to concentrate on the current project, or are team members frequently interrupted with calls to support production releases of previous projects?
Can the organization add team members to the new project as planned, or does it refuse to pull people off other projects before those projects have been completed?
Does the organization support the use of effective design, construction, quality assurance, and testing practices?
Does the organization operate in a regulated environment (for example, under FAA or FDA regulations) in which certain practices are dictated?
Can the project manager depend on team members staying until the project is complete, or does the organization have high turnover?

Accounting for each of these influences in an estimate one by one is difficult and error-prone. But historical data adjusts for all these factors, whether you're aware of the specifics or not.

Avoids Subjectivity and Unfounded Optimism

One way that subjectivity creeps into estimates is that project managers or estimators look at a new project, compare it with an old project, observe numerous differences between the two projects, and then conclude that the new project will go better than the old one did. They say, "We had a lot of turnover on the last project. That won't happen this time, so we'll be more productive. Also, people kept getting called back to support the previous version, and we'll make sure that that doesn't happen this time either. We also had a lot of late-breaking requirements from marketing. We'll do a better job on that, too. Plus we're working with better technology this time and newer, more effective development methods. With all those improvements, we should be able to be way more productive."

It's easy to identify with the optimism in these lines of reasoning. But the factors listed are controlled more by the organization than by the specific project manager, so most of these factors tend to be difficult to control for one specific project. The other factors tend to be interpreted optimistically, which introduces bias into the estimate.

With historical data, you use a simplifying assumption that the next project will go about the same as the last project did. This is a reasonable assumption. As estimation guru Lawrence Putnam says, productivity is an organizational attribute that cannot easily be varied from project to project (Putnam and Myers 1992, Putnam and Myers 2003). The same concept shows up in Extreme Programming as "Yesterday's Weather": the weather today won't always be the same as it was yesterday, but it's more likely to be like yesterday's weather than like anything else (Beck and Fowler 2001).

Tip #35

Use historical data as the basis for your productivity assumptions. Unlike mutual fund disclosures, your organization's past performance really is your best indicator of future performance.

Reduces Estimation Politics

One of the traps in estimation models that include a lot of control knobs is that many of the higher-leverage knobs are related to personnel. Cocomo II, for example, requires you to make assessments of your requirements analysts' and programmers' capabilities, along with several less subjective personnel factors related to experience. Cocomo requires the estimator to rate the programmers as 90th percentile, 75th percentile, 55th percentile, 35th percentile, or 15th percentile. (All these percentiles are industrywide.)

Suppose a manager takes a Cocomo II estimate into a meeting with an executive and the meeting agenda is to look for fat in the manager's estimate. It's easy to imagine the conversation going like this:

MANAGER: I know we had a goal of finishing this release in 12 weeks, but my estimates indicate that it will take 16 weeks. Let's walk through the estimate using this software estimation tool. Here are the assumptions I made. First, I had to calibrate the estimation model. For the "programmer capability" factor, I assumed our programmers are 35th percentile—

EXECUTIVE: What?! No one on our staff is below average! You need to have more confidence in your staff! What kind of manager are you? Well, maybe we've got a few people who aren't quite as good as the rest, but the overall team can't be that bad. Let's assume they're at least average, right? Can you enter that into the software?

MANAGER: Well, OK. Now, the next factor is the capability of the requirements engineers. We've never focused on recruiting good requirements engineers or developing those skills in our engineers, so I assumed they were 15th percentile—

EXECUTIVE: Hold on! 15th percentile? These people are very talented, even if they haven't had formal training in requirements engineering. They've got to be at least average. Can we change that factor to average?

MANAGER: I can't justify making them average. We really don't even have any staff we can call requirements specialists.

EXECUTIVE: Fine. Let's compromise and change the factor to 35th percentile then.

MANAGER: OK (sigh).

In this interaction, if the manager was using the Cocomo II adjustment factors, his estimate of effort required was just reduced by 23%. If the executive had succeeded in talking the manager into rating the requirements engineers as average rather than 35th percentile, the estimate would be reduced by 39%. In either case, a single conversation would result in a significant difference.

A manager who calibrates the estimate with historical data sidesteps the whole issue of whether the programmers are above average or below average. Productivity is whatever the data says it is. It's difficult for a non-technical stakeholder to argue with a statement like this one: "We've averaged 300 to 450 delivered lines of code per staff month, so we've calibrated the model with an assumption of 400 lines of code per staff month, which we believe is a little on the optimistic side but within a prudent planning range."

Clearly, half the programmers in the industry are below average, but I rarely meet project managers or executives who believe their programmers are the people who are below average.

Tip #36

Use historical data to help avoid politically charged estimation discussions arising from assumptions like "My team is below average."