4.6 CALIBRATION

< Day Day Up >

If there is debate about the number of cost drivers that are required within a cost model there is one thing that just about everyone agrees about as far as generalized cost models are concerned. They need to be "calibrated."

Checking my dictionary again I find that to calibrate means, "to fix or correct the scale." How is this done for a cost model? Well, calibration is more than simply redefining the questions that a package implementation of a cost model asks you so that the questions have meaning to your site. This is necessary because, while people tend to underestimate, they do tend to present worst-case scenarios when asked certain types of question. Ask a team leader how complex his or her application is and I guarantee that the answer will be "very complex," even if it is the most simple of batch processes.

So what is meant, in the practical sense, by this term "calibration?" I think that I can best illustrate the answer by example. Part of the work that I do includes assisting with the evaluation, and often the introduction, of generalized cost models. The evaluation goes something like this: For a number of projects that have been completed, data is put through the cost model and the predicted effort, and often the predicted duration, is recorded. The actuals are also obtained from historical data held by the installation. Obviously, if the organization does not record effort data or the start and end dates of projects then I have a problem with this approach, but such an organization may not be ready to use generalized cost models as they cannot calibrate them. An alternative that is sometimes viable within an enhancement environment is to use one project, across perhaps a six month period as the calibration vehicle.

The next step is to compare the predicted values with the actuals, at which point you usually find glaring discrepancies. It is at this point that people fling their hands up in horror and prepare to throw in the towel, but why? Do you really expect a model that was derived from a specific set of data in specific installation to be immediately applicable to your installation? Of course not.

Some tools salesmen will also throw their hands up in horror and then patiently explain that it is your "actuals" that are inaccurate, not the predictions. All I can say to this is that, time and time again, I have found locally collected cost data to be perfectly acceptable — not accurate to decimal places, but accurate enough. In fact, one piece of research that I was involved in, albeit to a limited extent, found that cost data extracted from a particular time recording mechanism on a specific site was very inaccurate, but it was consistently inaccurate. Think about that.

Because, of course, that is the key thing to look for when you compare predictions to actuals. You should expect to see inaccuracy but you should hope for consistent inaccuracy. For instance, one such evaluation found that two of the commonly used cost models were consistently underestimating effort by 40%, give or take a percent or two either way. Interestingly enough, I found that one of the models, when used within a different organization, was overestimating effort by about 80%, consistently. That certainly illustrates the need for calibration!

But what do you do if you find that there is not even consistent inaccuracy? Faced with that situation on one occasion I very nearly did throw in the towel until a consultant with one of the companies that sold the tool we were evaluating suggested that we look at the hot buttons, the drivers that really affected the results significantly. Which at least shows that there is good in some of these people!

Well, we went further than that, we reduced the parameters supplied to the tool to size together with the type of work being done in terms of it being new development or enhancement. The one other parameter we included was the type of system in terms of it being commercial, telecommunications, systems software, etc. What did we find? The inaccuracy was still there but now it was consistent inaccuracy, which again may illustrate that cost drivers should be treated with extreme caution.

There are three further points to make about these calibration exercises. First, I have found that calibrating for effort or cost predictions takes a lot less data than calibrating for duration. This is one reason why I feel that duration should be planned rather than estimated.

Second, while most if not all cost estimation tools will break the overall estimate down into the cost associated with specific phases of the project, I am still wary about using this. Again, you need a significant amount of data to calibrate these to your environment and I believe that few organizations are able to do any more than treat these as more than ballpark figures. Anyway, most organizations would be quite happy if they could get accurate project estimates let alone, for example, accurate design phase estimates!

Third, these types of calibration exercise do tend to use actual size figures as the input to the models. If you do have estimates available you could try running the models with that data but you may find problems in variability of size estimate accuracy. If you were to simply introduce the tools or models and expect that to solve your estimation problems then you may be in for a shock. I hope to show that there is a place for these types of tools but only as part of an overall estimating strategy or process. That strategy must address the question of estimating project size which is then used as input to the models.

A final point about calibration. If you are operating within an environment that develops products over a long period of time, and by that I mean about five years from initiation to first delivery, then you will have a problem with calibration. For all sorts of reasons, you will probably not have historical data available and you, obviously, cannot calibrate in real-time the way, say, a group producing new builds every three months can. The best you can probably hope for from generalized cost models is to use them as a stimulus for practical and sensible discussion. The great value of cost drivers, and one case where "the more, the merrier" principle applies, is that they are excellent vehicles for getting project managers to think about the things that can effect the cost and duration of their projects.

< Day Day Up >