2.2 Discussion of Quiz Results


2.2 Discussion of Quiz Results

The purpose of this quiz is not to determine whether you know when Alexander the Great was born or the latitude of Shanghai. Its purpose is to determine how well you understand your own estimation capabilities.

How Confident Is "90% Confident"?

The directions above are specific that the goal of the exercise is to estimate at the 90% confidence level. Because there are 10 questions in the quiz, if you were truly estimating at the 90% confidence level, you should have gotten about 9 answers correct. [1]

If you were cautious, you made your ranges conservatively wide, in which case you scored 10 out of 10 correctly. If you were just a little hasty, you made your ranges narrower than they needed to be, in which case you scored 7 or 8 out of 10 correctly. I've given this quiz to hundreds of estimators. Figure 2-1 shows the results from the most recent 600 people who have taken the quiz.

image from book
Figure 2-1: Results from administering the "How Good an Estimator Are You?" quiz. Most quiz-takers get 1–3 answers correct.

For the test takers whose results are shown in the figure, the average number of correct answers is 2.8. Only 2 percent of quiz takers score 8 or more answers correctly. No one has ever gotten 10 correct. I've concluded that most people's intuitive sense of "90% confident" is really comparable to something closer to "30% confident." Other studies have confirmed this basic finding (Zultner 1999, Jørgensen 2002).

Similarly, I've seen numerous project teams present "90% confident" schedules, and I've frequently seen those project teams overrun their "90% confident" schedules—more often than not. If those schedules represented a true 90% confidence, the project teams would overrun them only about 1 time out of 10.

I've concluded that specific percentages such as "90%" are not meaningful unless they are supported through some kind of statistical analysis. Otherwise those specific percentages are just wishful thinking. How to get to a real 90% confidence level will be discussed later in the book.

Tip #5 

Don't provide "percentage confident" estimates (especially "90% confident") unless you have a quantitatively derived basis for doing so.

If you didn't take the quiz earlier in this chapter, this would be a good time to go back and take it. I think you'll be surprised at how few answers you get correct even after reading this explanation.

How Wide should You Make Your Ranges?

When I find the rare person who gets 7 or 8 answers correct, I ask "How did you get that many correct?" The typical response? "I made my ranges too wide."

My response is, "No, you didn't! You didn't make your ranges wide enough!" If you got only 7 or 8 correct, your ranges were still too narrow to include the correct answer as often as you should have.

We are conditioned to believe that estimates expressed as narrow ranges are more accurate than estimates expressed as wider ranges. We believe that wide ranges make us appear ignorant or incompetent. The opposite is usually the case. (Of course, narrow ranges are desirable in the cases when the underlying data supports them.)

Tip #6 

Avoid using artificially narrow ranges. Be sure the ranges you use in your estimates don't misrepresent your confidence in your estimates.

Where Does Pressure to Use Narrow Ranges Come From?

When you were taking the quiz, did you feel pressure to make your ranges wider? Or did you feel pressure to make your ranges narrower? Most people report that they feel pressure to make the ranges as narrow as possible. But if you go back and review the instructions, you'll find that they do not encourage you to use narrow ranges. Indeed, I was careful to state that you should make your ranges neither too wide nor too narrow—just wide enough to give you a 90% confidence in including the correct answer.

After discussing this issue with hundreds of developers and managers, I've concluded that much of the pressure to provide narrow ranges is self-induced. Some of the pressure comes from people's sense of professional pride. They believe that narrow ranges are a sign of a better estimate, even though that isn't the case. And some of the pressure comes from experiences with bosses or customers who insisted on the use of overly narrow ranges.

This same self-induced pressure has been found in interactions between customers and estimators. Jørgensen and Sjøberg reported that information about customers' expectations exerts strong influence on estimates and that estimators are typically not conscious of the degree to which their estimates are affected (Jørgensen and Sjøberg 2002).

Tip #7 

If you are feeling pressure to make your ranges narrower, verify that the pressure actually is coming from an external source and not from yourself.

For those cases in which the pressure truly is coming from an external source, Chapter 22, "Estimate Presentation Styles," and Chapter 23, "Politics, Negotiation, and Problem Solving," discuss how to deal with that pressure.

How Representative Is This Quiz of Real Software Estimates?

In software, you aren't often asked to estimate the volume of the Great Lakes or the surface temperature of the Sun. Is it reasonable to expect you to be able to estimate the amount of U.S. currency in circulation or the number of books published in the U.S., especially if you're not in the U.S.?

Software developers are often asked to estimate projects in unfamiliar business areas, projects that will be implemented in new technologies, the impacts of new programming tools on productivity, the productivity of unidentified personnel, and so on. Estimating in the face of uncertainty is business as usual for software estimators. The rest of this book explains how to succeed in such circumstances.

[1]The mathematics behind "90% confident" are a little complicated. If you were really estimating with 90% confidence, you would have a 34.9% chance of getting 10 answers correct, 38.7% chance of getting 9 answers correct, and a 19.4% chance of getting 8 answers correct. In other words, you'd have a 93% chance of getting 8 or more answers correct.




Software Estimation. Demystifying the Black Art
Software Estimation: Demystifying the Black Art (Best Practices (Microsoft))
ISBN: 0735605351
EAN: 2147483647
Year: 2004
Pages: 212

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net