Uncertainty in Estimates of Risk Event Probability and Impact


Estimation and evaluation of uncertainty are core tasks in any process involving the management of uncertainty in projects. These tasks involve a number of important objectives addressed in the following, which contribute to a key aim of cost effective risk assessment.

  1. Understand uncertainty in general terms. Understanding uncertainty needs to go beyond variability and available data. It needs to address ambiguity and incorporate structure and knowledge, with a focus on making the best decisions possible given the available data, information, knowledge, and understanding of structure.

  2. Understand sources of uncertainty. One important aspect of structure is the need to understand uncertainty in terms of sources of uncertainty, because some (not all) appropriate ways of managing uncertainty are specific to its source.

  3. Determine what to quantify. It is very important to distinguish between what is usefully quantified and what is best treated as a condition or assumption in terms of decision-making effectiveness.

  4. Iterative processes. To facilitate insight and learning, uncertainty has to be addressed in terms of an iterative process, with process objectives that change on successive passes. An iterative approach is essential to optimize the use of time and other resources during the uncertainty management process, because initially where uncertainty lies, whether or not it matters or how best to respond to it are unknown. At the outset the process is concerned with sizing risk to discover what matters. Subsequent passes are concerned with refining assessments in order to effectively manage what matters. Final passes may be concerned with convincing others that what matters is being properly managed. The way successive iterations are used needs to be addressed in a systematic manner. A simple one-shot, linear approach is hopelessly inefficient.

  5. A minimalist first pass at estimation and evaluation. In order to "optimize" the overall process, a "minimalist" approach to the first pass at estimation and evaluation is critical. A minimalist first pass approach to estimation should be so easy to use that the usual resistance to appropriate quantification—based on lack of data and lack of comfort with subjective probabilities—is overcome.

  6. Avoid optimistic bias. Most approaches to estimation and evaluation induce optimistic bias, which leads to systematic underestimation of uncertainty. This needs direct and explicit attention to manage expectations. If successive estimates associated with managing risk do not narrow perceived variability and improve the perceived expected cost or profit on average, then the earlier analysis process is flawed. Very few organizations have processes that meet this test. They are failing to manage expectations. In general, the more sophisticated the process used, the more optimistic bias damages the credibility of estimation and evaluation processes.

  7. Simplicity with constructive complexity. Simplicity is an important virtue in its own right, not just with respect to the efficiency of a minimalist first pass approach, but because it can amplify clarity and deepen insight. However, appropriate "constructive complexity" is also important, for the same reasons. Getting the best balance is partly a question of structure and process, and partly a question of skills that can be learned via a process that is engineered to enhance learning.

The authors are not aware of any current approaches that explicitly address this set of objectives as a whole. Evidence of this is the sustained, widespread promotion and use of first pass approaches to estimation and evaluation employing a probability-impact matrix (PIM). This was deliberately accommodated, although not promoted, in the Project Risk Analysis and Management (PRAM) Guide (Simon, Hillson, and Newland 1997; Chapman 1997) because of differences of opinion amongst the working group. The PIM approach typically defines low, medium, and high bands for possible probabilities and impacts associated with identified sources of uncertainty (usually risks involving adverse impacts). These bands may be defined as quantified ranges or left wholly subjective. In either case assessment of probabilities and impacts is a relatively crude process whereby each source of uncertainty is assigned to a particular probability band and a particular impact band. This limited information about each source of uncertainty is often diluted by using "risk indices" with common values for different probability band and impact band combinations. Information about uncertainty is sometimes still further obscured by the practice of adding individual risk indices together to calculate spurious "project risk indices." The PIM approach seems to offer a rapid first pass assessment of the relative importance of identified sources of uncertainty, but it delivers very little useful information and even less real insight (Chapman and Ward 1997; Ward 1999).

Even with the availability of proprietary software products such as @Risk for quantifying, displaying, and combining uncertain parameters, the use of PIM has persisted (further encouraged by PIM software). This is surprising, but it suggests a gap between simple direct prioritization of sources of risk and quantification requiring the use of specialist software. In any event, none of these PIM approaches deals directly with the complete set of objectives set out above for estimation and evaluation.

To address this gap, Chapman and Ward (2000) describe a minimalist first pass approach to estimation and evaluation of uncertainty, set in the context of an approach that addresses all seven objectives. The minimalist approach defines uncertainty ranges for probability and impact associated with each source of uncertainty. Subsequent calculations preserve expected value and measures of variability, while explicitly managing associated optimistic bias.

The minimalist approach involves the following steps in a first pass attempt to estimate and evaluate uncertainty:

  1. Identify the parameters to be quantified.

  2. Estimate crude but credible ranges for probability of occurrence and impact.

  3. Calculate expected values and ranges for composite parameters.

  4. Present results graphically (optional).

  5. Summarize results.

In step 1 a clear distinction is made between sources of uncertainty that are useful to quantify and sources that are best treated as conditions, following project risk management (Chapman and Ward 1997) in this respect. For example, suppose an oil company project team wants to estimate the duration and the cost of the design of an offshore pipeline using the organization's design department. Current common best practice would require a list of sources of uncertainty (a risk list or risk log), which might include entries like "change of route", "demand for design effort from other projects", "loss of staff", and "morale problems." Change of route would probably be regarded as a source of uncertainty best treated as a condition by the project manager and the head of the design department. Subsequent steps apply only to those sources of uncertainty that are usefully quantified.

In step 2 the "probability" of a threat occurring is associated with an approximate order of magnitude minimum and maximum plausible probability, assuming a uniform distribution (and a mid-point expected value). This captures the users feel for a "low, medium, or high" probability class in a flexible manner, captures information about uncertainty associated with the probability, and yields a conservative (pessimistic) expected value. For trained users it should be easier than designing appropriate standard classes for all risks and putting a tick in an appropriate box.

Similarly, the "impact" of a threat that occurs is associated with an approximate order of magnitude minimum and maximum plausible value (duration and cost), also assuming a uniform duration. This captures the user's feel for a "low, medium, or high" impact class in a flexible manner, captures information about the uncertainty associated with the impact, and yields a conservative (pessimistic) expected value.

In step 4 the expected values and associated uncertainties for all quantified sources of uncertainty are shown graphically in a way that displays the contribution of each to the total in expected value and range terms. This clearly indicates what matters and what does not, and is used as a basis for managing subsequent passes of the project risk management process in terms of data acquisition to confirm important probability and impact assessment, refinement of response strategies, and key decision choices.

Although simple, the minimalist approach is sophisticated in the sense that it builds in pessimistic bias. This minimizes the risk of dismissing as unimportant risks what more information might reveal as important. Also, it is set in the context of an iterative approach, which leads to more refined estimates wherever potentially important risks are revealed. Sophistication does not require complexity. It requires "constructive" simplicity, increasing complexity only when it is useful to do so.

The concern of the minimalist first pass approach is not a defensible quantitative assessment. The concern is to develop a clear understanding of what seems to matter, based on the views of those able to shed some light on the issues. This is an attempt to resolve the ambiguity associated with the size of uncertainty about the impact of risk events and the size of uncertainty about the probability of risk events occurring, the latter often dominating the former. A first pass may lead to the conclusion "there is no significant uncertainty, and no need for further effort." This is one of the reasons why the approach must have a conservative bias. Another reason is the need to manage expectations, with subsequent refinements of estimates indicating less uncertainty or increased uncertainty providing an explicit indication that the earlier process failed. An estimator should be confident that more work on refining the analysis is at least as likely to decrease the expected value estimate as to increase it. A tendency for cost estimates to drift upwards as more analysis is undertaken indicates a failure of earlier analysis. The minimalist approach is designed to help manage the expectations of those the estimator reports to in terms of expected values. Preserving credibility should be an important concern.

The minimalist approach departs from the first pass use of probability density histograms or convenient probability distribution assumptions, which the authors and many others have used for years in similar contexts. Readers who are used to first pass approaches that attempt considerable precision, may feel uncomfortable with the deliberate lack of precision incorporated in the minimalist approach. However, more precise modelling is frequently accompanied by questionable underlying assumptions like independence and lack of attention to uncertainty in original estimates. The minimalist approach forces explicit consideration of these issues. It may be a step back in terms of taking a simple view of the "big picture", but it should facilitate more precise modelling of uncertainty where it matters, and confidence that precision is not spurious.

Chapman and Ward (2000) use a specific context to illustrate the minimalist approach, but there is considerable scope for applying the approach in other contexts and in various stages of the project life cycle (PLC). The minimalist approach was deliberately designed for simple manual processing and no supporting software requirements. However, relatively simple hardware and inexpensive commercially available software (like @Risk) could be used to minimize the input demands of such analysis, make the outputs easy to interpret and rich, and make the movement on to second and further passes relatively straightforward. In general, application specific software should be developed once it is clear what analysis is required, after significant experience of the most appropriate forms of analysis for first and subsequent passes has been acquired.




The Frontiers of Project Management Research
The Frontiers of Project Management Research
ISBN: 1880410745
EAN: 2147483647
Year: 2002
Pages: 207

flylib.com © 2008-2017.
If you may any questions please contact us: flylib@qtcs.net