Uncertainty About the Conditional Nature of Estimates

A large proportion of those using probabilistic project risk management processes often fail to address the conditional nature of probabilities and associated measures used for decision-making and control. The key outputs of the estimation and evaluation phases of the project risk management process are estimates of expected values for project parameters and measures of plausible variations on the high and low side. Interpretations of expected values or plausible extremes, like a 95 percent confidence value, have to be conditional on the assumptions made to estimate these values. For example, a sales estimate may be conditional on a whole set of assumed trading conditions, such as a particular promotion campaign and no new competitors. Invariably estimates ignore, or assume away, the existence of uncertainty that relates to three basic sources: known unknowns, unknown unknowns, and bias. These sources of uncertainty can have a very substantial impact on estimates that needs to be recognized and managed.

Known unknowns are of two types: explicit, extreme events (triple Es) and scope adjustment provisions (SAPs). Triple Es are "force majeure" events, like a change in legislation that would influence an oil company's pipeline design criteria in a fundamental way. SAPs are conditions or assumptions that may not hold, which are explicit, like the assumed operating pressure and flow value for an oil pipeline, given the assumed oil recovery rate.

Unknown unknowns are the unidentified triple Es or SAPs that should be factored into the project risk management process. We know that the realization of some unknown unknowns is usually inevitable. They do not include issues like "the world may end tomorrow" because it is sensible for most practical decision-making to assume we will still be here tomorrow, but the boundary between this extreme and what should be included is usually ill-defined.

Bias may be conscious or unconscious, pessimistic or optimistic, and clues, if not data, about the extent of bias may be available or not.

The impact of these three basic sources of uncertainty can be considered via the use of three scaling factors:

• F_k = known unknowns

• F_u = unknown unknowns

• F_b = bias

For example, an F_k scaling factor might be defined in relation to an estimated expected cost in the form:

Fk = 1.0 Probability of Fk = 0.1
1.1	0.7
1.2	0.1
1.3	0.1

This example involves a mean F_k = 1.12. In simple, crude terms, this would imply that an uplift in the estimated cost of the order of 30 percent is plausible for a pessimistic scenario value, like a 95 percentile, and an uplift of 12 percent is an appropriate expectation. In practice F_k values could be much higher than in this example.

Combining these three scaling factors provides a single "cube" (KUUB) factor, F³, defined by:

This is then applied as a scaling factor to conditional estimates. This KUUB factor, F³, can be estimated in probability terms directly or via these three components to clarify the conditional nature of the output of any quantitative risk analysis. This avoids the very difficult mental gymnastics associated with trying to interpret a quantitative risk analysis result that is conditional on exclusions and scope assumptions (which may be explicit or implicit) and no bias, without under-estimating the importance of the conditions.

The key value of the explicit quantification of F³ is forcing those involved to think about the implications of the factors that drive the expected size and variability of F³. Such factors may be far more important than the factors captured in the prior conventional quantitative risk analysis. There is a natural tendency to forget about conditions and assumptions and focus on the numbers. Attempting to explicitly size F³ makes it possible to avoid this. Even if different parties emerge with different views of an appropriate F³, the process of discussion is beneficial. If an organization refuses to estimate F³ explicitly, the issues involved do not go away. They simply become unmanaged risks. Many of them will be betting certainties.

An important source of ambiguity concerns the extent to which different project parties need to be concerned about particular F factors. For example, continuing the example of the last section, say the oil company project manager decides to contract out pipeline design. In estimating design costs the design company will not scale its estimates to allow for known unknowns if it can negotiate a contract to avoid bearing any risk associated with known unknowns. Similarly, it will not be appropriate for the project manager to scale the project design budget to incorporate an allowance for these known unknowns, unless they are wholly under the control of the project manager. For example, certain scope adjustments may come in this category. The potential impact of other unknown unknowns needs to be recognized in the oil company at some level above the project manager, where there is an ability to bear the consequences of any unknown unknown occurring. Thus, a KUUB factor needs to be estimated for the board of the oil company to adjust the design company's expected cost. A similar KUUB would need estimation if the oil company's design department undertook the work, but the risk allocation issues would be more complicated. Much post-project litigation arises because of a failure to appreciate or acknowledge exposure to KUUB factors, and a failure to resolve ambiguity about responsibility for KUUB factors earlier in the project.