Models Versus Experience

ILM analysis and Business Impact Modeling emphasize the importance of facts and their inter-relationships as expressed through statistical models of workforce and business outcomes. Could models ultimately replace the role of personal experience in management? Atul Gawande’s book Complications provides an illustration of what can happen when a model is pitted against experience.^[2] The example centers on electrocardiographic (ECG) tests. Those tests record and print out electrical impulses from heart muscles, whose patterns can reveal the difference between a healthy heart and, say, one that has experienced a heart attack. The patterns are not simple. The test output includes multiple waves. Further, each wave has many attributes, such as height and disturbance, and there are almost innumerable combinations of waves and attributes, all of which must be recognized and interpreted appropriately.

In Gawande’s example, one of Sweden’s most experienced cardiologists, a professional who read the ECGs of over 10,000 patients every year, was pitted against a statistical model. In that contest 2,240 real ECG results were presented to the expert. Half were known to be associated with healthy hearts, and half with cases of heart attacks. The cardiologist correctly identified over 55 percent of the heart attack cases. A computer-based mathematical model of ECG patterns, however, correctly identified just over 66 percent of heart attack cases, a significant improvement over the results with the experienced doctor.

Was this a quirk? Not at all. As far back as 1954 the psychologist Paul Meehl demonstrated that statistical and actuarial models consistently showed greater accuracy in diagnosing patients than did experienced clinicians. Robin Hogarth points out that these results hold up in other areas as well, such as predicting the prices of securities, the longevity of cancer patients, and success in graduate school.^[3]

What does this say about the role of statistical models in strategy making? Should executives’ experience be discounted or replaced by formulas? Before answering these questions let’s look briefly at why models are often more accurate than experienced professionals.

Part of the reason for the accuracy of models is that they efficiently reduce complex relationships to formulas—sets of rules for weighing and combining facts. The process of finding those formulas is designed to weed out irrelevant facts: those which are not consistently related to the outcome, for example, or those which are redundant. The process itself disciplines thinking and establishes an order of importance among relevant facts. It also helps identify the combinations of facts that best predict outcomes.

There is also a degree of consistency and objectivity in models that human experts generally lack if only because of the fact that they are human. Models benefit from being unthinking; their rules for recognizing and combining facts are used over and over and in the same way. Humans, in comparison, are less consistent. Their attention wanders. They sometimes rush to judgment or overemphasize certain facts. Cognitive limitations may prevent them from processing complex patterns accurately.

What is the role of experience? Simply put, experience is critical but insufficient. When practical models are available to inform strategic decision making, there is no reason for modern organizations to rely exclusively on experience and qualitative facts. It is necessary to use those things in the strategy-making process, but it is important to supplement the power of experience with fact-based models of workforce and business outcomes.

^[2]Atul Gawande, Complications: A Surgeon’s Notes on an Imperfect Science. New York: Henry Holt and Company, 2002.

^[3]Robin M. Hogarth, Educating Intuition. Chicago: University of Chicago Press, 2001.