Tuesday, January 25, 2011
I predict that you will find this review informative. If you do, you will congratulate my foresight. If you don’t, you’ll forget I was wrong.
My playful intro summarizes the main thesis of
’s excellent book, Future Babble: Why Expert Predictions Fail – and Why We Believe Them Anyway. Gardner, a columnist for the Ottawa Citizen and author of the bestselling Risk, returns to the format that made Risk such a success: Find some interesting psychological research from the past few decades; describe the research in accessible and pithy prose for a general audience; emphasize cognitive biases; extrapolate the research findings to popular events to indicate why they matter; and imply that we should change our behaviours and policies. Gardner
In Future Babble, the research area explored is the validity of expert predictions, and the primary researcher examined is Philip Tetlock. In the early 1980s, Tetlock set out to better understand the accuracy of predictions made by experts by conducting a methodologically sound large-scale experiment.
presents Tetlock’s experimental design in an excellent way, making it accessible to the lay person. Concisely, Tetlock examined 27450 judgements in which 284 experts were presented with clear questions whose answers could later be shown to be true or false (e.g., “Will the official unemployment rate be higher, lower or the same a year from now?”). For each prediction, the expert must answer clearly and express their degree of certainty as a percentage (e.g., dead certain = 100%). The usage of precise numbers adds increased statistical options and removes the complications of vague or ambiguous language. Gardner
After letting this impressive experiment run its course for several years and crunching all the numbers to see how the predictions bore out, Tetlock found the surprising and disturbing truth “that experts’ predictions were no more accurate than random guesses.” (p. 26) An important caveat is that there was a wide range of capability, with some experts being completely out of touch, and others able to make successful predictions.
“What distinguishes the impressive few from the borderline delusional is not whether they’re liberal or conservative. Tetlock’s data showed political beliefs made no difference to an expert’s accuracy. The same is true of optimists and pessimists. It also made no difference if experts had a doctorate, extensive experience, or access to classified information. Nor did it make a difference if experts were political scientists, historians, journalists, or economists.” (p. 26)
The big difference is in the way the experts think.
The experts who did poorly were not comfortable with complexity and uncertainty, and tended to reduce most problems to some core theoretical theme. It was as if they saw the world through one lens or had one big idea that everything else had to fit into. Alternatively, the experts who did decently were self-critical, used multiple sources of information and were more comfortable with uncertainty and correcting their errors. Their thinking style almost results in a paradox: “The experts who were more accurate than others tended to be less confident they were right.” (p.27)
Future Babble would make a great gift, and I hope that
’s popularization of Tetlock’s work succeeds and the issues raised become part of a larger discussion on the validity of expert predictions. Gardner
Appendix (of sorts)
So ends the book review proper. Below I examine the book in more detail by going chapter by chapter, presenting some of my thoughts and notes. This content will likely be useful to those who want more detail, but it might be especially useful for those who have already read the work or who are looking to tease out to discussion points.
Chapter 2 – The Unpredictable World
An exploration into how many events in the world are simply unpredictable.
discusses chaos theory and necessary and sufficient conditions for events to occur. He supports the idea of actually saying “I don’t know,” which many experts are reluctant to do. Gardner
Chapter 3 – In the Minds of Experts
A more detailed examination of Hedgehogs and Foxes.
discusses randomness and the illusion of control while using narratives to illustrate his points à la Gladwell. This chapter provides a lot of context and background information that should be very useful to those less initiated. Gardner
Chapter 4 – The Experts Agree: Expect Much More of the Same
An interesting and almost amusing analysis of how the rise of
Japan was the big fear in the in the early 1990s, and pretty much none of it came true. He wisely mentions how the same concerns are occurring with US now. Although these concerns might be true, we should be wary of believing them. China really drives home the notion that an ordinary person has about as good a chance at making correct predictions as most experts. Gardner
I found two flaws in this chapter, neither major but worth noting.
Gardner uses a gross national income statistic to compare the and other countries, but he doesn't per capita measures (p.94). This is misleading and doesn’t fit with the rigour of the rest of the book. US
could have had a more nuanced discussion of Tetlock’s work and how it fits into the status quo problem. The issue here is that Tetlock found that if you predict “no change,” you’ll actually do a decent job predicting things. A related notion is the status quo bias, where people assume that things will continue as they are. This is a problem because people invalidly extrapolate trend lines. There is a subtle distinction here between assuming that the present circumstances won’t change (good for prediction) and assuming that indicators in the present are valid predictors of future circumstances (bad for prediction). I don’t think it would have been too much trouble to tease this out (if only in a footnote). Gardner
Chapter 5 – Unsettled by Uncertainty
While there was a lot of interesting information in this chapter, it felt disjointed and had a few too many anecdotes for my comfort. It was mainly stories of how bad things were in the 1970s, or how dire the predictions were, and how nothing that bad came to pass. It might be the weakest chapter, but the social/intellectual history was decent. To be fair, a different reader might enjoy having the concepts elaborated upon. The problem for me is that once
displayed Tetlock’s findings in the early chapters, further anecdotal information does not increase how convinced I am. Gardner
Chapter 6 – Everyone Loves a Hedgehog
More about predictions and how the media picks up hedgehog stories and talking points without much investigation into their underlying source or concern for accuracy. It is a good demolition of the absurdity of so many news “discussion shows.”
demonstrates how the media prefer a show where Hedgehogs square off against each other, and it is important that these commentators not be challenged lest they become exposed and, by association, implicate the flawed structure of the program/network. Gardner
Minor issue: If you check footnote 56, you’ll see
admit to an error that he exposes numerous others making in the body of the text. I wondered why he did this in a footnote. Was he concerned that admitting he didn’t check common wisdom for accuracy would undermine his authority as a columnist and writer? Gardner
Chapter 7 – When Prophets Fail
This might be the most entertaining chapter as it looks at prophets and prophecies, including experts who predicted Y2K chaos and calamities that never happened. There is a good exploration of Leon Festinger’s cognitive dissonance, which can generally be explained by saying that two or more beliefs come into conflict and they are usually resolved in a self-enhancing manner, putting truth as a lower priority. Regarding the theme of this book, “a mind deeply committed to the truth of a predication will do almost anything to avoid seeing evidence of the prediction’s failure for what it is.” (p.196)
The chapter opened with a great quotation by John Maynard Keynes: “When the facts change, I change my mind. What do you do, Sir?”
Finally, it is in chapter 7 that
writes one of his best passages: Gardner
“An assertion that cannot be falsified by any conceivable evidence is nothing more than dogma. It can’t be debated. It can’t be proven or disproven. It’s just something people choose to believe or not for reasons that have nothing to do with fact and logic. And dogma is what predictions become when experts and their followers go to ridiculous lengths to dismiss clear evidence that they failed.” (p. 236)
Chapter 8 – The End
Once again, there are nice phrases throughout and he knows how to write quotable prose.
So, was my prediction correct?