More On Calibration: Managerial Miscalibration

Earlier this week, I was asking how well calibrated is the Bernank? Calibration is an important topic in decision making. I'm reading an interesting paper right now titled Managerial Miscalibration. As they put it,

"Miscalibration is a behavioral bias that is an aspect of overconfidence... miscalibrated people overestimate the precision of their own forecasts, or underestimate the variance of risky processes; in other words, their subjective probability distributions are too narrow."

The study is based on a sample of over 11,600 S&P 500 forecasts made by chief financial officers in the U.S. Each quarter from March 2001 to February 2010, they surveyed CFOs and asked them to predict one- and ten-year stock market returns, as well as the 10th and 90th percentiles of the distribution of market returns (i.e. “worst case” and “best case” scenarios).

If a CFO gives you their expectation for the 10th and 90th percentiles of stock market returns, they have given you an 80 percent confidence interval. You would expect 80% of the returns they were predicting to fall into the interval. However, according to the study,

"only 33% of the time do realized S&P 500 returns fall within the 80% confidence interval that respondents offer. Even during the least volatile quarters in our sample, only 59% of realized returns fall within the 80% confidence intervals provided"

That's bad. The CFOs are far too confident in their predictions, just like the Bernank is overconfident when he appraises his chance of success at "one hundred percent."