Monday, October 24, 2011

Managers, Pigeons, and The Perils of Overconfidence

Let's start the week with a pop quiz.

I am going to sit you down in front of a screen where I will flash two lights: red and green. I will tell you the sequence is completely random, but 80% of the time the light will flash green.

And, oh, by the way: one of my assistants will be doing the exact same experiment in the next room, only she will be having a pigeon doing the guessing.

Who do you think will get more right? You or the pigeon?

Now we'll start our experiment. I will start flashing the light, and I want you tell what color is coming up next.

Here's what you should do: I have already told you that 80% of the time the light will flash green, and that the sequence will be totally random. So, the correct guess will always be "green", so you will be right 80% of the time.

But that's not what most humans do. They tend to look for patterns where none exist. Here's how Jason Zweig described our human tendencies in his book Your Money & Your Brain: How the New Science of Neuroeconomics Can Help Make You Rich:

Human, however, tend to flunk this kind of experiment. Instead of just picking green all of the time and locking in an 80 percent chance of being right, people will typically pick green four out of five times, quickly getting caught up in the game of trying to call when the next red flash will come up. On average, this misguided confidence leads people to pick the next flash accurately on only 68 percent of their tries. Stranger still, humans will persist in this behavior even when the researchers tell them explicitly...that the flashing of the lights is random.

Meanwhile, the other room, your pigeon competition is guessing correctly 80% of the time, since the bird automatically picks green every time. Mr. Zweig explains:

...birds seem to stick within their limits of their abilities to identify patterns, giving them what amounts to a kind of natural humility in the face of random events. People, however, are a different story.

In short, the pigeon probably wins.


I was reminded of this story yesterday, when I read a piece by Dan Kahneman in yesterday's New York Times about the perils of overconfidence.

Dr. Kahneman was awarded a Nobel Prize in 2002 for his groundbreaking research into behavior finance. He has also written extensively about how one should view any predictions about future events, especially economic or related to the stock market, with a grain of salt.

In my field, you often read predictions about the future course of the stock market, or the direction of interest rates. These pronouncements are usually made with great confidence, yet are often just as likely to be wrong as they are right.

Just as in the behavior experiment, we humans like to look for patterns where none exist. We might know, for example, that investing in stocks will tend to outperform other asset classes, yet we persist in trading in and out of the market based on our belief that we will be able to "time" the market.

Here's Dr. Kahneman's advice:

We often interact with professionals who and confident predictions even when they know little or nothing. Overconfidence arises because people are often blind to their own blindness. ls who exercise their judgment with evident confidence, sometimes priding themselves on the power of their intuition. In a world rife with illusions of validity and skill, can we trust them? .... We can believe an expert who admits uncertainty but cannot take expressions of high confidence at face value. As I first learned on the obstacle field, people come up with coherent stories and confident predictions even when they know little or nothing. Overconfidence arises because people are often blind to their own blindness.