people were following expert advice, and so they must have been applying (prospectively) good judgment--is misguided

TO BE NOTED: from Statistical Modeling, Causal Inference, and Social Science:

"Irrationality versus Naivete

By Andrew Gelman on April 7, 2009 9:36 AM | No Comments

Richard Posner defended the rationality of people who bought stocks during the bubble, writing:

People buy common stock when stock prices are rising. They (notoriously) bought houses during the early 2000s when house prices were rising. Since almost no one can predict the ups and downs of the stock market or the housing market, these purchases must have been motivated, Akerlof and Shiller argue, by something other than a rational investment strategy. But this is not at all obvious . . . Stocks have generally been a good investment, at least when held for a considerable period. . . .

I agree with Nate, who disagrees with Richard Posner by pointing out that, in fact, there was evidence that stocks were overpriced during the early 2000s, even at the time.

I'd like to add one comment. During all these bubble years, the experts were telling us over and over again how we should be buying stocks, how stocks were the best investment over the long term, and how we were all irrational for not putting more of our money into the stock market.

What's the logic here? People were being irrational by hesitating to buy stocks when they were going up, then they were finally being rational by buying stocks when they had very high prices?

I think all this discussion is hindered by the overloading of the term "rational." I imagine that just about everybody takes his or her money management seriously, and I'm sure people are trying to behave rationally with their investments. The trouble is that there are lots of rules out there to follow, so there's more than one way to be rational. I agree with Nate that Posner's implicit assumption--that people were following expert advice, and so they must have been applying (prospectively) good judgment--is misguided."

If so, that would be an unfortunate neural response, since all the evidence suggests they really are spouting nonsense.

TO BE NOTED: From The Frontal Cortex:

"Phony Experts

Posted on: March 26, 2009 4:08 PM, by Jonah Lehrer

Nicholas Kristof has a great column today on Philip Tetlock and political experts, who turn out to be astonishingly bad at making accurate predictions:

The expert on experts is Philip Tetlock, a professor at the University of California, Berkeley. His 2005 book, "Expert Political Judgment," is based on two decades of tracking some 82,000 predictions by 284 experts. The experts' forecasts were tracked both on the subjects of their specialties and on subjects that they knew little about.

The result? The predictions of experts were, on average, only a tiny bit better than random guesses -- the equivalent of a chimpanzee throwing darts at a board.

"It made virtually no difference whether participants had doctorates, whether they were economists, political scientists, journalists or historians, whether they had policy experience or access to classified information, or whether they had logged many or few years of experience," Mr. Tetlock wrote.

Why are political pundits so often wrong? In my book, I devote a fair amount of ink to Tetlock's epic study. The central error diagnosed by Tetlock was the sin of certainty: pundits were so convinced they were right that they ended up neglecting all those facts suggesting they were wrong. "The dominant danger [for pundits] remains hubris, the vice of closed-mindedness, of dismissing dissonant possibilities too quickly," Tetlock writes. (This is also why the most eminent pundits were the most unreliable. Being famous led to a false sense of confidence.)

Is there a way to fix this mess? Or is cable news bound to be populated by talking heads who perform worse than random chance? As Kristof notes, Tetlock advocates the creation of a "trans-ideological Consumer Reports for punditry," which doesn't strike me as particularly feasible. Instead, I'd argue that the easiest fix is for anchors on news shows to simply spend more time asking pundits about all those predictions that turned out to be wrong. The point isn't to generate a public shaming - although that would certainly be more entertaining than most cable news shows - but to force pundits to genuflect and introspect on why they made a mistake in the first place. Were they too ideological? Were they thinking like a hedgehog? What relevant facts did they ignore? And why did they ignore them?

By asking the right questions, we can make our pundits better. They'll never be perfect, or even worth their inflated paychecks, but perhaps we can help them perform better than chimpanzees throwing darts. Tetlock ends his book with some take-home advice on what we can all learn from the failures of political experts: "We need to cultivate the art of self-overhearing," he says, "to learn how to eavesdrop on the mental conversations we have with ourselves." A big part of that mental conversation is studying the biases and habits that led us to err.

And here, via Steve, is a new study by Gregory Berns of Emory on how even bad "expert" advice can influence decision-making in the brain. Here's the Wired summary:

In the study, Berns' team hooked 24 college students to brain scanners as they contemplated swapping a guaranteed payment for a chance at a higher lottery payout. Sometimes the students made the decision on their own. At other times they received written advice from Charles Noussair, an Emory University economist who advises the U.S. Federal Reserve.

Though the recommendations were delivered under his imprimatur, Noussair himself wouldn't necessarily follow it. The advice was extremely conservative, often urging students to accept tiny guaranteed payouts rather than playing a lottery with great odds and a high payout. But students tended to follow his advice regardless of the situation, especially when it was bad.

When thinking for themselves, students showed activity in their anterior cingulate cortex and dorsolateral prefrontal cortex -- brain regions associated with making decisions and calculating probabilities. When given advice from Noussair, activity in those regions flat lined.

I'm most interested in the reduced activity seen in the ACC, since that brain area is often associated with cognitive conflict. (It's activated, for instance, when people are presented with contradictory facts or dissonant information.) I think one way to understand the influence of experts is that, because they're "experts," we aren't as motivated to think of all the reasons they might be spouting nonsense. If so, that would be an unfortunate neural response, since all the evidence suggests they really are spouting nonsense."

And it turns out that while foxes don’t give great sound-bites, they are far more likely to get things right

TO BE NOTED: From the NY Times:

"Learning How to Think

By NICHOLAS D. KRISTOF

Ever wonder how financial experts could lead the world over the economic cliff?

One explanation is that so-called experts turn out to be, in many situations, a stunningly poor source of expertise. There’s evidence that what matters in making a sound forecast or decision isn’t so much knowledge or experience as good judgment — or, to be more precise, the way a person’s mind works.

More on that in a moment. First, let’s acknowledge that even very smart people allow themselves to be buffaloed by an apparent “expert” on occasion.

The best example of the awe that an “expert” inspires is the “Dr. Fox effect.” It’s named for a pioneering series of psychology experiments in which an actor was paid to give a meaningless presentation to professional educators.

The actor was introduced as “Dr. Myron L. Fox” (no such real person existed) and was described as an eminent authority on the application of mathematics to human behavior. He then delivered a lecture on “mathematical game theory as applied to physician education” — except that by design it had no point and was completely devoid of substance. However, it was warmly delivered and full of jokes and interesting neologisms.

Afterward, those in attendance were given questionnaires and asked to rate “Dr. Fox.” They were mostly impressed. “Excellent presentation, enjoyed listening,” wrote one. Another protested: “Too intellectual a presentation.”

A different study illustrated the genuflection to “experts” another way. It found that a president who goes on television to make a case moves public opinion only negligibly, by less than a percentage point. But experts who are trotted out on television can move public opinion by more than 3 percentage points, because they seem to be reliable or impartial authorities.

But do experts actually get it right themselves?

The expert on experts is Philip Tetlock, a professor at the University of California, Berkeley. His 2005 book, “Expert Political Judgment,” is based on two decades of tracking some 82,000 predictions by 284 experts. The experts’ forecasts were tracked both on the subjects of their specialties and on subjects that they knew little about.

The result? The predictions of experts were, on average, only a tiny bit better than random guesses — the equivalent of a chimpanzee throwing darts at a board.

“It made virtually no difference whether participants had doctorates, whether they were economists, political scientists, journalists or historians, whether they had policy experience or access to classified information, or whether they had logged many or few years of experience,” Mr. Tetlock wrote.

Indeed, the only consistent predictor was fame — and it was an inverse relationship. The more famous experts did worse than unknown ones. That had to do with a fault in the media. Talent bookers for television shows and reporters tended to call up experts who provided strong, coherent points of view, who saw things in blacks and whites. People who shouted — like, yes, Jim Cramer!

Mr. Tetlock called experts such as these the “hedgehogs,” after a famous distinction by the late Sir Isaiah Berlin (my favorite philosopher) between hedgehogs and foxes. Hedgehogs tend to have a focused worldview, an ideological leaning, strong convictions; foxes are more cautious, more centrist, more likely to adjust their views, more pragmatic, more prone to self-doubt, more inclined to see complexity and nuance. And it turns out that while foxes don’t give great sound-bites, they are far more likely to get things right.

This was the distinction that mattered most among the forecasters, not whether they had expertise. Over all, the foxes did significantly better, both in areas they knew well and in areas they didn’t.

Other studies have confirmed the general sense that expertise is overrated. In one experiment, clinical psychologists did no better than their secretaries in their diagnoses. In another, a white rat in a maze repeatedly beat groups of Yale undergraduates in understanding the optimal way to get food dropped in the maze. The students overanalyzed and saw patterns that didn’t exist, so they were beaten by the rodent.

The marketplace of ideas for now doesn’t clear out bad pundits and bad ideas partly because there’s no accountability. We trumpet our successes and ignore failures — or else attempt to explain that the failure doesn’t count because the situation changed or that we were basically right but the timing was off.

For example, I boast about having warned in 2002 and 2003 that Iraq would be a violent mess after we invaded. But I tend to make excuses for my own incorrect forecast in early 2007 that the troop “surge” would fail.

So what about a system to evaluate us prognosticators? Professor Tetlock suggests that various foundations might try to create a “trans-ideological Consumer Reports for punditry,” monitoring and evaluating the records of various experts and pundits as a public service. I agree: Hold us accountable!"

"Instead, it is to provide certainty. People hate dissonance, doubt and uncertainty. Experts help dispel these. "

Stumbling And Mumbling has an interesting post about experts:

"It’s a bad day for experts. The Times complains that economic forecasters are as blind as ancient soothsayers, whilst proof that Colin Stagg was innocent discredits Paul Britton’s expertise as a forensic pyschologist.
To point out that experts are wrong, however, is to misunderstand the purpose of them. Their function is not to provide knowledge, and still less clear thinking. Instead, it is to provide certainty. People hate dissonance, doubt and uncertainty. Experts help dispel these. So, Paul Britton’s function was to tell the police that they had the right man, whilst economic forecasters’ job is to provide an impression that the future is knowable; no-one wants to hear about standard errors, parameter uncertainty or the Lucas critique.
What’s so pernicious here, though, is that people have ways of achieving an illusory certainty anyway. As Sir Harry Ognall - the judge who acquitted Stagg - says: “The police closed their minds to any other possibility than that of his guilt.”
There are several ways they got these closed minds. All have analogues in corporate planning and financial trading.
1. The confirmation bias. Having acquired the belief that Stagg was guilty - he fitted the profile, was on the scene and a bit of a weirdo - subsequent evidence was interpreted as corroborating this. So, the fact that he looked shifty in interview was seen as evidence of guilt, not as the sign of an innocent man nervous of being fitted up.
A similar thing happens in economic forecasting. If you thought last week that we’re heading for a very deep recession, you put great weight on Wednesdays’s jobless numbers and find ways of dismissing yesterday’s retail sales figures. If you thought the recession would be mild, you do the opposite.
2. The halo effect. In their book, Mistakes were Made, Carol Tavris and Elliot Aronson describe how policemen believe “I couldn’t have been wrong because I’m a good guy.” Even if we grant the premise, the error here lies in believing that good qualities - moral rectitude and cognitive skills - must be correlated. They are not. Of course, coppers are not unique in thinking this.
3. Groupthink. If our colleagues agree with us, our confidence in our judgment rises, especially if we like them.
This error arises in part because we fail to see that correlated data points add little to certainty. If our colleagues have the same training and evidence as us, and are also prone to groupthink, their beliefs will be correlated with ours, and so will not be new evidence - no more than a second copy of the Daily Mail corroborates the stories made in the first. But we interpret them as if they are.
4. Ego-involvement. Admitting that we are wrong means more than just fessing up to narrow technical error. We interpret it as a blow to our ego - a sign that we are not the infallible, uber-competent professionals we think. We’ll do anything to squirm out of facing this. Hence the failure of the police, until yesterday, to apologize to Colin Stagg, and the failure of many bank bosses, Tom McKillop excepted, to apologize for their errors.
And herein lies the purpose of experts. It’s to reinforce these mechanisms, to help people avoid the uncomfortable facts that the world is uncertain, that mistakes are inevitable, and that we are not as in control of things as we think.
Blaming experts for being wrong is like complaining that the economy is not yellow. It’s a category error so howling as to be nonsensical."

I don't take experts very seriously. I do take arguments very seriously. Given that, I consider every case or argument on its own merits. The list does provide some common errors of reasoning, of which there are many.

But surely there are experts. My doctor is an expert, and thank God for that. He is not always right, but he is right enough of the time that I prefer him to my neighbor for help with my ailments. One could argue that he has a skill, while some experts don't. These might the experts talked about. Their skill is dubious, so seeking their views is merely for comfort.

We don't have to look to experts for certainty, but for advice. Advice should be viewed from a pragmatic perspective. We should take it from people who gives us reason to trust it, and ignore from people who don't.

It might well be that people hate dissonance, doubt, and uncertainty. I wouldn't know. They're all I have.