Prestige-Driven "Science"?

Bad Science blogger Ben Goldacre notes the startling commonality of a certain type of statistical error in the academic literature of the field of neuroscience:

We all like to laugh at quacks when they misuse basic statistics. But what if academics, en masse, deploy errors that are equally foolish? This week Sander Nieuwenhuis and colleagues publish a mighty torpedo in the journal Nature Neuroscience.

They've identified one direct, stark statistical error that is so widespread it appears in about half of all the published papers surveyed from the academic neuroscience research literature. [bold added, link dropped, minor format edits]

Goldacre goes on to describe the nature of the error in layman's terms, and then speculates as to the answer to the obvious question, "Why?"

These errors are appearing throughout the most prestigious journals for the field of neuroscience. How can we explain that? Analysing data correctly, to identify a "difference in differences", is a little tricksy, so thinking very generously, we might suggest that researchers worry it's too longwinded for a paper, or too difficult for readers. Alternatively, perhaps less generously, we might decide it's too tricky for the researchers themselves.

But the darkest thought of all is this: analysing a "difference in differences" properly is much less likely to give you a statistically significant result, and so it's much less likely to produce the kind of positive finding you need to look good on your CV, get claps at conferences, and feel good in your belly. Seriously: I hope this is all just incompetence. [format edits, bold added]

Either possibility is disturbing to contemplate, but I see no reason to doubt that such slipshod methodology doesn't infect other fields, especially given work, such as that by John Ioannidis, that shows how dubious so many major medical findings, many receiving large amounts of press, have turned out to be.

-- CAV

--- In Other News ---

Here's a news story for anyone who is hoping that popular discontent with the mullahs will lead to the overthrow of Iran's theocratic regime: "Iranian Protestors Storm UK Compound in Tehran."

Statistician John Cook makes an interesting observation about the ad hominem fallacy: "A statement isn't necessarily false because it comes from an unreliable source, though it is more likely to be false." [format edits]

Heh: "How To Write Unmaintainable Code"