Saturday, November 06, 2010
Lies, Damned Lies, and ... Medical Research?
Years ago, in our lab's journal club, a colleague presented a curiously-titled 2005 paper by one John Ioannidis, "Why Most Published Research Findings Are False." Insufficiently intrigued by the presentation and too busy at the time to read it or follow up on it, I all but forgot about the paper.
But Ioannidis hasn't gone away. He is now getting attention in the popular press for this work and a companion paper, "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research" (PDF). The first of these papers is a mathematical model of a suspicion I have had for some time about certain areas of research and the second takes a look at the research literature as a check.
The Atlantic presents the above work for a lay audience. It's long, but worth thinking about.
Ioannidis was putting his contentions to the test not against run-of-the-mill research, or even merely well-accepted research, but against the absolute tip of the research pyramid. Of the 49 articles, 45 claimed to have uncovered effective interventions. Thirty-four of these claims had been retested, and 14 of these, or 41 percent, had been convincingly shown to be wrong or significantly exaggerated. If between a third and a half of the most acclaimed research in medicine was proving untrustworthy, the scope and impact of the problem were undeniable. That article was published in the Journal of the American Medical Association.This being science, Ioannidis is not being taken at his word. For example, two authors argued in 2007 that Ioannidis's analysis is suspect.
We agree with the paper's conclusions and recommendations that many medical research findings are less definitive than readers suspect, that P-values are widely misinterpreted, that bias of various forms is widespread, that multiple approaches are needed to prevent the literature from being systematically biased and the need for more data on the prevalence of false claims. But calculating the unreliability of the medical research literature, in whole or in part, requires more empirical evidence and different inferential models than were used. The claim that "most research findings are false for most research designs and for most fields" must be considered as yet unproven.Ioannidis has since answered them. The debate goes on, but the point is well taken.
"Don't forget who put you in office and why -- namely, the independent-minded Tea Party voters." -- Paul Hsieh in "GOP: Dance With The One Who Brung You" at Pajamas Media
"... [M]y message to conservatives this time is, 'I've given you a Republican voter... if you can keep me.'" -- Jared Rhodes in "A Republican Voter ..." at the web site of the Lucidicus Project (HT: Amit Ghate)
"What really frightens me -- as both an investor and a citizen of the [planet] -- is 'eco-horror,' a genre of storytelling now seen in fine art galleries, movie theaters and basic cable. Right now, there's a prime example on display inside the Museum of London." -- Jonathan Hoenig in "Forget Halloween: It's the Greens that Scare Me" at SmartMoney
Comment of the Week
A fellow blogger makes an important clarification.
"[T]he fact of morally condemning in and of itself is not sufficient to obligate explanation." -- Kendall J
What if ...
... you really needed to move an enormous generator?