Easier to Catch

The title of a recent posting by John Cook over at the Endeavour indirectly raises a question. In "Bad Science Is Tolerable, but Resume Padding Is Not," Cook discusses a scandal in cancer research conducted by Anil Potti that took the finding of resume padding on Potti's part to set off. Of course, no one but a fraud  would really "tolerate" bad science, so the first question, "Why?" that comes to mind is wrong, or perhaps rhetorical. This quickly becomes evident in Cook's post from the fact that some scientists have been crying foul ever since 2007.

The real question is this: "Why does it often take something unrelated to bad science to draw attention to bad science?" Cook quotes Keith Baggerly, one of the scientists who worked to expose Potti: "I find it ironic that we have been yelling for three years about the science, which has the potential to be very damaging to patients, but that was not what has started things rolling."

An important part of the problem I think lies in the nature of scientific research itself. How many members of the general public are going to be capable of critically analyzing a new result in a highly specialized field, or even understanding a calling-into-question like that Baggerly helped perform several years ago, and that Cook describes as "extraordinary" in another post? Often, peer review is hard enough for scientists in a position to understand the results and how to think about them.

Published analyses of complex data sets, such as microarray experiments, are seldom exactly reproducible. Authors inevitably leave out some detail of how they got their numbers. In a complex analysis, it’s difficult to remember everything that was done. And even if authors were meticulous to document every step of the analysis, journals do not want to publish such great detail. Often an article provides enough clues that a persistent statistician can approximately reproduce the conclusions. But sometimes the analysis is opaque or just plain wrong.

... Baggerly explained the extraordinary steps he and his colleagues went through in an attempt to reproduce the results in a medical article published last year by Potti et al. He called this process "forensic bioinformatics," attempting to reconstruct the process that lead to the published conclusions. He showed how he could reproduce parts of the results in the article in question by, among other things, reversing the labels on some of the groups. (For details, see "Microarrays: retracing steps" by Kevin Coombes, Jing Wang, and Keith Baggerly in Nature Medicine, November 2007, pp 1276-1277.)

But the fact that science, unlike philosophy, is not always accessible to the layman is hardly the whole picture. Cook goes on to explain that Baggerly and his colleagues went out of their way to make it easy for others to reproduce their analysis of the disputed result. The Economist article Cook points to in his first post discusses some other aspects of the way the peer review process works that can result in shoddy work flying under the radar for some time.

I suspect that another piece of the puzzle is cultural. It would take effort to understand what Potti's opponents did, no matter how easy they made it to do this (and probably no matter how well-described they might be in popular science media). Many laymen might not make the effort to understand what was going on, and perhaps many popular science reporters, responding to the prevalence of this type of reader, stick to stories that sound exciting, as Potti's results did when they were first reported. The bright and shiny new result can be exciting to anyone, but few have the patience for real-life detective work.

When most people are not in the habit of integrating their knowledge, too many things people could actually grasp end up getting neglected, and the only "arguments" many people ever engage in are of the pointless, "bike shed" variety. And all too often, the term "bike shed argument" would be generous to a fault. In such a cultural atmosphere, the lie on the resume gets the headline.

I am left wondering: Had critical thinking deeper prevalence in our culture, would the "Australian Rhodes Scholar" who perpetrated this fraud ever been hired in the first place?

-- CAV