Persuade Me to Listen
Thursday, June 07, 2012
Math blogger Scott Aaronson, reacting to the question of whether he'd read a
paper purported to be a major breakthrough nicely sums up a problem I
ran into some time back:
If I read all such papers, then I wouldn't have time for anything else. It's an interesting question how you decide whether a given paper crosses the plausibility threshold or not. For me personally, the AKS "PRIMES in P" paper somehow crossed it whereas this one somehow doesn't.Continuing, he noted that, "Three commenters wrote in to say the paper looked good. Then the author found a bug and retracted it." The mathematician then went on to discuss "Ten Signs a Claimed Mathematical Breakthrough is Wrong", but not without admitting inspiration from an amusing (but instructive) piece in Discover Magazine, "The Alternative-Science Respectability Checklist". There, author Sean Carroll makes a similar point to the mathematician I quoted above:
Of course, I'd welcome an opinion from anyone who's actually read the paper.
Scientists can't possibly pay equal attention to every conceivable hypothesis, they would literally never do anything else. Whether explicitly or not, they typically apply a Bayesian prior to the claims that are put before them. Purported breakthroughs are not all treated equally; if something runs up against their pre-existing notions of how the universe works, they are much less likely to pay it any attention. So what does it take for the truly important discoveries to get taken seriously? [link omitted]Carroll then goes on to elaborate, in great detail, why a scientist dismissing a purported breakthrough does so, although he makes the point very sarcastically.
I'll share a passage I found particularly amusing.
Likewise, if your breakthrough is an experiment, it had better be a dramatically obvious one -- and the more you are violating cherished scientific beliefs, the more dramatic the effect had better be. If what you're claiming requires a re-arrangement of the energy levels in organic molecules, in flagrant disregard of the Schrodinger equation, you are going to need much more than a two- or three-sigma effect. And, equally importantly, you have to be up front about what the apparatus is, so that anyone can reproduce the experiment. No fair saying "Well, if you come into my lab, I'll turn it on and show you how it works." And "This experiment was done in the '70′s in a secret underground lab in Gdansk, and the KGB has suppressed the lab notebooks" isn't any better. If you're actually playing the role of a scientist, share your procedure with everyone, so that they can become true believers themselves. If, on the other hand, you just want to make money, then by all means don't tell anyone; just start producing the free energy (or amazing stretchy widgets, or whatever) and sell it on the open market. The millions of dollars that will doubtless flow your way will be very comforting as you rail against the establishment for failing to appreciate your genius.What Aaronson and Carroll are describing (from the standpoint of someone being told of an alleged breakthrough) are heuristics, and each admits that such rules-of-thumb can cause them to miss a breakthrough from time to time. But both have a valuable points for anyone wanting to challenge an orthodoxy. (1) Actually know what the orthodoxy says and why. (2) Know why said orthodoxy is wrong and you are correct. (3) Be able to help someone who upholds the orthodoxy see, in terms they will understand, why they should listen to you.
The obviousness of these points explains (via weariness) the sarcasm in Carroll's piece: As he implies and I have often seen in my own personal experience, most people who champion something from left field have failed to do at least one of the above things, and usually, one of the first two. Only rarely is someone who has done all his homework such a poor communicator that he will only look like a crackpot. But it does happen.
-- CAV
5 comments:
Heh, this is something I run into often enough in linguistics. Of course, there's a wide range of linguistics cranks, since language is very important to people. So first you get the chauvinists: Most common, or at least loudest, are Sanskrit supremacists, Turkic supremacists--not helped by the fact that in the 1930s the Turkish government supported the crank "Sun Theory" for ideolgical reasons--and Greek supremacists (I remember one of them shouting in writing that Proto-Indo-European was an excrescence of anti-Greek bigotry that was also revealed in the fact that the United States refused to give Greece a few thousand neutron boms so they could re-establish the Byzantine Empire in Anatolia and the Balkans. Yes, really, and contrary to the stereotypes the guy was a hard socialist with strong Communist sympathies...hmm, what's Greek for "pestiferous scum"?). But I've even encountered Nahuatl supremacists (the language of the Aztecs and of some of their bitterest blood enemies) who point to such coincidences as "Michigan" sounding much like the Nahuatl for "place of fish" as evidence the Aztecs made it all the way to the Arctic and by right own the Americas.
Then you have the cranks who are more into grand paranoid theories of everything and damn the academics.
Those are easy to avoid. Then you have the harder problem of working through stuff that looks good at first glance but doesn't end up meaning anything. One example is a paper I read in a seminar on speech perception in which the fellows who wrote it ran four separate perception experiments, but varied two experimental parameters between each experiment, so the paper ended up being almost useless since there was no way to know which effects were due to which factors! Then there was a paper I read and reported on for a seminar in language typology that at first glance seemed like very interesting results about what consonants you can expect languages to have given other classes of consonants in them...but it turns out, if you went back to the original source they took their data from, they had without realizing it simply reclassified the classifications of the data, mucked about with statistics, and recapitulated the work of the original, and ended up with 100% correlations! The result was true by the definitions in the original source, if you dug into the details of what they did, and one suspects someone had put off writing his paper until the night before and fooled himself.
What he is saying is something I have been pounding on for quite some time. There is more to science than just data. You also need a good explanation (that means a logical mechanism for the data) as well. Your theory should mesh with the prevailing theories of how the world works; theories which have been built by mountains of data over decades; even centuries. If it doesn’t, then you’ve got some ‘splaining to do’. You will need a lot more data, a lot better data, a good explanation for the discrepancy AND a whole lot of time for people to mull it over and accept it.
A good example of this is the research into possible psychic phenomena. If you look at the data carefully you can pull out tiny statistical effects which seem to ‘prove’ remote sensing and other non sensory methods of taking in information from the environment. But the effects are tiny and hard to reproduce and most importantly there is NO known mechanism to account for them. So, most scientists would rather believe that other factors are contaminating these experiments, and leading to these effects, thereby producing only an illusion that ESP exists.
Mike,
Your account of linguistic supremacists reminds me of Gus Portokalos' "derivation" of kimono in My Big Fat Greek Wedding. Perhaps we can re-name the approach of this type of crank the "Gus Portokalos School" of linguistics.
Steve,
Indeed the challenge in such experiments lies well beyond finding data. Why would someone even run a test without having (or giving) a good reason to think he'd get results?
Gus
Gus, good question. It reminded me of a related one I've often asked of religious folks - why would anyone ever come up with the idea of a God (or more generally, the supernatural) in the first place? What set of facts, logic or data would make you need or want to think up that concept to try to explain them? Why not try something (anything) else, first.
I like that, and perhaps the answer you get can help gauge whether someone is second-handed, or at least how ingrained the notion of God is to someone.
Most people have this notion drilled into their heads from an early age, which explains why most people even have it. But some will dare to look at the idea in the same way they look at any other, and question it. (Your question might help them and even earn you thanks.) Others will refuse to do this at all. (They are the ones who are beyond any immediate help.)
Post a Comment