Facts Aren't Enough

A data scientist provides a few simple examples of why we have the maxim, "Extraordinary claims require extraordinary evidence". Contending that, "If your results are highly counterintuitive, they are almost certainly wrong", the author notes a finding from early in his career, working in the insurance industry:

... I discovered something amazing! Several of the most junior people in the department were fantastic, nearly perfect on all metrics. Several of the most senior people had performance all over the map. They were significantly below average on most metrics! Most of the claims money was spent on these underperformers! Big data had proven that a whole department in a company was nonsense lunacy!

But data exist within a context, and computers don't hold context. Strangely enough, it was context that showed this conclusion to be wrong:

Not so fast. Anyone with any insurance experience (or half a brain, or less of an arrogant physics-is-the-best mentality) would have realized something right away -- the kinds of claims handled by junior people are going to be different. Everything that a manager thought could be handled easily by someone fresh to the business went to the new guys. Simple cases, no headaches, assess the cost, pay the cost, done.

Cases with lots of complications (maybe uncertain liability, weird accidents, etc.) went to the senior people. Of course outcomes looked worse, more variance per claim makes the risk much harder to manage. I was the idiot, and misinterpreting my own results! [bold added, minor format edits]

Consider the above also a good example of why we also have the phrase, "Lies, damned lies, and statistics." I have noted here that data alone, even properly interpreted, will not sway everyone. That problem is compounded by another: Some hope to take advantage of a common kind of gullibility about data and allegedly sophisticated analysis. That kind of hucksterism, which is exposed often enough, thereby gives difficult analysis a bad name and make the job of those who really are concerned with relevant facts much more difficult.

Some people refuse to consider evidence at all: They are of little concern in a public debate. On the other hand, those who have been jaded by a stream of ridiculous claims might listen -- if one can get past an understandable kind of resistance.

-- CAV