According to Tim Harford’s write-up, the Good Judgment Project finds the following makes people better forecasters: some basic training in probabilistic reasoning; working in teams; having an ‘actively open-minded’ thinking style. And, you can improve your performance by tracking it. Thought-provoking, eh? When I learned this, I immediately wondered: would not all four factors apply to people’s performance in observing, studying, analyzing and learning about things in general? What about social science researchers? We know there is room for all kinds of bias in social science research; couldn’t these insights be applied to improve social science research? Could they be applied to give us a ‘red flag’ for studies that might manifest more a researcher’s “hedgehogian’ world view than an open-minded look at the evidence?
I should come clean. I am especially interested in the implications of an ‘open minded thinking style’. Because, if this thinking style is associated with more trustworthy research, then I have discovered some foundation for one of my long-standing rules-of-thumb: disregard the work of social science researchers whose findings consistently confirm the superiority of reforms moving social systems in one direction. Yes. I call them ‘hedgehog’ researchers. Though…not to their face.
Let me explain what I mean by “directionality”. Most studies of social system reform strategies can be categorized according to a “direction of change” in which the intervention moves the social system. Common directions include: more role for citizens & communities; larger role for markets; larger role for government, etc. Directionality can be more fine-grained, however. Reforms can be characterized in terms of whether they move a system in the direction of a social system prototype (e.g. an “NHS” like health system; a Bismarckian social health insurance system). Much health services research assesses the soundness of reform strategies which move health systems ‘away from” or “closer to” one of these prototypes. Some researchers do many studies that use the social system of their native land as the prototype-benchmark. I have encountered this often with respect to health systems researchers, and especially those from the UK. Delve into their analyses, and you find they are often assessing the soundness of reform strategies which move health systems ‘away from’ or ‘closer to’ the structure of their very own NHS.
If you categorize researchers’ studies in this way, many show no pattern of directionality. A few are…worryingly consistent. For one, every initiative to expand the role of market forces generates positive results. For another, every strategy that moves a health system to be more “NHS-like” is better than the alternative.
Once I started tracking this pattern, I intuitively stopped trusting these Researchers of Unusual Consistency (R.O.U.C.s). I now see this consistency as a sign that the researcher lacks an ‘actively open-minded thinking style’. And, since the Good Judgment Project findings suggest that this thinking style is protective against bias, I think others would do well to keep their eyes open for such patterns.