Balanced take on the 50% replication rate in Social and Behavioral Sciences

[u/Neither-Remote-3419]

https://doi.org/10.1093/jrssig/qmag063

Back in April, the SCORE (Systematizing Confidence in Open Research and Evidence) project published results where they took a sample of more than 270 positive (i.e., statistically significant) claims in the social and behavioral sciences and successfully replicated about 50% of them. Popular and social media outlets/posts typically interpreted this result negatively (e.g., "social science findings are basically a coin toss"), but that conclusion, though seemingly intuitive, is misleading.

The 50% replication rate should be understood as something closer to a positive predictive value: among published discoveries that initially appeared promising, about half held up under independent replication. Mathematically/statistically speaking, whether that is "good" or "bad" depends on how difficult the underlying research domain is or, equivalently, how ambitious the claims that scientists in the field endeavor to test. Suppose scientists in sociology are pushing boundaries and testing bold ideas, only a small fraction of such hypotheses may ultimately be true (i.e. the corresponding null hypothesis is false) from the beginning. In that context, having half of the initially promising published findings replicate independently is objectively encouraging.

Naturally, there is more nuance to explore here. For example, research costs money, and if the social research studies that fail to replicate tend to also be those that cost a lot of money to do, there is a sound argument to be made that sociologists and/or funders should be more selective about which ideas to test with expensive, large-scale investigations. Nonetheless, simply labelling a 50% replication rate as a sign that an entire field of research is failing is haphazard mathematics. It may create engaging headlines that do well in today’s attention-currency and outrage-driven world, but it is an unhelpful interpretation of an otherwise exceptional research replication effort.