Our July paper Are replication rates the same across academic fields? Community forecasts from the DARPA SCORE programme was briefly cited at the end of a recent unpublished manuscript by Tosh et al, including Gelman. We’re source 13:
5.3 Implications for social science research
As noted at the beginning of this article, there has been a crisis in psychology, economics, and other areas of social science, with prominent findings and apparently strong effects that do not appear in attempted replications by outside research groups (e.g., [24, 2, 13]).
Always happy for the shout out, but our paper [13] doesn’t do replications. It just shows what our forecasters generally expected in upcoming replications, and that before looking at the actual study candidates. About 60 of those replications have happened, but have to wait for the Center for Open Science to announce them. In the meantime, Anna Dreber Almenberg has reached out to Gelman’s team.
The unpublished “Gelman” manuscript is:
The piranha problem: Large effects swimming in a small pond by C Tosh, P Greengard, B Goodrich, A Gelman, D Hsu. Abstract:
In some scientific fields, it is common to have certain variables of interest that are of particular importance and for which there are many studies indicating a relationship with a different explanatory variable. In such cases, particularly those where no relationships are known among explanatory variables, it is worth asking under what conditions it is possible for all such claimed effects to exist simultaneously. This paper addresses this question with formal theorems that show, unless the explanatory variables also have sizable effects on each other, it is impossible to have many such large effects.
Phil Tetlock (and no doubt others) have noted informally that something like this must hold.
I look forward to reading this piece on my new Remarkable tablet.