Replicator degrees of freedom allow publication of misleading failures to replicate

In recent years, the field of psychology has begun to conduct replication tests on a large scale. Here, we show that "replicator degrees of freedom" make it far too easy to obtain and publish false-negative replication results, even while appearing to adhere to strict methodological standards. Specifically, using data from an ongoing debate, we show that commonly exercised flexibility at the experimental design and data analysis stages of replication testing can make it appear that a finding was not replicated when, in fact, it was. The debate that we focus on is representative, on key dimensions, of a large number of other replication tests in psychology that have been published in recent years, suggesting that the lessons of this analysis may be far reaching. The problems with current practice in replication science that we uncover here are particularly worrisome because they are not adequately addressed by the field's standard remedies, including preregistration. Implications for how the field could develop more effective methodological standards for replication are discussed. © 2019 National Academy of Sciences. All rights reserved.

Data deposition: All data and analysis scripts are posted on the Open Science Framework at https://osf.io/y5wsb/. An R package for conducting specification-curve analysis was developed for this work and is available for download at https://github.com/jmobrien/SpecCurve.