University of California, Santa Barbara
Co-Authors: John Protzko, Brian Nosek, Charlie Ebersole, Jon Krosnick, Sebastian Lundmark, Matt Berent, Matt Debell, Bo MacInnis, Leif Nelson, Hannah Perfecto, Mike O’Donnell, Scott Roeder, Jan Walleczek
How replicable can psychological science be?: A highly powered multi-site investigation of the robustness of newly discovered findings
There has been an increasing concern among scientists regarding irreproducibility of scientific findings in general, and psychological findings in particular. To date, the understanding of reproducibility has been impeded by two related challenges: 1) the lack of transparency of the scientific record, and 2) the retrospective nature of reproducibility studies. In order to overcome these obstacles, four labs conducted a large scale multi-site prospective multi-replication study. Each lab independently discovered new psychological findings that were then systematically replicated by the originating laboratory and by the others, following a complex pre-specified sequence of various replications and analyses. In so doing, this project 1) developed a gold standard for replication protocol, in which every effort was made to design experiments and implement replications in a manner that simultaneously maintained ecological validity while maximizing the likelihood of full replicability, and 2) tested whether the replications of newly devised experimental protocols are associated with declining effect sizes, even when all reasonable efforts are made to minimize such declines. Although the project is still underway, preliminary analyses indicate that when a gold standard approach is applied psychological findings are highly robust.