Michael Gordon
Massey University
Co-Authors: Thomas Pfeiffer, Domenico Viganola
Aggregating Replication Outcome Forecasts with Surveys and Prediction Markets
A number of large-scale replication projects were initiated in the behavioural and social sciences to systematically evaluate a large sample of studies from specific research fields through direct replication. Four of these large scale studies were accompanied by prediction markets and surveys aimed at forecasting the replication outcomes before the replications were conducted. We here analyse the pooled data from these four studies that in total elicited peer beliefs about the replication outcomes of 103 published studies. By pooling the data of the four projects, we can substantially increase the statistical power to test the performance of the prediction markets and surveys. We furthermore evaluate methods to aggregate individual survey responses. We find that while prediction market prices are better predictors of replication outcomes than average survey response, alternative survey aggregation methods can yield a similar forecasting performance.