Yang Liu
UC Santa Cruz
Co-Authors: Juntao Wang, Yiling Chen
Aggregation via Peer Assessment
Knowing the accuracy of individuals is of great importance in accurately aggregating the wisdom of crowds to answer questions that usually require expert knowledge. In the real application, however, neither history data of participants nor the verification questions are practically accessible.
In this work, we explore the possibility of using purely peer information to i) access the accuracy of individual participants, and ii) improve the aggregation accuracy. By developing new aggregation framework and running experiments over 18 real-world datasets, we show that carefully crafted peer prediction scores are highly correlated with the true accuracy of participants and that by using these scores as weights, we can improve the performance of existing aggregation methods on forecasting questions as well as scientific study replicable questions.