Katherine S. Corker
Abstract: The Magnitude of Between Sample Differences in Psychological Effects
When researchers combine results of studies (e.g., in meta-analysis), they estimate not only average effect sizes, but also heterogeneity. The magnitude of heterogeneity quantifies between sample differences in effects (after taking account of sampling error). If heterogeneity is large, it suggests unexplained between sample differences may be responsible for differences in observed effects. Meta-analyses of psychology studies often find evidence of substantial heterogeneity, whereas large, pre-registered experiments have found little trace of the same. I investigate these discrepancies in the context of the ongoing “credibility revolution” in psychology with examples from large multi-site studies (Many Labs 2 – Klein et al., 2018; EAMMi2 – Grahe et al., in press; MUSIC – Castillo & Schwartz, 2013) to assess the conditions under which we might expect heterogeneity to be large vs. small, along with conditions under which heterogeneity might be accurately vs. inaccurately estimated.
Poster: The Magnitude of Between Sample Differences in Psychological Effects
