A Bayesian bird's eye view of ‘Replications of important results in social psychology’

• Main Authors: Maarten Marsman, Felix D. Schönbrodt, Richard D. Morey, Yuling Yao, Andrew Gelman, Eric-Jan Wagenmakers
• Format: Article
• Language: English
• Published: The Royal Society 2017-01-01
• Series: Royal Society Open Science
• Subjects: preregistration; evidence; reproducibility; credible interval; bayes factor
• Online Access: https://royalsocietypublishing.org/doi/pdf/10.1098/rsos.160426

We applied three Bayesian methods to reanalyse the preregistered contributions to the Social Psychology special issue ‘Replications of Important Results in Social Psychology’ (Nosek & Lakens. 2014 Registered reports: a method to increase the credibility of published results. Soc. Psychol. 45, 137–141. (doi:10.1027/1864-9335/a000192)). First, individual-experiment Bayesian parameter estimation revealed that for directed effect size measures, only three out of 44 central 95% credible intervals did not overlap with zero and fell in the expected direction. For undirected effect size measures, only four out of 59 credible intervals contained values greater than 0.10 (10% of variance explained) and only 19 intervals contained values larger than 0.05. Second, a Bayesian random-effects meta-analysis for all 38 t-tests showed that only one out of the 38 hierarchically estimated credible intervals did not overlap with zero and fell in the expected direction. Third, a Bayes factor hypothesis test was used to quantify the evidence for the null hypothesis against a default one-sided alternative. Only seven out of 60 Bayes factors indicated non-anecdotal support in favour of the alternative hypothesis (BF10>3), whereas 51 Bayes factors indicated at least some support for the null hypothesis. We hope that future analyses of replication success will embrace a more inclusive statistical approach by adopting a wider range of complementary techniques.