Heterogeneity in the Results of Close and Conceptual Replications: Implications for Scientific Progress and Practical Applications

Background: Replicability of the results of original studies and close replications have become a prominent concern in psychology. In particular, heterogeneity, i.e. variability in the results of replication studies over and above what is expected due to sampling error, has received increased attention, because it decreases the statistical power of studies. Heterogeneity-induced low power might therefore explain the low success in Open Science Collaboration’s (2015) 100 close replications of studies in cognitive and social psychology (Stanley, Carter, & Doucouliagos, in press). Objectives: To derive a reliable estimate of heterogeneity in close replication studies and to compare this to levels of heterogeneity in meta-analyses (conceptual replications). To investigate to what extent moderators explain heterogeneity. To investigate some of the possible causes of high heterogeneity. Research question(s) and/or hypothesis/es: We expect heterogeneity in close replications to be low. Based on the results in Open Science Collaboration (2015), we expect that heterogeneity in conceptual replications will be larger in social psychology than in cognitive psychology. Based on Michell et al. (2012) we expect that heterogeneity in social psychology will be larger than in organisational psychology. Method/Approach: We analysed heterogeneity in all available Many-Labs type replications (n = 40). We compared this against heterogeneity in conceptual replications observed in 147 meta-analyses in cognitive, social, and organisational psychology. In all cases, we used Cohen’s d as a measure of effect size, and τ, the standard deviation of true population effect sizes, to quantify heterogeneity. Results/Findings: Heterogeneity in close replications was found to be typically very low (T = 0.08). In contrast, heterogeneity turned out to be very high in the average (conceptual replication) meta-analysis (T = 0.33). This was largely unexplained by moderators, but appeared strongly driven by the magnitude of effects. The distinctive success rates of close replications in cognitive and social psychology (Open Science Collaboration) were not reflected in those disciplines’ heterogeneity levels in conceptual replications. Conclusions and implications (expected): We argue (contrary to Stanley et al., in press) that this pattern of results indicates sufficient power in Open Science Collaboration’s close replications.