Please use this identifier to cite or link to this item:
Title: Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution
Authors: Erdfelder, Edgar
Heck, Daniel W.
Issue Date: 14-Mar-2019
Publisher: ZPID (Leibniz Institute for Psychology Information)
Abstract: Simonsohn, Nelson, and Simmons (2014a) proposed p-curve – the distribution of statistically significant p-values for a set of studies – as a tool to assess the evidential value of these studies. They argued that, whereas right-skewed p-curves indicate true underlying effects, left-skewed p-curves indicate selective reporting of significant results from a much larger set of tests conducted on the same data when there is no true effect (“p-hacking”). We first review research that criticized the first claim by showing that null effects may indeed produce right-skewed p-curves under some conditions. We then question the second claim by showing that not only selective reporting but also selective non-reporting of significant results (e.g., of an ANCOVA for randomized 2-groups designs) due to a significant outcome of a more popular alternative test of the same hypothesis (e.g., a two-group t-test) may produce left-skewed p-curves, even if all studies included in a p-curve reflect true effects. Thus, although it is true that left-skewed p-curves indicate selection bias, it is possible that the bias is due to studies excluded from the p-curve rather than to those included in it. Hence, just as right-skewed p-curves do not necessarily imply evidential value, left-skewed p-curves do not necessarily imply p-hacking and absence of true effects in the studies involved.
Citation: Erdfelder, E., & Heck, D. W. (2019, March 14). Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution. ZPID (Leibniz Institute for Psychology Information).
Appears in Collections:Conference Object

Files in This Item:
File Description SizeFormat 
m_1_Erdfelder Open Science Conference Trier 2019.pdfConference Talk2,04 MBAdobe PDF Preview PDFDownload

This item is licensed under a Creative Commons License Creative Commons