|Title:||Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution|
Heck, Daniel W.
|Publisher:||ZPID (Leibniz Institute for Psychology Information)|
|Abstract:||Simonsohn, Nelson, and Simmons (2014a) proposed p-curve – the distribution of statistically significant p-values for a set of studies – as a tool to assess the evidential value of these studies. They argued that, whereas right-skewed p-curves indicate true underlying effects, left-skewed p-curves indicate selective reporting of significant results from a much larger set of tests conducted on the same data when there is no true effect (“p-hacking”). We first review research that criticized the first claim by showing that null effects may indeed produce right-skewed p-curves under some conditions. We then question the second claim by showing that not only selective reporting but also selective non-reporting of significant results (e.g., of an ANCOVA for randomized 2-groups designs) due to a significant outcome of a more popular alternative test of the same hypothesis (e.g., a two-group t-test) may produce left-skewed p-curves, even if all studies included in a p-curve reflect true effects. Thus, although it is true that left-skewed p-curves indicate selection bias, it is possible that the bias is due to studies excluded from the p-curve rather than to those included in it. Hence, just as right-skewed p-curves do not necessarily imply evidential value, left-skewed p-curves do not necessarily imply p-hacking and absence of true effects in the studies involved.|
|Citation:||Erdfelder, E., & Heck, D. W. (2019, March 14). Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2399|
|Appears in Collections:||Conference Object|
|m_1_Erdfelder Open Science Conference Trier 2019.pdf||Conference Talk||2,04 MB||Adobe PDF||Preview PDF Download|
This item is licensed under a Creative Commons License