Conference Object

Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution

Author(s) / Creator(s)

Erdfelder, Edgar
Heck, Daniel W.

Abstract / Description

Simonsohn, Nelson, and Simmons (2014a) proposed p-curve – the distribution of statistically significant p-values for a set of studies – as a tool to assess the evidential value of these studies. They argued that, whereas right-skewed p-curves indicate true underlying effects, left-skewed p-curves indicate selective reporting of significant results from a much larger set of tests conducted on the same data when there is no true effect (“p-hacking”). We first review research that criticized the first claim by showing that null effects may indeed produce right-skewed p-curves under some conditions. We then question the second claim by showing that not only selective reporting but also selective non-reporting of significant results (e.g., of an ANCOVA for randomized 2-groups designs) due to a significant outcome of a more popular alternative test of the same hypothesis (e.g., a two-group t-test) may produce left-skewed p-curves, even if all studies included in a p-curve reflect true effects. Thus, although it is true that left-skewed p-curves indicate selection bias, it is possible that the bias is due to studies excluded from the p-curve rather than to those included in it. Hence, just as right-skewed p-curves do not necessarily imply evidential value, left-skewed p-curves do not necessarily imply p-hacking and absence of true effects in the studies involved.

Persistent Identifier

Date of first publication

2019-03-14

Is part of

Open Science 2019, Trier, Germany

Publisher

ZPID (Leibniz Institute for Psychology Information)

Citation

Erdfelder, E., & Heck, D. W. (2019, March 14). Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2399
  • Author(s) / Creator(s)
    Erdfelder, Edgar
  • Author(s) / Creator(s)
    Heck, Daniel W.
  • PsychArchives acquisition timestamp
    2019-04-03T12:58:30Z
  • Made available on
    2019-04-03T12:58:30Z
  • Date of first publication
    2019-03-14
  • Abstract / Description
    Simonsohn, Nelson, and Simmons (2014a) proposed p-curve – the distribution of statistically significant p-values for a set of studies – as a tool to assess the evidential value of these studies. They argued that, whereas right-skewed p-curves indicate true underlying effects, left-skewed p-curves indicate selective reporting of significant results from a much larger set of tests conducted on the same data when there is no true effect (“p-hacking”). We first review research that criticized the first claim by showing that null effects may indeed produce right-skewed p-curves under some conditions. We then question the second claim by showing that not only selective reporting but also selective non-reporting of significant results (e.g., of an ANCOVA for randomized 2-groups designs) due to a significant outcome of a more popular alternative test of the same hypothesis (e.g., a two-group t-test) may produce left-skewed p-curves, even if all studies included in a p-curve reflect true effects. Thus, although it is true that left-skewed p-curves indicate selection bias, it is possible that the bias is due to studies excluded from the p-curve rather than to those included in it. Hence, just as right-skewed p-curves do not necessarily imply evidential value, left-skewed p-curves do not necessarily imply p-hacking and absence of true effects in the studies involved.
    en_US
  • Citation
    Erdfelder, E., & Heck, D. W. (2019, March 14). Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2399
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/2031
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.2399
  • Language of content
    eng
    en_US
  • Publisher
    ZPID (Leibniz Institute for Psychology Information)
    en_US
  • Is part of
    Open Science 2019, Trier, Germany
    en_US
  • Dewey Decimal Classification number(s)
    150
  • Title
    Detecting Evidential Value and P-Hacking With the P-curve tool: A Word of Caution
    en_US
  • DRO type
    conferenceObject
    en_US