Article Accepted Manuscript

When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing.

Author(s) / Creator(s)

Rubin, Mark

Abstract / Description

Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.

Keyword(s)

experimentwise error familywise error multiple testing multiple comparisons simultaneous testing Type I error

Persistent Identifier

Date of first publication

2021-07-07

Journal title

Synthese

Publisher

PsychArchives

Publication status

acceptedVersion

Review status

peerReviewed

Is version of

Citation

Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese. https://doi.org/10.1007/s11229-021-03276-4
  • Author(s) / Creator(s)
    Rubin, Mark
  • PsychArchives acquisition timestamp
    2021-07-07T18:41:28Z
  • Made available on
    2021-07-07T18:41:28Z
  • Date of first publication
    2021-07-07
  • Abstract / Description
    Scientists often adjust their significance threshold (alpha level) during null hypothesis significance testing in order to take into account multiple testing and multiple comparisons. This alpha adjustment has become particularly relevant in the context of the replication crisis in science. The present article considers the conditions in which this alpha adjustment is appropriate and the conditions in which it is inappropriate. A distinction is drawn between three types of multiple testing: disjunction testing, conjunction testing, and individual testing. It is argued that alpha adjustment is only appropriate in the case of disjunction testing, in which at least one test result must be significant in order to reject the associated joint null hypothesis. Alpha adjustment is inappropriate in the case of conjunction testing, in which all relevant results must be significant in order to reject the joint null hypothesis. Alpha adjustment is also inappropriate in the case of individual testing, in which each individual result must be significant in order to reject each associated individual null hypothesis. The conditions under which each of these three types of multiple testing is warranted are examined. It is concluded that researchers should not automatically (mindlessly) assume that alpha adjustment is necessary during multiple testing. Illustrations are provided in relation to joint studywise hypotheses and joint multiway ANOVAwise hypotheses.
    en
  • Publication status
    acceptedVersion
    en
  • Review status
    peerReviewed
    en
  • Citation
    Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese. https://doi.org/10.1007/s11229-021-03276-4
    en
  • ISSN
    1573-0964
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/4388
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.4960
  • Language of content
    eng
  • Publisher
    PsychArchives
    en
  • Is version of
    https://doi.org/10.1007/s11229-021-03276-4
  • Keyword(s)
    experimentwise error
    en
  • Keyword(s)
    familywise error
    en
  • Keyword(s)
    multiple testing
    en
  • Keyword(s)
    multiple comparisons
    en
  • Keyword(s)
    simultaneous testing
    en
  • Keyword(s)
    Type I error
    en
  • Dewey Decimal Classification number(s)
    150
  • Title
    When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing.
    en
  • DRO type
    article
    en
  • Journal title
    Synthese
    en
  • Visible tag(s)
    Accepted Manuscript