Conference Object

Reproducibility-related Reporting Practices in Meta-analyses on the Effectiveness of Psychological Interventions

Author(s) / Creator(s)

Lopez-Nicolas, Ruben
Sanchez-Meca, Julio
Lopez-Lopez, Jose Antonio
Rubio-Aparicio, Maria

Abstract / Description

Background: The last decade has revealed significant problems in terms of reproducibility in psychological research. The scientific psychology community has detected various errors and questionable research practices in the process of studying psychological phenomena. In this context, different proposals have been put forward in order to improve transparency and reproducibility practices. In the present study, we focus on meta-analytic methodology. Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data or reanalysed applying new and more refined analysis techniques. Objetives: We attempted to empirically assess the prevalence of reproducibility-related methodological reporting practices in published meta-analyses of psychological interventions. Our purpose was to identify the key points that could be improved with the aim to provide some recommendations to carry out reproducible meta-analyses. Research question: Are the meta-analytical reports comprehensive enough to allow for potential reproduction attempts? What are the key points where there is a lack of reporting? Method: We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. Then, we selected a random sample of 100 meta-analyses to examine and extract a range of transparency and reproducibility-related indicators using an ad hoc checklist based on existing meta-analysis guidelines. We collected data concerning the systematic review methods (identification and selection of studies and data collection process) and the meta-analysis methods (effect measures handling and synthesis and analysis methods). Results: Major issues concerning completely reproducible search procedures report, specification of exact methods to compute effect sizes and specification of certain elements of the model used were found. Conclusion and implications: We will use the findings from our meta-review to articulate some recommendations intended to improve the transparency, openness and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.

Keyword(s)

Replicability Meta-analysis Transparency Reproducibility Meta-science

Persistent Identifier

Date of first publication

2021-05-21

Is part of

Research Synthesis & Big Data, 2021, online

Publisher

ZPID (Leibniz Institute for Psychology)

Citation

Lopez-Nicolas, R., Sanchez-Meca, J., Lopez-Lopez, J. A., & Rubio-Aparicio, M. (2021). Reproducibility-related Reporting Practices in Meta-analyses on the Effectiveness of Psychological Interventions. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4823
  • Author(s) / Creator(s)
    Lopez-Nicolas, Ruben
  • Author(s) / Creator(s)
    Sanchez-Meca, Julio
  • Author(s) / Creator(s)
    Lopez-Lopez, Jose Antonio
  • Author(s) / Creator(s)
    Rubio-Aparicio, Maria
  • PsychArchives acquisition timestamp
    2021-05-14T11:36:44Z
  • Made available on
    2021-05-14T11:36:44Z
  • Date of first publication
    2021-05-21
  • Abstract / Description
    Background: The last decade has revealed significant problems in terms of reproducibility in psychological research. The scientific psychology community has detected various errors and questionable research practices in the process of studying psychological phenomena. In this context, different proposals have been put forward in order to improve transparency and reproducibility practices. In the present study, we focus on meta-analytic methodology. Meta-analysis is a powerful and important tool to synthesize the literature about a research topic. Like other kinds of research, meta-analyses must be reproducible to be compliant with the principles of the scientific method. Furthermore, reproducible meta-analyses can be easily updated with new data or reanalysed applying new and more refined analysis techniques. Objetives: We attempted to empirically assess the prevalence of reproducibility-related methodological reporting practices in published meta-analyses of psychological interventions. Our purpose was to identify the key points that could be improved with the aim to provide some recommendations to carry out reproducible meta-analyses. Research question: Are the meta-analytical reports comprehensive enough to allow for potential reproduction attempts? What are the key points where there is a lack of reporting? Method: We conducted a meta-review of meta-analyses of psychological interventions published between 2000 and 2020. We searched PubMed, PsycInfo and Web of Science databases. Then, we selected a random sample of 100 meta-analyses to examine and extract a range of transparency and reproducibility-related indicators using an ad hoc checklist based on existing meta-analysis guidelines. We collected data concerning the systematic review methods (identification and selection of studies and data collection process) and the meta-analysis methods (effect measures handling and synthesis and analysis methods). Results: Major issues concerning completely reproducible search procedures report, specification of exact methods to compute effect sizes and specification of certain elements of the model used were found. Conclusion and implications: We will use the findings from our meta-review to articulate some recommendations intended to improve the transparency, openness and reproducibility-related reporting practices of meta-analyses in clinical psychology and related areas.
    en
  • Publication status
    unknown
    en
  • Review status
    unknown
    en
  • Sponsorship
    This research has been funded with a grant from the Ministerio de Ciencia e Innovación and by FEDER funds (Project nº PID2019-104080GB-I00).
    en
  • Citation
    Lopez-Nicolas, R., Sanchez-Meca, J., Lopez-Lopez, J. A., & Rubio-Aparicio, M. (2021). Reproducibility-related Reporting Practices in Meta-analyses on the Effectiveness of Psychological Interventions. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4823
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/4260
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.4823
  • Language of content
    eng
  • Publisher
    ZPID (Leibniz Institute for Psychology)
    en
  • Is part of
    Research Synthesis & Big Data, 2021, online
    en
  • Keyword(s)
    Replicability
    en
  • Keyword(s)
    Meta-analysis
    en
  • Keyword(s)
    Transparency
    en
  • Keyword(s)
    Reproducibility
    en
  • Keyword(s)
    Meta-science
    en
  • Dewey Decimal Classification number(s)
    150
  • Title
    Reproducibility-related Reporting Practices in Meta-analyses on the Effectiveness of Psychological Interventions
    en
  • DRO type
    conferenceObject
    en
  • Visible tag(s)
    ZPID Conferences and Workshops