Conference Object

The Reproducibility in Reliability Generalization Meta-Analysis

Author(s) / Creator(s)

López-Ibáñez, Carmen
Sánchez-Meca, Julio

Abstract / Description

Background One of the principles of scientific method is the ability of an experiment to be reproducible. This means that the results of a study should be the same if the research is repeated by other researchers, following the same procedure. It is essential that the empirical findings be replicable by other researchers. In recent years, the reproducibility of the experimental results has been questioned by many studies. This problem has become increasingly evident, especially regarding to research fields such as psychology. Objectives Considering all this information, our aim is to examine how the reproducibility drama affects into our area of study: The Reliability Generalization Meta-Analysis. In order to do it, we want to prove whether the results of RG meta-analysis can be reproduced following the described procedure in each meta-analysis and applying better analytic strategies. Method At the current stage where we are, we have collected 145 databases of reliability generalization studies which presented the data of primary studies to repeat the described analysis in the paper, proving so the degree of concordance between those results and ours. Our reproducibility study has focused on the average Alpha Coefficient, its 95% confidence interval and the heterogeneity I2 index. These analyses have been carried out, essentially, with metafor package of R. In some cases, we have also used the same software presented by the authors of the paper. Results Through this research we expect to obtain that the results of RG meta-analysis are reproducible. We expect to find that our results and the previous ones are practically identical. Moreover, we expect that much of the reproducibility problems are due to a lack of transparency in specifying the analytical strategy that has been chosen to be used. The results of our reproducibility study are discussed, as well as their implications for improving the reporting practices of reliability generalization meta-analyses.

Keyword(s)

Meta-analysis Reliability Reproducibility R

Persistent Identifier

Date of first publication

2021-05-21

Is part of

Research Synthesis & Big Data, 2021, online

Publisher

ZPID (Leibniz Institute for Psychology)

Citation

López-Ibáñez, C., & Sánchez-Meca, J. (2021). The Reproducibility in Reliability Generalization Meta-Analysis. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4834
  • Author(s) / Creator(s)
    López-Ibáñez, Carmen
  • Author(s) / Creator(s)
    Sánchez-Meca, Julio
  • PsychArchives acquisition timestamp
    2021-05-14T13:29:45Z
  • Made available on
    2021-05-14T13:29:45Z
  • Date of first publication
    2021-05-21
  • Abstract / Description
    Background One of the principles of scientific method is the ability of an experiment to be reproducible. This means that the results of a study should be the same if the research is repeated by other researchers, following the same procedure. It is essential that the empirical findings be replicable by other researchers. In recent years, the reproducibility of the experimental results has been questioned by many studies. This problem has become increasingly evident, especially regarding to research fields such as psychology. Objectives Considering all this information, our aim is to examine how the reproducibility drama affects into our area of study: The Reliability Generalization Meta-Analysis. In order to do it, we want to prove whether the results of RG meta-analysis can be reproduced following the described procedure in each meta-analysis and applying better analytic strategies. Method At the current stage where we are, we have collected 145 databases of reliability generalization studies which presented the data of primary studies to repeat the described analysis in the paper, proving so the degree of concordance between those results and ours. Our reproducibility study has focused on the average Alpha Coefficient, its 95% confidence interval and the heterogeneity I2 index. These analyses have been carried out, essentially, with metafor package of R. In some cases, we have also used the same software presented by the authors of the paper. Results Through this research we expect to obtain that the results of RG meta-analysis are reproducible. We expect to find that our results and the previous ones are practically identical. Moreover, we expect that much of the reproducibility problems are due to a lack of transparency in specifying the analytical strategy that has been chosen to be used. The results of our reproducibility study are discussed, as well as their implications for improving the reporting practices of reliability generalization meta-analyses.
    en
  • Publication status
    unknown
    en
  • Review status
    unknown
    en
  • Sponsorship
    This research has been funded with a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (project no PID2019-104080GB-I00).
    en
  • Citation
    López-Ibáñez, C., & Sánchez-Meca, J. (2021). The Reproducibility in Reliability Generalization Meta-Analysis. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4834
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/4271
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.4834
  • Language of content
    eng
  • Publisher
    ZPID (Leibniz Institute for Psychology)
    en
  • Is part of
    Research Synthesis & Big Data, 2021, online
    en
  • Keyword(s)
    Meta-analysis
    en
  • Keyword(s)
    Reliability
    en
  • Keyword(s)
    Reproducibility
    en
  • Keyword(s)
    R
  • Dewey Decimal Classification number(s)
    150
  • Title
    The Reproducibility in Reliability Generalization Meta-Analysis
    en
  • DRO type
    conferenceObject
    en
  • Visible tag(s)
    ZPID Conferences and Workshops