The Reproducibility in Reliability Generalization Meta-Analysis
Author(s) / Creator(s)
López-Ibáñez, Carmen
Sánchez-Meca, Julio
Abstract / Description
Background
One of the principles of scientific method is the ability of an experiment to be reproducible. This means that the results of a study should be the same if the research is repeated by other researchers, following the same procedure. It is essential that the empirical findings be replicable by other researchers. In recent years, the reproducibility of the experimental results has been questioned by many studies. This problem has become increasingly evident, especially regarding to research fields such as psychology.
Objectives
Considering all this information, our aim is to examine how the reproducibility drama affects into our area of study: The Reliability Generalization Meta-Analysis. In order to do it, we want to prove whether the results of RG meta-analysis can be reproduced following the described procedure in each meta-analysis and applying better analytic strategies.
Method
At the current stage where we are, we have collected 145 databases of reliability generalization studies which presented the data of primary studies to repeat the described analysis in the paper, proving so the degree of concordance between those results and ours. Our reproducibility study has focused on the average Alpha Coefficient, its 95% confidence interval and the heterogeneity I2 index. These analyses have been carried out, essentially, with metafor package of R. In some cases, we have also used the same software presented by the authors of the paper.
Results
Through this research we expect to obtain that the results of RG meta-analysis are reproducible. We expect to find that our results and the previous ones are practically identical. Moreover, we expect that much of the reproducibility problems are due to a lack of transparency in specifying the analytical strategy that has been chosen to be used. The results of our reproducibility study are discussed, as well as their implications for improving the reporting practices of reliability generalization meta-analyses.
Keyword(s)
Meta-analysis Reliability Reproducibility RPersistent Identifier
Date of first publication
2021-05-21
Is part of
Research Synthesis & Big Data, 2021, online
Publisher
ZPID (Leibniz Institute for Psychology)
Citation
López-Ibáñez, C., & Sánchez-Meca, J. (2021). The Reproducibility in Reliability Generalization Meta-Analysis. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4834
-
LI_Carmen.pdfAdobe PDF - 9.81MBMD5: 8296a196e75f5ab09c78699a1a47dbd6
-
There are no other versions of this object.
-
Author(s) / Creator(s)López-Ibáñez, Carmen
-
Author(s) / Creator(s)Sánchez-Meca, Julio
-
PsychArchives acquisition timestamp2021-05-14T13:29:45Z
-
Made available on2021-05-14T13:29:45Z
-
Date of first publication2021-05-21
-
Abstract / DescriptionBackground One of the principles of scientific method is the ability of an experiment to be reproducible. This means that the results of a study should be the same if the research is repeated by other researchers, following the same procedure. It is essential that the empirical findings be replicable by other researchers. In recent years, the reproducibility of the experimental results has been questioned by many studies. This problem has become increasingly evident, especially regarding to research fields such as psychology. Objectives Considering all this information, our aim is to examine how the reproducibility drama affects into our area of study: The Reliability Generalization Meta-Analysis. In order to do it, we want to prove whether the results of RG meta-analysis can be reproduced following the described procedure in each meta-analysis and applying better analytic strategies. Method At the current stage where we are, we have collected 145 databases of reliability generalization studies which presented the data of primary studies to repeat the described analysis in the paper, proving so the degree of concordance between those results and ours. Our reproducibility study has focused on the average Alpha Coefficient, its 95% confidence interval and the heterogeneity I2 index. These analyses have been carried out, essentially, with metafor package of R. In some cases, we have also used the same software presented by the authors of the paper. Results Through this research we expect to obtain that the results of RG meta-analysis are reproducible. We expect to find that our results and the previous ones are practically identical. Moreover, we expect that much of the reproducibility problems are due to a lack of transparency in specifying the analytical strategy that has been chosen to be used. The results of our reproducibility study are discussed, as well as their implications for improving the reporting practices of reliability generalization meta-analyses.en
-
Publication statusunknownen
-
Review statusunknownen
-
SponsorshipThis research has been funded with a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (project no PID2019-104080GB-I00).en
-
CitationLópez-Ibáñez, C., & Sánchez-Meca, J. (2021). The Reproducibility in Reliability Generalization Meta-Analysis. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4834en
-
Persistent Identifierhttps://hdl.handle.net/20.500.12034/4271
-
Persistent Identifierhttps://doi.org/10.23668/psycharchives.4834
-
Language of contenteng
-
PublisherZPID (Leibniz Institute for Psychology)en
-
Is part ofResearch Synthesis & Big Data, 2021, onlineen
-
Keyword(s)Meta-analysisen
-
Keyword(s)Reliabilityen
-
Keyword(s)Reproducibilityen
-
Keyword(s)R
-
Dewey Decimal Classification number(s)150
-
TitleThe Reproducibility in Reliability Generalization Meta-Analysisen
-
DRO typeconferenceObjecten
-
Visible tag(s)ZPID Conferences and Workshops