Transparency and Reporting Practices in Reliability Generalization Meta-analyses: Assessment with the REGEMA checklist
Author(s) / Creator(s)
Sánchez-Meca, Julio
Marín-Martínez, Fulgencio
Núñez-Núñez, Rosa María
Rubio-Aparicio, María
López-López, José Antonio
Blázquez-Rincón, Desirée María
López-Ibáñez, Carmen
López-Nicolás, Rubén
López-Pina, José Antonio
López-García, Juan José
Abstract / Description
1. Background
A reliability generalization (RG) meta-analysis is a special type of psychometric meta-analysis that aims to integrate reliability coefficients obtained when a given test is applied to different primary studies, to examine how reliability of a test scores varies from an application to the next. Its purpose is to estimate the average reliability of a test scores, to investigate whether reliability can be generalized to different contexts, situations, and target populations, and in case of heterogeneity, to identify study characteristics that might be statistically associated to reliability coefficients (e.g., mean and SD of tests cores, target population, test version, etc.).
Current checklists to guide the reporting of meta-analyses are not adequate for RG meta-analyses:
PRISMA checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Moher et al., 2009)
AMSTAR checklist for reporting meta-analyses on intervention efficacy (Grimshaw, Wells, et al., 2007)
MOOSE checklist for reporting meta-analyses of observational studies (Stroup, Berlin, Morton et al., 2000)
MARS guidelines for reporting meta-analyses on intervention efficacy (APA Publications and Communications Board Working Group on Journal Article Reporting Standards, 2008)
Due to the absence in the literature of guidelines specifically devised to inform researchers and consumers of RG meta-analyses on how to conduct and report an RG meta-analysis, we developed the REGEMA checklist, a tool to guide researchers to report RG meta-analyses fulfilling the transparency and reproducibility principles of the Open Science.
1.1 Objectives
The main objectives of this research were: (a) To examine the inter-coder reliability of the REGEMA checklist; (b) to investigate the capacity of REGEMA checklist to assess the reporting quality of RG meta-analyses, and (c) to assess the degree of compliance of REGEMA checklist by RG meta-analyses.
2. Method
2.1 Selection criteria of the studies
To be included in this systematic review the meta-analyses had to fulfil the following selection criteria: (a) RG meta-analysis reporting reliability estimates of one or several tests to assess psychological constructs; (b) it had to be carried out between 1998 and 2019; (c) it had to include all primary studies that applied the test/s of interest (meta-analyses of only psychometric studies were excluded); (d) in case of focusing on more than one test, it had to report separate reliability estimates of each one of them, and (e) to be written in English or Spanish.
2.2 Searching for the studies
The following databases were consulted: PsycINFO, Web of Science, and Google Scholar, using the key words in the title: ‘reliability generalization’, ‘meta-analysis AND reliability’, ‘meta-analysis AND internal consistency’, ‘meta-analysis AND test-retest’, ‘meta-analysis AND (interrater OR intrarater)’, ‘meta-analysis AND (alpha coefficient OR coefficient alpha)’, ‘meta-analysis AND intraclass’.
A total of 150 RG meta-analyses (19 unpublished and 131 published RG meta-analyses) were identified and included in this systematic review.
2.3 Data extraction
REGEMA checklist was applied to each RG meta-analysis. Other characteristics registered were the publication source, the year of the meta-analysis, and the idiom. Inter-coder reliability was assessed with two coders that independently extracted the data from the RG meta-analyses and applied the REGEMA checklist. Inconsistencies were resolved by consensus.
2.4 Statistical analysis
To estimate the inter-coder reliability of REGEMA checklist, Cohen’s kappa coefficient and inter-coder agreement percentage were calculated for each item and subitem of the checklist. To assess the degree of compliance of each item of the REGEMA checklist, the compliance percentage from all the RG meta-analyses analyzed was calculated.
3. Results
REGEMA checklist exhibited a very satisfactory degree of inter-coder reliability. The degree of compliance of REGEMA by RG meta-analyses is deficient in many of their items (< 50% of compliance), especially in: (a) Background: defining the psychological construct assessed; (b) determining the selection criteria of the studies; (c) the reliability analysis of data extraction data from the studies, and (d) how heterogeneity among reliability coefficients was assessed.
4. Conclusions and implications
REGEMA checklist is easy to be applied and enables to assess the reporting quality of RG meta-analyses. REGEMA checklist can be useful for: (a) Researchers interested in conducting an RG meta-analysis; (b) Potential readers of RG meta-analyses, as a tool to make a critical reading of them, and (c) Editors of scientific journals that publish RG meta-analyses, as a guide for an adequate reporting of this type of meta-analyses.
* Funding
This research was funded by a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (Project nº PID2019-104080GB-I00).
Persistent Identifier
Date of first publication
2021-05-21
Is part of
Research Synthesis & Big Data, 2021, online
Publisher
ZPID (Leibniz Institute for Psychology)
Citation
Sánchez-Meca, J., Marín-Martínez, F., Núñez-Núñez, R. M., Rubio-Aparicio, M., López-López, J. A., Blázquez-Rincón, D. M., López-Ibáñez, C., López-Nicolás, R., López-Pina, J. A., & López-García, J. J. (2021). Transparency and Reporting Practices in Reliability Generalization Meta-analyses: Assessment with the REGEMA checklist. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4813
-
Transparency and Reporting Practices in Reliability Generalization Meta-analyses.pdfAdobe PDF - 953.13KBMD5: 9590cc53d1efbe7043a4a5c8b3f67d9e
-
There are no other versions of this object.
-
Author(s) / Creator(s)Sánchez-Meca, Julio
-
Author(s) / Creator(s)Marín-Martínez, Fulgencio
-
Author(s) / Creator(s)Núñez-Núñez, Rosa María
-
Author(s) / Creator(s)Rubio-Aparicio, María
-
Author(s) / Creator(s)López-López, José Antonio
-
Author(s) / Creator(s)Blázquez-Rincón, Desirée María
-
Author(s) / Creator(s)López-Ibáñez, Carmen
-
Author(s) / Creator(s)López-Nicolás, Rubén
-
Author(s) / Creator(s)López-Pina, José Antonio
-
Author(s) / Creator(s)López-García, Juan José
-
PsychArchives acquisition timestamp2021-05-11T11:06:54Z
-
Made available on2021-05-11T11:06:54Z
-
Date of first publication2021-05-21
-
Abstract / Description1. Background A reliability generalization (RG) meta-analysis is a special type of psychometric meta-analysis that aims to integrate reliability coefficients obtained when a given test is applied to different primary studies, to examine how reliability of a test scores varies from an application to the next. Its purpose is to estimate the average reliability of a test scores, to investigate whether reliability can be generalized to different contexts, situations, and target populations, and in case of heterogeneity, to identify study characteristics that might be statistically associated to reliability coefficients (e.g., mean and SD of tests cores, target population, test version, etc.). Current checklists to guide the reporting of meta-analyses are not adequate for RG meta-analyses: PRISMA checklist (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Moher et al., 2009) AMSTAR checklist for reporting meta-analyses on intervention efficacy (Grimshaw, Wells, et al., 2007) MOOSE checklist for reporting meta-analyses of observational studies (Stroup, Berlin, Morton et al., 2000) MARS guidelines for reporting meta-analyses on intervention efficacy (APA Publications and Communications Board Working Group on Journal Article Reporting Standards, 2008) Due to the absence in the literature of guidelines specifically devised to inform researchers and consumers of RG meta-analyses on how to conduct and report an RG meta-analysis, we developed the REGEMA checklist, a tool to guide researchers to report RG meta-analyses fulfilling the transparency and reproducibility principles of the Open Science. 1.1 Objectives The main objectives of this research were: (a) To examine the inter-coder reliability of the REGEMA checklist; (b) to investigate the capacity of REGEMA checklist to assess the reporting quality of RG meta-analyses, and (c) to assess the degree of compliance of REGEMA checklist by RG meta-analyses. 2. Method 2.1 Selection criteria of the studies To be included in this systematic review the meta-analyses had to fulfil the following selection criteria: (a) RG meta-analysis reporting reliability estimates of one or several tests to assess psychological constructs; (b) it had to be carried out between 1998 and 2019; (c) it had to include all primary studies that applied the test/s of interest (meta-analyses of only psychometric studies were excluded); (d) in case of focusing on more than one test, it had to report separate reliability estimates of each one of them, and (e) to be written in English or Spanish. 2.2 Searching for the studies The following databases were consulted: PsycINFO, Web of Science, and Google Scholar, using the key words in the title: ‘reliability generalization’, ‘meta-analysis AND reliability’, ‘meta-analysis AND internal consistency’, ‘meta-analysis AND test-retest’, ‘meta-analysis AND (interrater OR intrarater)’, ‘meta-analysis AND (alpha coefficient OR coefficient alpha)’, ‘meta-analysis AND intraclass’. A total of 150 RG meta-analyses (19 unpublished and 131 published RG meta-analyses) were identified and included in this systematic review. 2.3 Data extraction REGEMA checklist was applied to each RG meta-analysis. Other characteristics registered were the publication source, the year of the meta-analysis, and the idiom. Inter-coder reliability was assessed with two coders that independently extracted the data from the RG meta-analyses and applied the REGEMA checklist. Inconsistencies were resolved by consensus. 2.4 Statistical analysis To estimate the inter-coder reliability of REGEMA checklist, Cohen’s kappa coefficient and inter-coder agreement percentage were calculated for each item and subitem of the checklist. To assess the degree of compliance of each item of the REGEMA checklist, the compliance percentage from all the RG meta-analyses analyzed was calculated. 3. Results REGEMA checklist exhibited a very satisfactory degree of inter-coder reliability. The degree of compliance of REGEMA by RG meta-analyses is deficient in many of their items (< 50% of compliance), especially in: (a) Background: defining the psychological construct assessed; (b) determining the selection criteria of the studies; (c) the reliability analysis of data extraction data from the studies, and (d) how heterogeneity among reliability coefficients was assessed. 4. Conclusions and implications REGEMA checklist is easy to be applied and enables to assess the reporting quality of RG meta-analyses. REGEMA checklist can be useful for: (a) Researchers interested in conducting an RG meta-analysis; (b) Potential readers of RG meta-analyses, as a tool to make a critical reading of them, and (c) Editors of scientific journals that publish RG meta-analyses, as a guide for an adequate reporting of this type of meta-analyses. * Funding This research was funded by a grant from the Ministerio de Ciencia e Innovación of the Spanish Government and by FEDER funds (Project nº PID2019-104080GB-I00).en
-
Publication statusunknownen
-
Review statusunknownen
-
CitationSánchez-Meca, J., Marín-Martínez, F., Núñez-Núñez, R. M., Rubio-Aparicio, M., López-López, J. A., Blázquez-Rincón, D. M., López-Ibáñez, C., López-Nicolás, R., López-Pina, J. A., & López-García, J. J. (2021). Transparency and Reporting Practices in Reliability Generalization Meta-analyses: Assessment with the REGEMA checklist. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4813en
-
Persistent Identifierhttps://hdl.handle.net/20.500.12034/4250
-
Persistent Identifierhttps://doi.org/10.23668/psycharchives.4813
-
Language of contenteng
-
PublisherZPID (Leibniz Institute for Psychology)en
-
Is part ofResearch Synthesis & Big Data, 2021, onlineen
-
Dewey Decimal Classification number(s)150
-
TitleTransparency and Reporting Practices in Reliability Generalization Meta-analyses: Assessment with the REGEMA checklisten
-
DRO typeconferenceObjecten_US
-
Visible tag(s)ZPID Conferences and Workshops