Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach
Author(s) / Creator(s)
Campos, Diego
Scherer, Ronny
Cheung, Mike
Abstract / Description
International large-scale assessments (ILSAs) provide educationally relevant
findings from different populations on a common topic. Researchers can use ILSA data to
examine whether an effect size is consistent across the body of data, summarize the
relationships among variables, and quantify possible heterogeneity in the data. Common
multilevel modeling approaches face computational issues with synthesizing the data from
multiple ILSAs or ILSA cycles and cannot quantify explicitly the heterogeneity of effect sizes
that are derived from the parameters of a specific model. However, meta-analytic approaches,
such as the Split, Analyze, and Meta-analyze (SAM) approach, offer ways to circumvent these
issues by bringing together the best features of multilevel modeling and meta-analysis. In this
study, we showcase how the SAM approach can be used to synthesize findings from ILSA
data, focusing on the Big-Fish-Little-Pond-Effect (BFLPE) in mathematics, that is, the negative
contextual effect of classroom mathematics achievement on students’ mathematics selfconcept.
We analyzed the data from fourth-grade students across five TIMSS cycles. As part
of the SAM approach, we performed multi-group multilevel confirmatory factor analysis (MGMCFA) and two-level MSEM to obtain the BFLPE per country and synthesized the resultant
effect sizes via cross-classified random-effects meta-analysis. Our results shed light on the
average effect size of the BFLPE, the heterogeneity within and between countries and cycles,
and the extent to which country-level variables can explain variation in the BFLPE. Overall,
this study illustrates how researchers can use the SAM approach for the analysis and synthesis
of research evidence using ILSA data.
Keyword(s)
Meta-Analysis Complex Survey Data Three-level meta-analysis ILSA Multilevel meta-analysisPersistent Identifier
Date of first publication
2021-05-21
Is part of
Research Synthesis & Big Data, 2021, online
Publisher
ZPID (Leibniz Institute for Psychology)
Citation
Campos, D., Scherer, R., & Cheung, M. (2021). Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4827
-
Meta-Analyzing International Large-Scale Assessment Data Campos, Scherer, Cheung _2021.pdfAdobe PDF - 7.58MBMD5: 1c9c250b5fbdc8ed280c74e5f180117fDescription: Slides Presentation: Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach.
-
There are no other versions of this object.
-
Author(s) / Creator(s)Campos, Diego
-
Author(s) / Creator(s)Scherer, Ronny
-
Author(s) / Creator(s)Cheung, Mike
-
PsychArchives acquisition timestamp2021-05-14T12:20:14Z
-
Made available on2021-05-14T12:20:14Z
-
Date of first publication2021-05-21
-
Abstract / DescriptionInternational large-scale assessments (ILSAs) provide educationally relevant findings from different populations on a common topic. Researchers can use ILSA data to examine whether an effect size is consistent across the body of data, summarize the relationships among variables, and quantify possible heterogeneity in the data. Common multilevel modeling approaches face computational issues with synthesizing the data from multiple ILSAs or ILSA cycles and cannot quantify explicitly the heterogeneity of effect sizes that are derived from the parameters of a specific model. However, meta-analytic approaches, such as the Split, Analyze, and Meta-analyze (SAM) approach, offer ways to circumvent these issues by bringing together the best features of multilevel modeling and meta-analysis. In this study, we showcase how the SAM approach can be used to synthesize findings from ILSA data, focusing on the Big-Fish-Little-Pond-Effect (BFLPE) in mathematics, that is, the negative contextual effect of classroom mathematics achievement on students’ mathematics selfconcept. We analyzed the data from fourth-grade students across five TIMSS cycles. As part of the SAM approach, we performed multi-group multilevel confirmatory factor analysis (MGMCFA) and two-level MSEM to obtain the BFLPE per country and synthesized the resultant effect sizes via cross-classified random-effects meta-analysis. Our results shed light on the average effect size of the BFLPE, the heterogeneity within and between countries and cycles, and the extent to which country-level variables can explain variation in the BFLPE. Overall, this study illustrates how researchers can use the SAM approach for the analysis and synthesis of research evidence using ILSA data.en
-
Publication statusunknownen
-
Review statusunknownen
-
CitationCampos, D., Scherer, R., & Cheung, M. (2021). Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4827en
-
Persistent Identifierhttps://hdl.handle.net/20.500.12034/4264
-
Persistent Identifierhttps://doi.org/10.23668/psycharchives.4827
-
Language of contenteng
-
PublisherZPID (Leibniz Institute for Psychology)en
-
Is part ofResearch Synthesis & Big Data, 2021, onlineen
-
Keyword(s)Meta-Analysisen
-
Keyword(s)Complex Survey Dataen
-
Keyword(s)Three-level meta-analysisen
-
Keyword(s)ILSA
-
Keyword(s)Multilevel meta-analysisen
-
Dewey Decimal Classification number(s)150
-
TitleMeta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approachen
-
DRO typeconferenceObjecten
-
Leibniz subject classificationPsychologiede_DE
-
Visible tag(s)ZPID Conferences and Workshops