Conference Object

Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach

Author(s) / Creator(s)

Campos, Diego
Scherer, Ronny
Cheung, Mike

Abstract / Description

International large-scale assessments (ILSAs) provide educationally relevant findings from different populations on a common topic. Researchers can use ILSA data to examine whether an effect size is consistent across the body of data, summarize the relationships among variables, and quantify possible heterogeneity in the data. Common multilevel modeling approaches face computational issues with synthesizing the data from multiple ILSAs or ILSA cycles and cannot quantify explicitly the heterogeneity of effect sizes that are derived from the parameters of a specific model. However, meta-analytic approaches, such as the Split, Analyze, and Meta-analyze (SAM) approach, offer ways to circumvent these issues by bringing together the best features of multilevel modeling and meta-analysis. In this study, we showcase how the SAM approach can be used to synthesize findings from ILSA data, focusing on the Big-Fish-Little-Pond-Effect (BFLPE) in mathematics, that is, the negative contextual effect of classroom mathematics achievement on students’ mathematics selfconcept. We analyzed the data from fourth-grade students across five TIMSS cycles. As part of the SAM approach, we performed multi-group multilevel confirmatory factor analysis (MGMCFA) and two-level MSEM to obtain the BFLPE per country and synthesized the resultant effect sizes via cross-classified random-effects meta-analysis. Our results shed light on the average effect size of the BFLPE, the heterogeneity within and between countries and cycles, and the extent to which country-level variables can explain variation in the BFLPE. Overall, this study illustrates how researchers can use the SAM approach for the analysis and synthesis of research evidence using ILSA data.

Keyword(s)

Meta-Analysis Complex Survey Data Three-level meta-analysis ILSA Multilevel meta-analysis

Persistent Identifier

Date of first publication

2021-05-21

Is part of

Research Synthesis & Big Data, 2021, online

Publisher

ZPID (Leibniz Institute for Psychology)

Citation

Campos, D., Scherer, R., & Cheung, M. (2021). Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4827
  • Meta-Analyzing International Large-Scale Assessment Data Campos, Scherer, Cheung _2021.pdf
    Adobe PDF - 7.58MB
    MD5: 1c9c250b5fbdc8ed280c74e5f180117f
     Download
    Description: Slides Presentation: Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach.
  • Author(s) / Creator(s)
    Campos, Diego
  • Author(s) / Creator(s)
    Scherer, Ronny
  • Author(s) / Creator(s)
    Cheung, Mike
  • PsychArchives acquisition timestamp
    2021-05-14T12:20:14Z
  • Made available on
    2021-05-14T12:20:14Z
  • Date of first publication
    2021-05-21
  • Abstract / Description
    International large-scale assessments (ILSAs) provide educationally relevant findings from different populations on a common topic. Researchers can use ILSA data to examine whether an effect size is consistent across the body of data, summarize the relationships among variables, and quantify possible heterogeneity in the data. Common multilevel modeling approaches face computational issues with synthesizing the data from multiple ILSAs or ILSA cycles and cannot quantify explicitly the heterogeneity of effect sizes that are derived from the parameters of a specific model. However, meta-analytic approaches, such as the Split, Analyze, and Meta-analyze (SAM) approach, offer ways to circumvent these issues by bringing together the best features of multilevel modeling and meta-analysis. In this study, we showcase how the SAM approach can be used to synthesize findings from ILSA data, focusing on the Big-Fish-Little-Pond-Effect (BFLPE) in mathematics, that is, the negative contextual effect of classroom mathematics achievement on students’ mathematics selfconcept. We analyzed the data from fourth-grade students across five TIMSS cycles. As part of the SAM approach, we performed multi-group multilevel confirmatory factor analysis (MGMCFA) and two-level MSEM to obtain the BFLPE per country and synthesized the resultant effect sizes via cross-classified random-effects meta-analysis. Our results shed light on the average effect size of the BFLPE, the heterogeneity within and between countries and cycles, and the extent to which country-level variables can explain variation in the BFLPE. Overall, this study illustrates how researchers can use the SAM approach for the analysis and synthesis of research evidence using ILSA data.
    en
  • Publication status
    unknown
    en
  • Review status
    unknown
    en
  • Citation
    Campos, D., Scherer, R., & Cheung, M. (2021). Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach. ZPID (Leibniz Institute for Psychology). https://doi.org/10.23668/PSYCHARCHIVES.4827
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/4264
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.4827
  • Language of content
    eng
  • Publisher
    ZPID (Leibniz Institute for Psychology)
    en
  • Is part of
    Research Synthesis & Big Data, 2021, online
    en
  • Keyword(s)
    Meta-Analysis
    en
  • Keyword(s)
    Complex Survey Data
    en
  • Keyword(s)
    Three-level meta-analysis
    en
  • Keyword(s)
    ILSA
  • Keyword(s)
    Multilevel meta-analysis
    en
  • Dewey Decimal Classification number(s)
    150
  • Title
    Meta-Analyzing International Large-Scale Assessment Data: An Application of the Split, Analyze, and Meta-Analyze (SAM) Approach
    en
  • DRO type
    conferenceObject
    en
  • Leibniz subject classification
    Psychologie
    de_DE
  • Visible tag(s)
    ZPID Conferences and Workshops