Conference Object

Moving from What Works to What Replicates: A New Framework for Evidence Based Policy Analysis

Author(s) / Creator(s)

Wong, Vivian C.
Steiner, Peter M.

Abstract / Description

Background: Efforts to promote evidence-based practices in medicine and the social sciences (e.g., What Works Clearinghouse) assume that scientific findings are of sufficient validity to warrant their use in decision making. Replication has long been a cornerstone for establishing trustworthy scientific results. At its core is the belief that scientific knowledge should not be based on chance occurrences. Rather, it is established through systematic and transparent methods, results that can be independently replicated, and findings that are generalizable to at least some target population of interest (Bollen, Cacioppo, Kaplan, Krosnick, & Olds, 2015). However, despite consensus on the need to promote replication, there remains considerable disagreement about what constitutes as replication, how a replication study should be implemented, how results from these studies should be interpreted, and whether direct replication of results is even possible (Hansen, 2011). Objectives: This paper seeks to address these concerns by developing the methodological foundations for a “replication science.” The paper introduces the Causal Replication Framework, which defines “replication” as a research design that tests whether two (or more) studies produce the same causal effect within the limits of sampling error. Using potential outcomes notation, the framework provides a clear definition of replication and highlights the conditions under which results are likely to replicate. The paper also demonstrates how different research designs may be used to evaluate the replication of results and identify sources of effect heterogeneity. Research question(s): To this end, the paper will address three research questions: (1) Under the Causal Replication Framework, what is replication? (2) What research design assumptions are required for successful replication of results? (3) How may research design features be used to address replication assumptions in high quality, systematic replication studies? Method/Approach: Over the years, researchers have sought to clarify what is meant by a replication. Most prior definitions have focused on repeating methods and procedures in conducting a replication study (Schmidt, 2009). Schmidt also describes direct replication as a “methodological tool based on a repetition procedure,” but adds that its purpose is for “establishing a fact, truth or piece of knowledge” (2009, p. 91, emphasis in original). In the Causal Replication Framework, we begin with the premise that replication is for establishing a “fact, truth or piece of knowledge.” Here, the “piece of knowledge” can be described as the causal effect of a well-defined treatment-control contrast on a clearly specified outcome for a well-defined target population. We refer to this unknown causal effect as the causal estimand, which is the target of inference in the original and replication study. Using a potential outcomes framework, we show that five research design assumptions are required for the direct replication of results: A1 Treatment & Outcome Stability A1.1 No hidden variation in treatment and control conditions. A1.2 No variation in outcome measures. A1.3 No mode-of-study selection effects. A1.4 No peer, spillover, or carryover effects. A2 Equivalence of Causal Estimands A2.1 Same causal quantity of interest. A2.2 Identical effect-generating processes. A2.3 Identical distribution of population characteristics. A2.4 Identical distribution of setting variables. A3 Identification of Causal Estimands. In both studies, the causal estimand must be identified using an experimental or quasi-experimental research design. A4 Unbiased Estimation of Causal Estimands. In both studies, the causal estimand (ATE) is estimable without bias. A5 Correct Reporting of Estimands, Estimators, and Estimates. For both studies, the estimands, estimators, and estimates are correctly reported. The replication assumptions highlight the difference between traditional, procedure-based approaches to replication and the Causal Replication Framework. In procedure-based approaches, the goal and purpose of replication is repetition of methods. In the Causal Replication Framework, the goal is for two studies to identify and estimate the same well-defined causal estimand of interest. This means that while two studies may use the same procedures and methods to generate the same corresponding causal effect, it is also possible for two studies to use different methods and procedures as long as they identify and estimate the same well-defined causal estimand of interest. Results/Findings: Conceptualizing replication through the Causal Replication Framework yields two important implications for practice in the planning of replication studies. First, although assumptions for the direct replication of results are stringent, it is possible for researchers to address and probe these assumptions through the thoughtful use of research designs and diagnostic tests. A second implication of the framework is that research designs may be used to identify potential sources of effect heterogeneity by systematically violating one or multiple assumptions (while meeting all other assumptions). Here, a prospective design may be applied to address all other replication design assumptions with the exception of the one that is under investigation. If results are found to not replicate, the researcher will know why there was a difference in effects. The paper highlights real world examples of how research design approaches and empirical diagnostics may be used to conduct high quality replication studies. The first example will come from a series of prospectively planned, highly controlled replication studies in the context of a teacher preparation program; the second is a post-hoc replication of a random assignment field trial offering full- and half-day preschool. Data from these two sets of replication studies will be used to demonstrate various research design approaches for replication, the feasibility of proposed methods on real world applications of replication, and how researchers may address plausible threats to replication design assumptions when they do occur in field settings. Conclusions and implications (expected): The paper demonstrates that research designs for replication may be used to systematically evaluate the replicability of effects and identify sources of effect heterogeneity. The paper concludes with a discussion of methodological tools that are needed to conduct high quality replication studies in field settings. References: Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committy to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences. Hansen, W. B. (2011). Was Herodotus Correct? Prevention Science, 12(2), 118–120. https://doi.org/10.1007/s11121-011-0218-5

Persistent Identifier

Date of first publication

2019-03-13

Is part of

Open Science 2019, Trier, Germany

Publisher

ZPID (Leibniz Institute for Psychology Information)

Citation

Wong, V. C., & Steiner, P. M. (2019, March 13). Moving from What Works to What Replicates: A New Framework for Evidence Based Policy Analysis. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2393
  • Author(s) / Creator(s)
    Wong, Vivian C.
  • Author(s) / Creator(s)
    Steiner, Peter M.
  • PsychArchives acquisition timestamp
    2019-04-01T15:29:36Z
  • Made available on
    2019-04-01T15:29:36Z
  • Date of first publication
    2019-03-13
  • Abstract / Description
    Background: Efforts to promote evidence-based practices in medicine and the social sciences (e.g., What Works Clearinghouse) assume that scientific findings are of sufficient validity to warrant their use in decision making. Replication has long been a cornerstone for establishing trustworthy scientific results. At its core is the belief that scientific knowledge should not be based on chance occurrences. Rather, it is established through systematic and transparent methods, results that can be independently replicated, and findings that are generalizable to at least some target population of interest (Bollen, Cacioppo, Kaplan, Krosnick, & Olds, 2015). However, despite consensus on the need to promote replication, there remains considerable disagreement about what constitutes as replication, how a replication study should be implemented, how results from these studies should be interpreted, and whether direct replication of results is even possible (Hansen, 2011). Objectives: This paper seeks to address these concerns by developing the methodological foundations for a “replication science.” The paper introduces the Causal Replication Framework, which defines “replication” as a research design that tests whether two (or more) studies produce the same causal effect within the limits of sampling error. Using potential outcomes notation, the framework provides a clear definition of replication and highlights the conditions under which results are likely to replicate. The paper also demonstrates how different research designs may be used to evaluate the replication of results and identify sources of effect heterogeneity. Research question(s): To this end, the paper will address three research questions: (1) Under the Causal Replication Framework, what is replication? (2) What research design assumptions are required for successful replication of results? (3) How may research design features be used to address replication assumptions in high quality, systematic replication studies? Method/Approach: Over the years, researchers have sought to clarify what is meant by a replication. Most prior definitions have focused on repeating methods and procedures in conducting a replication study (Schmidt, 2009). Schmidt also describes direct replication as a “methodological tool based on a repetition procedure,” but adds that its purpose is for “establishing a fact, truth or piece of knowledge” (2009, p. 91, emphasis in original). In the Causal Replication Framework, we begin with the premise that replication is for establishing a “fact, truth or piece of knowledge.” Here, the “piece of knowledge” can be described as the causal effect of a well-defined treatment-control contrast on a clearly specified outcome for a well-defined target population. We refer to this unknown causal effect as the causal estimand, which is the target of inference in the original and replication study. Using a potential outcomes framework, we show that five research design assumptions are required for the direct replication of results: A1 Treatment & Outcome Stability A1.1 No hidden variation in treatment and control conditions. A1.2 No variation in outcome measures. A1.3 No mode-of-study selection effects. A1.4 No peer, spillover, or carryover effects. A2 Equivalence of Causal Estimands A2.1 Same causal quantity of interest. A2.2 Identical effect-generating processes. A2.3 Identical distribution of population characteristics. A2.4 Identical distribution of setting variables. A3 Identification of Causal Estimands. In both studies, the causal estimand must be identified using an experimental or quasi-experimental research design. A4 Unbiased Estimation of Causal Estimands. In both studies, the causal estimand (ATE) is estimable without bias. A5 Correct Reporting of Estimands, Estimators, and Estimates. For both studies, the estimands, estimators, and estimates are correctly reported. The replication assumptions highlight the difference between traditional, procedure-based approaches to replication and the Causal Replication Framework. In procedure-based approaches, the goal and purpose of replication is repetition of methods. In the Causal Replication Framework, the goal is for two studies to identify and estimate the same well-defined causal estimand of interest. This means that while two studies may use the same procedures and methods to generate the same corresponding causal effect, it is also possible for two studies to use different methods and procedures as long as they identify and estimate the same well-defined causal estimand of interest. Results/Findings: Conceptualizing replication through the Causal Replication Framework yields two important implications for practice in the planning of replication studies. First, although assumptions for the direct replication of results are stringent, it is possible for researchers to address and probe these assumptions through the thoughtful use of research designs and diagnostic tests. A second implication of the framework is that research designs may be used to identify potential sources of effect heterogeneity by systematically violating one or multiple assumptions (while meeting all other assumptions). Here, a prospective design may be applied to address all other replication design assumptions with the exception of the one that is under investigation. If results are found to not replicate, the researcher will know why there was a difference in effects. The paper highlights real world examples of how research design approaches and empirical diagnostics may be used to conduct high quality replication studies. The first example will come from a series of prospectively planned, highly controlled replication studies in the context of a teacher preparation program; the second is a post-hoc replication of a random assignment field trial offering full- and half-day preschool. Data from these two sets of replication studies will be used to demonstrate various research design approaches for replication, the feasibility of proposed methods on real world applications of replication, and how researchers may address plausible threats to replication design assumptions when they do occur in field settings. Conclusions and implications (expected): The paper demonstrates that research designs for replication may be used to systematically evaluate the replicability of effects and identify sources of effect heterogeneity. The paper concludes with a discussion of methodological tools that are needed to conduct high quality replication studies in field settings. References: Bollen, K., Cacioppo, J., Kaplan, R., Krosnick, J. A., & Olds, J. L. (2015). Social, Behavioral, and Economic Sciences Perspectives on Robust and Reliable Science. Report of the Subcommittee on Replicability in Science Advisory Committy to the National Science Foundation Directorate for Social, Behavioral, and Economic Sciences. Hansen, W. B. (2011). Was Herodotus Correct? Prevention Science, 12(2), 118–120. https://doi.org/10.1007/s11121-011-0218-5
    en_US
  • Sponsorship
    Supported by NSF grant #2015‐0285‐00
    en_US
  • Citation
    Wong, V. C., & Steiner, P. M. (2019, March 13). Moving from What Works to What Replicates: A New Framework for Evidence Based Policy Analysis. ZPID (Leibniz Institute for Psychology Information). https://doi.org/10.23668/psycharchives.2393
    en
  • Persistent Identifier
    https://hdl.handle.net/20.500.12034/2025
  • Persistent Identifier
    https://doi.org/10.23668/psycharchives.2393
  • Language of content
    eng
    en_US
  • Publisher
    ZPID (Leibniz Institute for Psychology Information)
    en_US
  • Is part of
    Open Science 2019, Trier, Germany
    en_US
  • Dewey Decimal Classification number(s)
    150
  • Title
    Moving from What Works to What Replicates: A New Framework for Evidence Based Policy Analysis
    en_US
  • DRO type
    conferenceObject
    en_US
  • Visible tag(s)
    ZPID Conferences and Workshops