Causal Mediation Analysis in Multi-Site Trials

Xu Qin

There has been an increasing use of multisite randomized trials in evaluations of educational programs. Multisite designs provide unique opportunities for investigating between-site heterogeneity in the mediation mechanism that characterizes an educational process central to a program theory. However, such opportunities have not been fully utilized due to some important constraints of existing analytic tools. To enable researchers to assess the generalizability of an education program theory across a wide range of contexts, I develop a new analytic procedure that enhances both the external validity and internal validity of multisite causal mediation analysis. This includes incorporating a sample weight to adjust for sample and survey designs, employing a non-response weight to account for non-random attrition, and utilizing a novel propensity score-based weighting strategy to flexibly decompose the average program impact at each site into indirect effects transmitted through hypothesized focal mediators and a direct effect attributable to all the other pathways. Extending a theoretical model of causal inference, I define, identify, and estimate the population average and the between-site variance of the direct and indirect effects. I also develop a sensitivity analysis strategy to assess the consequences of potential violations of key identification assumptions. I employ the proposed methods in an in-depth evaluation of Job Corps, the nation’s largest education and training program for 16-24 year old disadvantaged youths, most of whom have dropped out of high school. I test the Job Corps program theory that emphasizes academic education and vocational training along with a wide array of services for risk reduction. My preliminary results suggest that Job Corps centers may not be equally effective in service provision. Such evidence is crucial for improving program implementation. I am also developing an open-source R package, MultisiteMediation (http://cran.r-project.org/web/packages/MultisiteMediation), offering applied researchers a convenient implementation tool.

About Xu Qin

Xu Qin is a doctoral candidate in the Department of Comparative Human Development at the University of Chicago. She holds a B.S. and an M.S. in Statistics from the Renmin University of China. Supported by a Fellowship for Quantitative Methods in Education and Human Development, her doctoral training has been highly interdisciplinary with a strong focus on theories and methods in education research. Her research focuses on improving statistical methods for educational program evaluations. Specifically, she is working on cutting-edge methodological problems in multisite causal mediation analysis, with an attempt to quantify between-site heterogeneity in program mechanisms. Her work has won a 2016 Joint Statistical Meetings Student Paper Award granted by the American Statistical Association.