Detecting Intervention Effects across Context: An Examination of the Power of Experimental Studies Launched by the Institute of Education Sciences

Jessaca Spybrook

Since 2002 the Institute of Education Sciences (IES) has funded more than 55 experiments to evaluate educational interventions in an effort to generate scientific evidence of program effectiveness on which to base education policy and practice. These studies are typically designed with the goal of having adequate statistical power to detect the average treatment effect. However, the average treatment effect is of limited utility to a practitioner in a particular site if the treatment effects vary substantially from site to site. This project will examine the power of studies to detect differences in intervention effects across contexts or sub-populations. Using data from the IES group randomized trials funded between 2002 and 2006, I will assess the statistical power of the studies to detect 1) variability in treatment effects across sites, typically districts or schools, 2) non-zero site-specific treatment effects, and 3) moderator effects at the setting or student-levels.

About Jessaca Spybrook

Jessaca Spybrook, PhD, is an assistant professor in the Department of Educational Leadership, Research, and Technology in the College of Education at Western Michigan University. Her research has focused on improving the quality of the designs and power analyses of group randomized trials in education. She is currently leading a study funded by the Institute of Education Sciences to examine the methodological changes that occur between the design and implementation of group randomized trials in education. She coauthored the software and documentation for “Optimal Design,” a program that assists researchers in planning adequately powered experiments. She has also published on the topic of improving research designs in journals such as Educational Evaluation and Policy Analysis and the Journal for Research on Educational Effectiveness.