Evaluating Educational Programs with a New Item Response Theory Perspective

Sun-Joo Cho

There has been an increasing number of educational interventions in an effort to generate scientific evidence of program effectiveness on which to base education policy and practice. Because educational interventions often take place in school settings, it is common for the level of assignment to treatment to be at the level of the classroom or school, although it sometimes occurs at the student level as well. It is also common that student binary outcome variables are used to assess differences between multiple groups. The purpose of analyzing intervention data is to compare scores between an experimental group(s) and a control group(s). A widespread analysis method that is used to detect group differences on outcomes is hierarchical linear models (HLMs) based on total scores. However, the HLM approach with the total scores neglects crucial aspects of measurement properties because the measurement model is conducted prior to and separately from the model used to detect the intervention effect. This study will show how intervention data having hierarchical data structure and multiple domains can be analyzed with newly developed item response theory (IRT) models to simultaneously take measurement errors of scores into account and to investigate item characteristics. In particular, this project will focus on (1) developing IRT models to overcome the limitations of conventional HLMs based on the total scores and (2) evaluating group differences by comparing alternative models, HLM and IRT. Results from this study would provide evidence of the measurement properties achieved with IRT, especially in detecting group differences on constructs. In addition, results from a simulation study will provide researchers with guidelines on which method should be used in a variety of circumstances.

About Sun-Joo Cho

Sun-Joo Cho is an Assistant Professor at Peabody College of Vanderbilt University (2009-present). She teaches courses on item response theory and psychometric theory at Vanderbilt University. Her Ph.D. is from the University of Georgia’s School of Education, with an emphasis on educational measurement. She was a post-doctoral scholar for 2 years at University of California, Berkeley and worked on assessment design, psychometric models, and estimation methods. Her quantitative program of research involves modeling of individual differences within complex data structures using generalized latent variable models. Specifically, she concentrates on the development and application of item response theory (IRT) models and their estimation methods. She has dealt with data complexity stemming from (1) multiple manifest person categories such as a control group versus an experimental group in an experimental design, (2) multiple latent person categories (or mixtures or latent classes) such as a mastery group versus a non-mastery group in a cognitive test, (3) multiple manifest item groups that may lead to multidimensionality such as number operation, measurement, and representation item groups in a math test, (4) multiple manifest person groups such as schools where students are nested in a multilevel (or hierarchical) data structure, and (5) multiple time points such as pretest and posttest in intervention studies.