The purpose of this study is to develop a nonparametric DIF method that (a) compares focal groups directly to the composite group that will be used to develop the reported test score scale, and (b) allows practitioners to explore for DIF related to focal groups stemming from multicategorical variables that constitute a small proportion of the…

In randomized control trials (RCT), the recent focus has shifted to how an intervention yields positive results on its intended outcome. This aligns with the recent push of implementation science in healthcare (Bauer et al., 2015) but goes beyond this. RCTs have moved to evaluating the theoretical framing of the intervention as well as differing…

Traditional pre-test (TpT)/post-test (PT) and retrospective pre-test (RpT)/post-test (PT) designs are used to collect data on self-reported measures to assess the magnitude of change that occurs from interventions. If measurement invariance does not exist across the measurement occasions within these research designs, it is inappropriate to…

Little is known empirically about intraclass correlations (ICCs) for multisite cluster randomized trial (MSCRT) designs, particularly in science education. In this study, ICCs suitable for science achievement studies using a three-level (students in schools in districts) MSCRT design that block on district are estimated and examined. Estimates of…

Some form of a short interrupted time series (ITS) is often used to evaluate state and national programs. An ITS design with a single treatment group assumes that the pretest functional form can be validly estimated and extrapolated into the postintervention period where it provides a valid counterfactual. This assumption is problematic. Ambiguous…

Scientifically based research used to inform evidence based school reform efforts has been required by the federal government in order to receive grant funding since the reenactment of No Child Left Behind (2002). Educational evaluators are thus faced with the challenge to use rigorous research designs to establish causal relationships. However,…

A previous Monte Carlo study examined the relative powers of several simple and more complex procedures for testing the significance of difference in mean rates of change in a controlled, longitudinal, treatment evaluation study. Results revealed that the relative powers depended on the correlation structure of the simulated repeated measurements.…

The purpose of this Guide is to advise researchers, policymakers, and others on when it is possible to conduct a high-quality randomized controlled trial in education at reduced cost. Well-designed randomized controlled trials are recognized as the gold standard for evaluating the effectiveness of an intervention (i.e., program or practice) in…

The equipercentile assumption states that students in traditional classrooms who receive no other instructional assistance, will maintain their relative rank order over time. To test this assumption, fall to fall test results on the SRA Achievement Tests were obtained for grades 2-3, and 6-7. Total reading and total mathematics growth scale values…

Twelve common errors which can invalidate an otherwise sound evaluation are identified, and ways to avoid them are presented. The hazards are: (1) grade-equivalent scores; (2) inappropriate statistical adjustments with nonequivalent control groups; (3) administering norm-referenced tests at inappropriate times of the school year; (4) inappropriate…

The norm-referenced evaluation model (RMC Model A) for Title I project evaluation, consists of procedures whereby the expected posttest standing of a treatment group under the null condition is generated from their pretest standing. It is assumed that the treatment group is not selected on the basis of their pretest scores and can be considered…

Title I evaluations using the RMC Model C design depend for their interpretation on the assumption that the regression of posttest on pretest is linear across the cut score level when there is no treatment; but there are many instances where nonlinearities may occur. If one applies the analysis of covariance, or model C analysis, large errors may…

A discussion of second language program evaluation focuses on the interpretability of test scores as a criterion in program evaluation. It looks at both test design and research design issues. First, eight method-comparison, program evaluation studies that compare acquisition-based and analysis/practice based methods are described. Acquisition…