Questions regarding the generalizability of results from educational experiments have been at the forefront of methods development over the past five years. This work has focused on methods for estimating the effect of an intervention in a well-defined inference population (e.g., Tipton, 2013; O'Muircheartaigh and Hedges, 2014); methods for…

Objective structured clinical examinations (OSCEs) are used worldwide for summative examinations but often lack acceptable reliability. Research has shown that reliability of scores increases if OSCE checklists for medical students include only clinically relevant items. Also, checklists are often missing evidence-based items that high-achieving…

While there are appropriate tools to progress monitor academics, there is no universally accepted tool to progress monitor social behavior. The current study proposed the development of a series of brief behavior rating scales to correspond to important social skills domains on the Social Skills Rating Scale (Gresham&Elliott, 2008) and to…

Direct Behavior Rating-Multi-Item Scales (DBR-MIS) have been developed as formative measures of behavioral assessment for use in school-based problem-solving models. Initial research has examined the dependability of composite scores generated by summing all items comprising the scales. However, it has been argued that DBR-MIS may offer assessment…

Two studies were conducted for the development and validation of a multidimensional test to assess undergraduate students' mathematical thinking about derivative. The first study involved two phases: question generation and refinement of the Thinking-about-Derivative Test (TDT). The second study included four phases as follows: test…

Multiple-choice questions are a gold-standard tool in medical school for assessment of knowledge and are the mainstay of licensing examinations. However, multiple-choice questions items can be criticized for lacking the ability to test higher-order learning or integrative thinking across multiple disciplines. Our objective was to develop a novel…

This study used generalizability theory to measure reliability on the Recognizing Effective Special Education Teachers (RESET) observation tool designed to evaluate special education teacher effectiveness. At the time of this study, the RESET tool included three evidence-based instructional practices (direct, explicit instruction; whole-group…

Understanding the effects of class size reduction (CSR) has been an enduring issue in education. For the past 3 decades, Project STAR has stimulated research and policy discussions regarding the effects of CSR on a variety of outcomes. Schanzenbach (2007) reviewed STAR studies and concluded that small classes improved student academic outcomes.…

The trans-contextual model outlines the processes by which autonomous motivation toward activities in a physical education context predicts autonomous motivation toward physical activity outside of school, and beliefs about, intentions toward, and actual engagement in, out-of-school physical activity. In the present article, we clarify the…

E. L. Thorndike contributed significantly to the field of educational and psychological testing as well as more broadly to psychological studies in education. This article follows in his testing legacy. I address the escalating demand, across societal sectors, to measure individual and group competencies. In formulating an approach to measuring…

In their examination of the effectiveness of affect regulation strategies, Webb, Miles, and Sheeran (2012) offered the results of a broad meta-analysis of studies on regulatory interventions. Their analysis provides an alternative to our earlier, more focused meta-analysis of the affect regulation literature (Augustine & Hemenover, 2009).…

The present paper aims to discuss how data from international large-scale assessments (ILSAs) can be utilized and combined, even with other existing data sources, in order to monitor educational outcomes and study the effectiveness of educational systems. We consider different purposes of linking data, namely, extending outcomes measures,…

This study assessed the score reliability of the Drinking Motives Questionnaire-Revised (DMQ-R) via generalizability theory. Participants (n = 367 college students) completed the DMQ-R at three time points. Across subscale scores, persons, persons x occasions, and persons x items interactions accounted for meaningful variance. Findings illustrate…

In adolescence research, the treatment of measurement reliability is often fragmented, and it is not always clear how different reliability coefficients are related. We show that generalizability theory (G-theory) is a comprehensive framework of measurement reliability, encompassing all other reliability methods (e.g., Pearson "r,"…

Reliability of direct observation outcomes ensures the results are consistent, dependable, and trustworthy. Typically, reliability of direct observation measurement approaches is assessed using interobserver agreement (IOA) and the calculation of observer agreement (e.g., percentage of agreement). However, IOA does not address intraobserver…