The usefulness of evaluation is critical. Evaluation use occurs when, from its results or process, decisions are made about the program, it changes people's mindsets, or persuasive or legitimation actions happen (instrumental, conceptual, and symbolic uses respectively). Few quantitative evaluation use studies have been conducted in recent years.…

Classical evaluation methods, assessments, exams, and so forth accentuate the perception of one against all, professor versus learners. Including students in the assessment process, allows transforming the professor from an opponent to a critical friend, with the role of helping students to recognize both their strengths and weaknesses. However,…

Recent developments have enabled the modeling of longitudinal assessment data in a diagnostic classification model (DCM) framework. These longitudinal DCMs were developed to provide measures of student growth on a discrete scale in the form of attribute mastery transitions, thereby supporting categorical and criterion-referenced interpretations of…

Offering explanations is a central part of teaching mathematics, and understanding those explanations is a vital activity for learners. Given this, it is natural to ask what makes a good mathematical explanation. This question has received surprisingly little attention in the mathematics education literature, perhaps because the field has no…

Creative orientation is the extent to which different individuals are drawn toward creative activities (e.g., art, music). We know relatively little about child-level creative orientation given certain testing limitations. Adult tools often measure time spent engaged in creative pursuits, but this method is unsuitable for children because their…

Coefficient alpha reliability persists as the most common reliability coefficient reported in research. The assumptions for its use are, however, not well-understood. The current paper challenges the commonly used expressions of coefficient alpha and argues that while these expressions are correct when estimating reliability for summed scores,…

Our study investigates whether confidence correlates with consistency in reasoning, specifically about radioactive decay. In prior work, we developed and tested a survey designed to measure consistency of student reasoning about radioactive decay by comparing responses to three prompts that are isomorphic, meaning that, despite having different…

The assessment of human performance is not a new phenomenon. We have evidence that people have been required to prove their worth dating back at least to the Epic of Gilgamesh. What has changed, at least on a large scale, is the importance given to quantitative evidence in the evaluation process. For example, many employers have begun subjecting…

Though qualitative research has become more prevalent in practice over the last 30 years, there is still considerable uncertainty among researchers regarding how to ensure inter-rater consistency when teams are tasked with coding qualitative data. In this article, we offer an explanation of a methodology our qualitative team used to achieve…

In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze…

As an alternative to Cronbach's [alpha] for estimating scale reliability, McDonald's [omega] has attracted increased attention within the methodological community for its less stringent measurement assumptions. Notwithstanding, [omega] is still seldom used by practitioners, likely due to its unavailability in popular software packages (e.g., SPSS)…

The advent of artificial intelligence (AI), particularly large language models (LLMs) like ChatGPT and Gemini, has significantly impacted the educational landscape, offering unique opportunities for learning and assessment. In the realm of written assessment grading, traditionally viewed as a laborious and subjective process, this study sought to…

Purpose: The purpose of this study was to extend the assessment of the psychometric properties of the Modified Barium Swallow Impairment Profile (MBSImP). Here, we re-examined structural validity and internal consistency using a large clinical-registry data set and formally examined rater reliability in a smaller data set. Method: This study…

Questionnaire scales that are mixed-worded, i.e. include both positively and negatively worded items, often suffer from issues like low reliability and more complex latent structures than intended. Part of the problem might be that some responders fail to respond consistently to the mixed-worded items. We investigated the prevalence and impact of…

We study the returns to experience in teaching, estimated using supervisor ratings from classroom observations. We describe the assumptions required to interpret changes in observation ratings over time as the causal effect of experience on performance. We compare two difference-in-differences strategies: the two-way fixed effects estimator common…