Purpose: Estimating the sequential associations between educators' and children's talk during language learning interactions requires careful consideration of factors that may impact measurement stability and resultant inferences. This research note will describe a preliminary study that used generalizability theory to understand the contribution…

Carl Kaestle defines a generalization as "how we know when we know." Kaestle sketches a model of increasing certainty in historical claims as they are developed and refined at increasing scales of research, from local to international. A historical claim might originate in the study of a particular place or case, but to know that the…

This study aims to compare the G and Phi coefficients as estimated by D studies for a measurement tool with the G and Phi coefficients obtained from real cases in which items of differing difficulty levels were added and also to determine the conditions under which the D studies estimated reliability coefficients closer to reality. The study group…

Machine learning has achieved state-of-the-art performance in many areas, including image recognition and natural language processing. However, there are still many challenges and mysteries attracting numerous researchers. This dissertation comprises a series of works concerning problems at the intersection of computer science theory, adversarial…

This article serves as an introduction to the Special Issue on "Decolonizing and Diversifying Research in Cognitive Development." The Special Issue comprises six articles: two articles are empirical articles that focus on executive function development in under-represented environments, two articles address barriers pathways toward…

Game-based assessment (GBA), a specific application of games for learning, has been recognized as an alternative form of assessment. While there is a substantive body of literature that supports the educational benefits of GBA, limited work investigates the validity and generalizability of such systems. In this paper, we describe applications of…

Writing assessments often consist of students responding to multiple prompts, which are judged by more than one rater. To establish the reliability of these assessments, there exist different methods to disentangle variation due to prompts and raters, including classical test theory, Many Facet Rasch Measurement (MFRM), and Generalizability Theory…

Background/Context: Previous research in educational assessment has consistently emphasized the importance of reliability as a cornerstone of test quality. Traditional measures of reliability, such as test-retest and split-half reliability, offer a broad view of how internally consistent a measure is but overlook the variability in this internal…

Likert-type item is the most popular response format for collecting data in social, educational, and psychological studies through scales or questionnaires. However, there is no consensus on whether parametric or non-parametric tests should be preferred when analyzing Likert-type data. This study examined the statistical power of parametric and…

Students often believe that statistical significance is the only determinant of whether a quantitative result is "important." In this paper, I review traditional null hypothesis statistical testing to identify what questions inferential statistics can and cannot answer, including statistical significance, effect size and direction,…

Teacher evaluation systems often include classroom observations in which raters use rating scales to evaluate teachers' effectiveness. Recently, researchers have promoted the use of multifaceted approaches to investigating reliability using Generalizability theory, instead of rater reliability statistics. Generalizability theory allows analysts to…

The aim of this paper is to introduce a software that is appropriate for the generalizability theory for not only balanced but also unbalanced data sets. Because it is possible to have unbalanced data sets while conducting a study, the researchers have devised an easy solution, other than deleting data, to balance the design to cope with this…

Validity is broadly defined as how well something measures what it's supposed to measure. The reliability and validity of scores from assessments are two concepts that are closely knit together and feed into each other.

It has become a critical question what the reliability level would be when open-ended questions are used in large-scale selection tests. One of the aims of the present study is to determine what the reliability would be in the event that the answers given by test-takers are scored by experts when open-ended short answer questions are used in…

Learning curves can support a competency-based approach to assessment for learning. When interpreting repeated assessment data displayed as learning curves, a key assessment question is: "How well is each learner learning?" We outline the validity argument and investigation relevant to this question, for a computer-based repeated…