The purpose of this article is to discuss reliability and precision through the lens of a modern measurement approach, item response theory (IRT). Reliability evidence in the field of counseling is primarily generated using Classical Test Theory (CTT) approaches, although recent studies in the field of counseling have shown the benefits of using…

Design is core element of general technology education internationally. While there is a degree of contention with regards to its treatment, there is general consensus that the inclusion of design in some form is important, if not characteristic, of the subject area. Acknowledging that design is important, there are many questions which need to be…

In this work, we have developed an alternative device composed by an Arduino board and an INA219 sensor to experimentally obtain the mathematical formulas describing the charge and discharge of the capacitor for educational proposes. We have obtained excellent agreement between theoretical prediction and experimental measurements. The INA219 DC…

The cloud-based research synthesis platform metaBUS is presented as a tool for human resource development research. A general overview of the meta-analysis tool metaBUS is provided. In addition, this article shows how metaBUS can be utilized for other research purposes such as a starting point for identifying keywords when conducting literature…

Whenever classification decisions are made on educational tests, such as pass/fail, or basic, proficient, or advanced, the consistency and accuracy of those decisions should be estimated and reported. Methods for estimating the reliability of classification decisions made on the basis of educational tests are well-established (e.g., Rudner, 2001;…

This methodological paper examines current conceptions of reliability in chemistry education research (CER) and provides recommendations for moving beyond the current reliance on reporting coefficient alpha (a) as reliability evidence without regard to its appropriateness for the research context. To help foster a better understanding of…

This article maps the territory of visual ethnography as a key and accessible research methodology in education. It aims to provide an overview and to present theory and practice for future research. The origins and principles of visual ethnography are disclosed as well as some methods to gather data. From the premise that either created by the…

There are two main lines of research in estimating classification accuracy (CA) and classification consistency (CC) under Item Response Theory (IRT). The R package cacIRT provides computer implementations of both approaches in an accessible and unified framework. Even with available implementations, there remains decisions a researcher faces when…

In adolescence research, the treatment of measurement reliability is often fragmented, and it is not always clear how different reliability coefficients are related. We show that generalizability theory (G-theory) is a comprehensive framework of measurement reliability, encompassing all other reliability methods (e.g., Pearson "r,"…

A classic topic in the fields of psychometrics and measurement has been the impact of the number of scale categories on test score reliability. This study builds on previous research by further articulating the relationship between item response theory (IRT) and classical test theory (CTT). Equations are presented for comparing the reliability and…

Massive Open Online Courses (MOOCs) are becoming an increasingly popular choice for education but, to reach their full extent, they require the resolution of new issues like assessing students at scale. A feasible approach to tackle this problem is peer assessment, in which students also play the role of assessor for assignments submitted by…

Many states and school districts are rethinking how they do educational assessment. A few are going further: attempting to build "balanced," "comprehensive," or "next-generation" assessment systems. At the same time, practitioners and researchers have long mulled the purposes and parts such systems should have. But…

The dangers of a single story in current U.S. large-scale writing assessment are that assessment practice does not align with theory and this practice has negative effects on instruction and students. In this article, I analyze the connections or lack of connections among writing theory, writing assessment, and writing instruction, critique the…

Assuming item parameters on a test are known constants, the reliability coefficient for item response theory (IRT) ability estimates is defined for a population of examinees in two different ways: as (a) the product-moment correlation between ability estimates on two parallel forms of a test and (b) the squared correlation between the true…

Cultural consensus theory (CCT) is a data aggregation technique with many applications in the social and behavioral sciences. We describe the intuition and theory behind a set of CCT models for continuous type data using maximum likelihood inference methodology. We describe how bias parameters can be incorporated into these models. We introduce…