In this digital ITEMS module, Dr. Jacqueline Leighton and Dr. Blair Lehman review differences between think-aloud interviews to measure problem-solving processes and cognitive labs to measure comprehension processes. Learners are introduced to historical, theoretical, and procedural differences between these methods and how to use and analyze…

The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have unveiled a wealth of opportunities and challenges in assessment. Applying cutting-edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely,…

Experimental psychology is witnessing an increase in research on individual differences, which requires the development of new tasks that can reliably assess variations among participants. To do this, cognitive researchers need statistical methods that many researchers have not learned during their training. The lack of expertise can pose…

The case study is a widely used method in qualitative research. Although defining the case study can be simple, it is complex to develop its strategy. Furthermore, it is still often not considered to be a sufficiently robust research strategy in the education field because it does not offer well-defined and use well-structured protocols. One of…

Practitioners often struggle to assess reflective learning in the workplace because of difficulties conceptualizing reflection and its effects in the workplace. This article addresses this problem by offering a pragmatic approach to assessment that asks practitioners to specify why they are using reflection, what they are hoping to gain from it,…

To identify the dominant cultural models among parents transmitting faith to their children, we find few methodological guidelines to guide coding and analysis of semi-structured interviews. We thus developed a three-phase procedure for our research team. Phase-one follows Campbell et al. by unitizing on meanings rather than words/pages, including…

Over the last 20 years there has been significant growth in the volume of higher education pedagogical research across disciplines and national contexts, but inherent tensions in defining quality remain. In this paper we present a framework to support understanding of what constitutes internationally excellent research, drawing on a range of…

The Student Evaluation of Teaching (SET) instrument provides insight for instructors and administrators alike, often touting high response-rates to endorse their validity and reliability. However, response-rate alone omits consideration for "adequate quantity of 'observational sampling opportunity' (OSO) data points" (e.g., high student…

An overview of the evolution of career information in light of the changing nature of the world of work is presented. Owing to the constant fundamental changes in the labour market, the distribution of paid work has been also constantly changing. In this article, a more dynamic and -- often temporary -- interplay between citizens and their…

This concept paper highlights the problem of the lack of a unified Arab list of Bloom's taxonomy to be used in teacher-preparation programs across Arab universities. The paper illustrates the current problem and offers steps needed for completing a project that would produce a unified list. The unified list would have both the required validity…

Rater-mediated assessments require the evaluation of the accuracy and consistency of the inferences made by the raters to ensure the validity of score interpretations and uses. Modeling rater response processes allows for a better understanding of how raters map their representations of the examinee performance to their representation of the…

The identification of rapid guessing is important to promote the validity of achievement test scores, particularly with low-stakes tests. Effective methods for identifying rapid guesses require reliable threshold methods that are also aligned with test taker behavior. Although several common threshold methods are based on rapid guessing response…

Single-case design has a long history of use for assessing intervention effectiveness for children with disabilities. Although these designs have been widely employed for more than 50 years, recent years have been especially dynamic in terms of growth in the use of single-case design and application of standards designed to improve the validity…

The author discusses valid and reliable ways to assign grades in an academic course in any discipline. "Validity" means the accuracy of a grade's reflection of student learning and achievement. "Reliability" concerns the degree of random error that might be present and affect validity. First the author defines a grade as a…

Resistant behaviors in multicultural courses are a common dilemma within counselor education. On the basis of psychological reactance theory (J. W. Brehm, 1966), the authors created a multicultural training reactance scale to measure the construct. Exploratory factor analysis and initial reliability and validity data are presented, followed by…