The emergence of digital-first assessments is prompting reconsideration of, and innovation in, aspects of psychometrics, test validation, and test use. Using the Duolingo English Test (DET) as an example, this three-paper series seeks to address issues concerning the estimation of classification consistency and the reporting of results for such…

Strategic competence, or the cognitive resources necessary for utilizing language knowledge, is widely recognized as an essential component of second language (L2) ability. However, research on strategic competence in the context of L2 speaking assessment has been limited, and findings have been inconsistent, making it challenging to discern the…

The study aimed to elicit information from Vietnamese EFL university instructors about their knowledge and skills regarding the principles, theory, and practices of language assessment by means of revision and validation of the Language Assessment Literacy--Revised Vietnam (LAL-RV), which was previously developed by Kremmel and Harding (2020). A…

Under the framework of item response theory (IRT) and generalizability (G-) theory, this study examined the effects of item difficulty on rating reliability and construct validity on both the constructed-response (CR) items and essay items on English examinations. The data collected for this study were students' scores and responses on the two…

Assessment is an integral part of learning as it is used to gather information about a test-taker. Those in the field of academia, such as educational policy makers, instructors, and administrators are able to use information gathered from tests to further instruction and learning decisions (Baker, 2006; Drianna, 2007; Kasper & Ross, 2013;…

Rater variation in performance assessment can impact test-takers' scores and compromise assessments' fairness and validity (Crooks, Kane, & Cohen, 1996). Rater variation can also undermine a test's validity and fairness; therefore, it is important to investigate raters' scoring patterns in order to inform rater training. Substantial work has…

Few classroom measures of English language proficiency have been evaluated for reliability and validity. This research examined the concurrent and predictive validity of an oral language test, titled A Developmental English Language Proficiency Test (ADEPT), and the relationship to the California English Language Development Test (CELDT) in the…

This study investigated the breakdown effect of a listening comprehension test, whether test takers are affected in comprehending lectures by impediments, and collected test takers' cognitive awareness on test tasks which contain listening breakdown factors how they perceived these impediments. In this context of the study, a "Breakdown" is a test…

Speaking performance tests typically employ raters to produce scores; accordingly, variability in raters' scoring decisions has important consequences for test reliability and validity. One such source of variability is the rater's level of expertise in scoring. Therefore, it is important to understand how raters' performance is influenced by…

Performance assessments have become the norm for evaluating language learners' writing abilities in international examinations of English proficiency. Two aspects of these assessments are usually systematically varied: test takers respond to different prompts, and their responses are read by different raters. This raises the possibility of undue…

This study examined the second language (L2) development of adult learners of Spanish at three levels of proficiency during and after a semester of instruction. A fundamental goal was to identify cognitive and psychosocial individual differences (IDs) that can explain between-learner variation over time in order to expand our understanding of the…

In writing assessment, there are quite a number of factors influencing the marking stability and the reliability of the assessment such as the attitude towards marking and consistency of markers, the physical environment, the design of the items, and marking rubrics. Even the methods to train markers have effects on the reliability of the…

A renewed focus on foreign language (FL) learning and speech for communication has resulted in computer-assisted language learning (CALL) software developed with Automatic Speech Recognition (ASR). ASR features for FL pronunciation (Lafford, 2004) are functional components of CALL designs used for FL teaching and learning. The ASR features…

This dissertation reports development and validation studies of a Web-based standardized test of Japanese as a foreign language (JFL), designed to measure learners' off-line grammatical and pragmatic knowledge in multiple-choice format. Targeting Japanese majors in the U.S. universities and colleges, the test is designed to explore possible…