The Comprehensive English Regents Exam, also referred to as the English Language Arts (ELA) Examination, is a widely used measure of student achievement in the K-12 system in New York, yet it is not without criticism. One concern is that the ELA Regents Exam may be contributing to the widening of the gap in graduation rates between English language learners (ELLs) and English-proficient students (NYSED, 2014b). This suggests a threat to the validity of the inferences made based on test scores. When considering the features of test design and the characteristics of the test's diverse target population under an argument-based validity approach (Bachman, 2003, 2005; Kane, 1992, 2002), a mismatch between the intended consequences of test use and the actual effects on the involved stakeholders can be identified. The purpose of this commentary is to analyze possible threats to validity, with issues ranging from domain analysis and construct definition to test misuse from a language assessment perspective and to discuss the impact of the Regents Examination in ELA on newly-arrived ELLs.