Research indicates that the performance-gap between English Language Learners (ELLs) and their non-ELL peers is partly due to ELLs' difficulty in understanding assessment language. Accommodations have been shown to narrow this performance-gap, but many accommodations studies have not used a randomized design and are based on relatively small sample sizes. Addressing such issues, we administered a standard-based mathematics assessment to approximately 3,000 Grade 9 ELL and non-ELL students under five different language-based accommodations. Results indicate that many of these accommodations did not produce significant gains for the recipients. Some even had a negative impact. We believe several factors may explain these findings. First, newer assessments, including those developed for this study, may have been linguistically modified to the point that further modification has only a limited effect. Second, the language of instruction may have not adequately prepared students for the assessment. If the language of instruction (textbook, etc.) contains unnecessary linguistic complexity, then students may not have had the opportunity to learn the assessed content. A third factor is students' unfamiliarity with these accommodations because they are seldom used in classroom instruction and teacher assessments. We discuss our findings and implications for policymakers, assessment developers, practitioners, and researchers.