Purpose: We compared outcomes from 2 measures of language ability in children who displayed a range of dialect variation: 1 using features that do not contrast between mainstream American English (MAE) and nonmainstream dialects (NMAE), and 1 using contrastive features. We investigated how modified scoring procedures affected the diagnostic accuracy of the measure with contrastive features. Method: Second-grade students (N = 299; 167 White, 106 African American, 26 other) completed measures of language variation and ability (the Diagnostic Evaluation of Language Variation-Screening Test and the Clinical Evaluation of Language Fundamentals-Fourth Edition [CELF-4]). The CELF-4 was scored with and without the recommended scoring modifications for children who spoke African American English. Results: Partial correlations controlling for socioeconomic status revealed small to moderate correlations between measures of language ability and the use of NMAE features. Modified scoring yielded higher scores for children who spoke African American English and a reduced association between the use of NMAE features and CELF-4 scores. Modified scoring also affected the diagnostic accuracy of the CELF-4, resulting in a lower positive likelihood ratio and a higher negative likelihood ratio. Conclusions: The decision to apply scoring modifications affects both the false positive and false negative rates. Implications for language assessment for children who speak NMAE dialects are discussed, including the need for further investigation.