Although there is abundant evidence for the use of first-language (L1) knowledge by bilinguals when using a second language (L2), investigation into the impact of L1 knowledge in large-scale L2 language assessments and discussion of how such impact may be controlled has received little attention in the language assessment literature. This study examines these issues through investigating the use of L1-Japanese loanword knowledge in test items targeting L2-English lexical knowledge in the Reading section of EIKEN grade-level tests, which are primarily taken by Japanese learners of English. First, the proportion of English target words that have loanwords in Japanese was determined through analysis of corpus-derived wordlists, revealing that the distribution of such items is broadly similar to that in language in general. Second, the impact of loanword frequency in Japanese (and cognate status) was demonstrated through statistical analysis of response data for the items. Taken together, the findings highlight the scope and impact of such cognate items in large-scale language assessments. Discussion centers on how test developers can and/or should deal with the inclusion of cognate words in terms of context validity and test fairness.