"Pseudowords", or non-real words, were introduced to the Yes/No (YN) vocabulary test format to provide a means of checking for overestimation of word knowledge by test takers. The purpose of this study is to assess the assumption that more pseudoword checks (false alarms) indicate more instances of overestimation of word knowledge in YN tests. Thirty English classes in five different Japanese universities with TOEIC[R] scores ranging from 230 to 730 participated (n = 490). YN test results were compared with a multiple-choice test of the same 96 real words to provide a way to check directly for instances of underestimation and overestimation of word knowledge on the YN tests. Results showed that students from the higher proficiency universities had a slightly higher pseudoword false alarm rate than students from the lower ability universities (4.28% and 3.96%, respectively). However, overestimation rates were considerably lower for these same students from the higher proficiency universities (3.24% and 5.67%, respectively). This discrepancy between false alarm rates and overestimation rates questions the value of pseudowords for measuring overestimation in YN vocabulary tests when student ability levels differ significantly. (Contains 15 tables.)