The testing of bilingual students poses particular problems for analyses of performance, item bias, and test adequacy. A test administered in two languages to children selected for their language facility provides a special arena for the study of these problems. The Comprehensive Tests of Basic Skills (CTBS) was selected because the test content between the English and Spanish language versions is similar in rationale, administration, and interpretation; the differences that exist between the language versions are the result of literal translation problems. Evidence based on performance of English- and Spanish-speaking pupils suggests that the CTBS contains multiple sources of bias. The vocabulary subtest of the CTBS was administered to 1162 second-graders in bilingual education programs in the Southwest; 58 students received both test versions because the students were equally proficient in both languages. Results show that patterns of performance for these students differ markedly between the two language versions, supporting the contention that the method of direct translation from English to Spanish for bilingual vocabulary testing may not be fully adequate for the needs of the bilingual program student, even when the Spanish version is a rather faithful translation of the English original. (Author/PN)