This paper reports on a test that is being used to fulfill a university writing examination requirement for non-native speakers of English. The test, which requires students to read two passages, write short-answer comprehension and synthesis questions, and write an argument essay on a topic related to the passages, replaces a test that was based on a choice among several short prompts. In the paper, three aspects of test validation are considered. First, a comparison between the new test and the old test shows that rater reliability and consistency across test topics is substantially higher on the new test. Second, a review of transcripts of students who failed the new test three or more times reveals that these students are also having difficulty in English and History courses. Finally, an interview with a test preparation course instructor reveals that the new test is having positive washback: that is, the course to prepare students for the new course focuses on skills that are more easily transferable to other academic courses than the course preparing students for the old test. The costs of preparing a test of this nature are addressed, and the argument is put forward that if students are required to pass an examination as a condition of graduation the institution must be willing to provide sufficient resources to test students fairly. (Contains 2 figures and 6 tables.)