Writing assessment essentially juxtaposes two elements: how "good writing" is to be defined, and how "good measurement" of that writing is to be carried out. The timed test is often used in large-scale L2 writing assessments because it is considered to provide reliable measurement. It is, however, highly inauthentic. One way of enhancing authenticity in a way that might also promote good writing is to include access to support resources in the test. This article reports a study into whether allowing a bilingual dictionary enhances good writing in German writing tests. Participants (n = 47) took two tests, one with and one without a bilingual dictionary. The essays of sub-sets of the participants were compared in terms of lexical sophistication, lexical accuracy and test score. Lexical sophistication was measured by means of the Lexical Frequency Profile (Laufer & Nation, 1995). Lexical accuracy was determined through a quantitative analysis of words looked up in the dictionary. The results revealed an increase in lexical sophistication in "with dictionary" tests but frequent misuse of look-ups. There was also no improvement in test scores. Subsequent issues for the use of dictionaries in writing tests are raised. (Contains 12 tables.)