This paper reports on the challenge of evaluating students' scientific literacy in a writing-to-learn context, as illustrated by our experience with an online science-writing project. In this mixed methods study, year 9 students in a case study class (13-14 year olds, n?=?26) authored a series of two "hybridised" short stories that merged scientific and narratives genres about the socioscientific issue of biosecurity. In seeking to measure the efficacy of the intervention, we sought evidence of students' conceptual understanding communicated through their stories. Finding a suitable instrument presented our first challenge. This led to the development of scoring matrices to evaluate students' derived sense of scientific literacy. Student interviews were also conducted to explore their understanding of concepts related to the biosecurity context. While the results of these analyses showed significant improvements in students' understanding arising from their participation in the writing tasks, the interviews highlighted a second challenge in evaluating students' scientific literacy: a disparity between their written and vocalised understandings. The majority of students expressed a deeper level of conceptual understanding during the interviews than they did in their stories. The interviews also revealed alternative conceptions and instances of superficial understanding that were not expressed in their writing. Aside from the methodological challenge of analysing stories quantitatively, these findings suggest that in a writing-to-learn context, evaluating students' scientific literacy can be difficult. An examination of these artefacts in combination with interviews about students' written work provided a more comprehensive evaluation of their developing scientific literacy. The implications of this study for our understanding of the derived sense of scientific literacy, as well as implications for classroom practice, are discussed.