Based on the premise that teachers gain useful insights by considering what students notice when evaluating a written text, this pilot project reports on the implementation of an evaluative activity with one group of students and considers the implications of the results for classroom practice. Thirteen students in a university-level Spanish composition class completed three stages of data collection; they identified strong and weak points of a sample composition, shared their assessments in a class discussion, and completed reflective questionnaires focused on perceptions of writing. Of their 86 observations, 19 were focused on technical issues of grammar/mechanics and 24 on communicative issues of argumentation. The manner in which learners presented those features also provides insight into their perceptions. Certain features, inclusion of personal opinion, use of person, and traditional three-part organisation, were sometimes framed as hard-and-fast rules; participants did not always mention their value as communicative tools. Other participants took a critical perspective that considered the role of features like organisation and spelling in the effective communication of a written message. Results indicate that attending to what learners notice has potential as a pedagogical tool.