Engaging students in the creation of learning resources is an effective way of developing a repository of revision items. However, a selection process is needed to separate high- from low-quality resources as some of the materials created by students can be ineffective, inappropriate or incorrect. In this study, we share our experiences and findings in incorporating three approaches for peer evaluation of student-generated content, using an educational technology called RiPPLE which supports learner sourcing of study materials. All approaches asked students to score peer-developed resources, rate their confidence in that judgment and to provide written feedback or justification of their evaluation to the resource's author. The first approach provided no guidance to scoring. The second supplied a simple instructor-developed rubric for scoring. The third utilised a simple data-informed rubric incorporating criteria derived from thematic analysis of students' comments provided in the first two conditions. The analysis of 8,213 evaluations of student-generated content from 813 students shows that the data-informed rubric produced more rigorous evaluations, higher agreement in scores but lower student confidence in the accuracy of their assessments. The analysis of comments shows that students went beyond the criteria supplied in the rubrics and applied their own understanding of resource quality.