In peer assessment, students assess a task done by their peers, provide feedback and usually a grade. The extent to which these peer grades can be used to formally grade the task is unclear, with doubts often arising regarding their validity. The instructor could supervise the peer assessments, but would not then benefit from workload reduction, one of the most appealing features of peer assessment for instructors. Our proposal uses a probabilistic model to estimate a grade for each test, accounting for the degree of precision and bias of grading peers. The grade that the instructor would assign to a test can help enhance the model. Our main hypothesis is that guiding the instructor through supervision of a peer-assessed task by pointing out to them which test to evaluate next can lead to improvement in the validity of the model-estimated grades at an early stage. Moreover, the instructor can decide how many tests to grade based on their own criteria of tolerable uncertainty, as measured by the model. We validate the method using both synthetically generated data and real data collected in an actual class. Models that link the roles of the student as grading peer and as test-taker appear to better exploit available information, although simpler models are more appropriate in specific conditions. The best performing technique for guiding the instructor is that which selects the test with the highest expected entropy reduction. In general, empirical results are in line with the hypothesis of this study.