Creativity research crucially relies on creativity evaluations by external raters, but it is not clear what properties characterize good raters. In the present study, we investigated whether rater personality and rater creativity are related to discernment (i.e., the ability to distinguish creative from uncreative responses) when evaluating others' responses. To this end, 166 novice raters evaluated a pre-selected set of responses from the alternate uses task for creativity. We further assessed the rater's creativity and personality and explored the role of task factors such as rating motivation and rating time. We found that creativity in terms of creative potential and creative achievement independently predicted rater discernment. At the level of personality, openness to experience was related to higher discernment, but not any other Big-Five factors or dark triad traits. Rater discernment also increased with rating time, rating motivation, and rater leniency (i.e., the tendency to appreciate creative responses). Finally, we seized the large rater sample to estimate to what extent inter-rater reliability depends on rater number and how dependable ratings are across the rating scale. Together, these findings highlight relevant factors at person, task, and method level predicting individual discernment, which should be considered to ensure good inter-rater reliability in creativity ratings.