Speech perception research has commonly relied on untrained listeners to provide impressionistic judgments of second language (L2) speech, and training listeners has been rare. The current study therefore explores the implications of training novice listeners (NLs) to rate accented speech. A total of 60 listeners were recruited, 30 who remained novice and 30 who underwent training. Both groups provided ratings of accentedness, comprehensibility, and rated listener comprehension using 5-point scales; however, the NLs did so with the traditional Likert scales, and the trained listeners (TLs) did so with evidence-based rubrics. Multi-faceted Rasch measurement was used to investigate differences in severity, consistency, and rating behavior between the two groups. Results showed that, when compared to NLs, TLs rated the speech criteria with fewer levels of severity, more consistency, and improved scalar precision, increasing the construct validity of their ratings. Implications suggest that a short training intervention can change the way NLs make judgments of L2 speech in addition to how they interact with the rating scale, thereby decreasing listener error and boosting the quality of speech measurements.