The purpose of this study was to examine the assigned ratings, interrater reliability, and possible influences of school level and instrumentation on adjudicators' evaluations of orchestra performances at a national-level adjudicated music festival. Data consisted of the overall ratings assigned to orchestra performances (N = 55) at the 2017, 2018, and 2019 American String Teachers Association's National Orchestra Festival (NOF). Analysis revealed that 83.64% of all participating orchestras earned a I/Superior or II/Excellent overall rating. A logistic regression model revealed that both school level and instrumentation were not significant predictors of earning a I/Superior overall rating. Individual members of each three-judge adjudication panel assigned identical ratings to 61.82% of orchestra performances and possessed a mean internal consistency of .90. Orchestra teachers can use the results of this study to weigh the educational benefits of participating in the NOF.