The assessment of noncognitive traits is challenging due to possible response biases, "subjectivity" and "faking." Standardized third-party evaluations where an external evaluator rates an applicant on their strengths and weaknesses on various noncognitive traits are a promising alternative. However, accurate score-based inferences from third-party evaluations requires disentangling score variance due to raters versus applicants by utilizing a multilevel factor analysis (MFA). To date, MFA is highly underutilized in the measurement field. In this study, we apply the MFA to analyze third-party evaluations using data from the Personal Potential Index (PPI). The PPI is a third-party measure used to evaluate graduate school applicants noncognitive traits to help inform admissions decisions. We analyzed 12,693 ratings of 6,249 applicants divided into two randomly selected subgroups. We conducted multilevel exploratory factor analysis with one subgroup and tested the hypothesized structure with the other subgroup. This work illustrates the advantages and challenges of using MFA approach to support the meaningful and valid interpretation of scores from third-party evaluations.