Like instruments used in afterschool programs to assess children's social and emotional growth or to evaluate staff members' performance, instruments used to evaluate program quality should be free from bias. Practitioners and researchers alike want to know that assessment instruments, whatever their type or intent, treat all people fairly and do not privilege people from certain groups over others. Overcoming this possible source of bias is the concern of the authors in this article as members of the research and evaluation team for the Assessment of Program Practices Tool (APT). This article describes the follow-up study they conducted to address the sources of that apparent bias, with the goal of making the APT rater reliability exam as free from cultural bias as possible. This goal is critical for any educational assessment, though it is often dismissed. During this follow-up study, they addressed practical concerns that have implications for the development of culturally fair program quality assessments across the field.