Out-of-school time (OST) youth programs are inherently difficult to assess. They are often very dynamic: Many youth interact with one another and with staff members in various physical environments. Despite the challenge, measuring quality is critical to help program directors and policy makers identify where to improve and how to support those improvements. This article describes recent research on the Assessment of Program Practices Tool (APT), establishing its strength as an evaluation and tracking tool for OST programs. The validation was conducted in two phases. The first phase was designed to evaluate the scientific rigor of the tool. Based on the findings from the first phase, the second aimed to inform improvements in the tool and its training. The testing so far shows that online video-based training needs to be more specialized in order to improve rating reliability for high-stakes users, such as third-party evaluators.