The Every Student Succeeds Act (ESSA) began a new wave of school accountability under which states draw on multiple measures to assess school quality. States have options in terms of how to weight components in their school quality indices and how many years of data to use to determine school ratings. In this study, we simulate school ratings using eight years of administrative data in North Carolina to demonstrate how state decisions about school ratings and identification influence school ratings and the list of schools identified for improvement. We then evaluate these decisions against a framework that considers the validity, stability, and equity of the ratings, underscoring the inherent tradeoffs that come with each. We show that while a system that weights proficiency more heavily than growth produces more stable school ratings, identifying schools based on multiple years of performance data instead of one more than offsets the loss of stability in shifting to a growth measure. We conclude with recommendations for state accountability systems under ESSA and for federal policymaking moving forward.