Recent trends in higher education (rising debt, school closures, increasing tuition) have increased interest in improving accountability and oversight in U.S. higher education beyond current accreditation practices. Common solutions include using quantitative measures like graduation and default rates to benchmark performance. Using historical data on accreditation actions, we compare the performance of a hypothetical quantitative evaluation system to contemporary accreditation standards in identifying low-performing schools. We find first that schools facing accreditation sanctions are, on average, also low performing on the quantitative outcomes we consider. However, if we use this average performance as a benchmark to identify other low-performing schools, then a substantial portion of the higher education sector would be implicated. These results raise questions about the capacity of a purely quantitative evaluation system to narrowly identify low-performing schools. We conclude with a discussion of the issues concerning data, equity, and other considerations for higher education accountability moving forward.