We retrieved data from China Education Panel Survey (CEPS) and adopted the hierarchical linear model as well as the quantile regression method to examine the effect of education groups on the quality of urban and rural compulsory education. Student cognitive ability was cited as the evaluation index of education quality. According to the research findings, the overall effect of education groups on the quality of compulsory education is significantly positive, but there is urban-rural heterogeneity in this effect: the positive impact occurs only to urban member schools, while the education quality of rural member schools has not been improved by school grouping. In addition, there exists individual difference in this effect; school grouping significantly and negatively affects students from rural member schools and with medium and lower cognitive ability.