With the rapid development of internet technology, the importance of data in various industries has become increasingly prominent. However, in the field of domestic education, especially the research on teaching quality evaluation is relatively sparse. This paper aims to build an evaluation model of physical education teaching quality with big data and machine learning in order to comprehensively evaluate teachers' physical education teaching ability. Firstly, a multi-dimensional evaluation system is designed to measure teachers' teaching level. Secondly, through the questionnaire and the statistical analysis of students' physical education teaching evaluation, the availability of the college physical education teaching evaluation system is verified. On this basis, a teacher's ability prediction model based on a neural network is established, and the experimental results show that the prediction accuracy of the model is 90.17%. This provides strong support for improving the quality of physical education teaching.