Background: This study investigated the scoring and criterion-related validity of the TEAP, a newly developed Test of English for Academic Purposes. In this study, scoring validity was examined by investigating the factor structure, while criterion-related validity was examined by first investigating the longitudinal change of test takers' perceived test task value toward the measured construct and then investigating the relationship of test takers' perceived value to the factor structure of the TEAP. Methods: Confirmatory item-level factor analysis was conducted using the data obtained from 2217 first-year university students comparing four models (unitary, correlated, receptive-productive, higher order). Additional confirmatory factor analyses were conducted to first investigate the longitudinal change of perceived value toward the measured construct and then to investigate the relationship of test takers' perceived value of the construct measured by the test to the factor structure of the TEAP. Results: The results show that the higher-order model was the best-fitting model. This confirmed a previous small-scale study suggesting the generalizability of the test's factor structure. Furthermore, it was found that test takers' perceived values measured at the start of university positively affected the values measured about 6 months later. In addition, perceived values measured both at the start of university study and about 6 months later positively correlated with the higher-order factor of the test. Conclusions: The results provide further support of the scoring validity of the test. In addition, a positive relationship between the higher-order factor of TEAP and the factors of perceived values provide evidence of the usefulness of test takers' perception to further support the criterion-related validity of the test.