The study evaluated a recently postponed national test policy on the use of external agencies' English tests measuring four skills for Japanese university admission purposes. Using Kunnan's principles of fairness and justice, we generated and evaluated three claims regarding the test policy: vocabulary coverage of the tests, the justifiability of the Common European Framework of References for Languages (CEFR)-based concordance table, and tests' impact on teaching and learning productive skills. Scrutinizing the backing and rebuttal evidence from the test agencies' voices to the key literature, we arrived at a partial acceptance of all three claims with the need for further research, calling into question a optimistic outlook underlying the test policy.