The aim of the present study is to develop a valid and reliable scale evaluating the effectiveness of language preparatory programs in the acquisition of language skills. In the development of Foreign Language Skills Scale (FLSS) in this study, research sample consisted of 326 preparatory school students for the exploratory factor analysis (EFA) and 350 preparatory school students for the confirmatory factor analysis (CFA). Based on the data obtained from the first sample, an EFA was carried out on the FLSS. EFA has identified that 27 items of the scale have factor loads between 0.519 and 0.729, while they explain 65.376% of the total variance and are distributed under five factors. These factors are named as "writing skill," "speaking skill," "listening skill," "core skills," and "reading skill." A CFA was applied on the data obtained from the second sample that consisted of 350 students. As a result of the CFA, it was confirmed that the FLSS consisted of 27 items and five factors. For all the items in the scale, item-subscale, item-test correlation coefficients and mean differences between the upper and the lower 27% of the participants were calculated, and it is determined that each item is consistent with not only the subscale it is under but also the whole test. In addition, the Cronbach's Alpha reliability coefficients of the total scale's and five sub-scales' internal consistency is quite high. The FLSS is expected to offer a comprehensive evaluation of the acquisition of four language skills in foreign language teaching programs.