It is necessary to supply proof regarding the construct validity of the scales. Especially, when new scales are developed the construct validity is researched by the Exploratory Factor Analysis (EFA). Generally, factor extraction is performed via the Principal Component Analysis (PCA) which is not exactly factor analysis and the Principal Axis Factoring (PAF) among EFA methods. Factors may also be determined with different techniques depending on the advances in psychometry. In the context of nonparametric item response theory, the Mokken Scale Analysis (MSA) and the Automatic Item Selection Procedure (AISP) provide significant contributions to researchers in scale development studies. The aim of the current study is to compare the AISP and the EFA methods in determining the factor structures. The Revised Life Orientation Test, whose factor structure was previously known and validated, and the draft Expectation Scale from Academics in Distance Education Scale, which was at the preliminary stage of the scale development process with the unknown factor structure, were considered in this comparison. It was determined that the consistency of the findings obtained from the EFA, and the ones obtained from the AISP provided strong evidence in ensuring the construct validity. The PAF and the AISP produced similar results in this research. The PAF results should be taken into consideration instead of the PCA, especially in scale development studies. It is recommended that the AISP and the PAF results be evaluated together for strong evidence in the investigation of construct validity in scale development studies.