Studies have shown that language assessment literacy (LAL) is important for language teachers since they make important classroom decisions to improve student learning based on their assessment. However, some studies have shown that teachers need more knowledge and skills in assessment. Teachers also seem unconfident in assessing their students and need proper training. In order to help them to be more assessment literate, an instrument is needed to identify their LAL levels and needs. However, AL/LAL instruments available to date were based on dated 1990 Standards that did not address the latest development in assessment. Those instruments were not developed based on a specific domain of interest or teachers' needs in their local assessment context. A new LAL instrument is needed, which should be developed based on sound principles of test development framework, on teachers' assessment needs or profiles, and on a specific domain interest. In addition, since language assessment is a complex construct, the instrument should be developed following the principles of scenario-based assessment, which some studies suggest might have the potential to measure complex constructs. This dissertation study is aimed at developing such an instrument. In order to achieve the aim of this dissertation study, a scenario-based LAL test (SBLALT) was developed following the principles of Evidence-Centered Design. The activities in the first step of Evidence-Centered Design, Domain Analysis, were distributing a LAL questionnaire to Indonesian EFL teachers, interviewing eight Indonesian EFL teachers, and reviewing related literature. The results of the exploratory factor analysis (EFA) of the 324 questionnaire responses were the six factors of LAL deemed important by Indonesian EFL teachers. In the next stage of ECD, Domain Modeling, the results of EFA and those of the literature review were organized into assessment arguments. This organized assessment argument was the input for the next step of ECD, the Conceptual Assessment Framework, where technical specifications for developing the test were elaborated. After getting feedback from five content experts of ELESP, five Ph.D. students of Applied Linguistics and Technology at Iowa State University, and five ELESP students who took the pencil and paper version of SBLALT, SBLALT was revised, and its final draft consists of four tasks of a test development process. Content analysis of SBLALT was conducted by distributing a short Likert-scale survey (0 = very poor to 5 = very good) asking five content experts of ELESP to judge the content of SBLALT, whether the test items match the proficiency level of ELESP students, align with the objectives and the course materials of a language assessment-related course offered in ELESP. The content experts of ELESP agree that the SBLALT test items are very good, and the test items align with the course's objectives and materials. Classical Test Theory was used to analyze the psychometric properties of SBLALT. One hundred ELESP students took the final (online) version of SBLALT. Three indices were computed to analyze the effectiveness of SBLALT at the test level: score dependability, confidence interval, and classification dependability. Three indices were computed to analyze the effectiveness of SBLALT at the item level: B-index*, item phi index*, and item facility*. To set the cut score to determine the master and non-master students, five ELESP lecturers were invited to determine the cut score through the standard-setting process following the Angoff method. The cut score for SBLALT was 75%. The score dependability of SBLALT was not satisfactory (0.64) but only marginally below the acceptable level of 0.70 for low-stakes, classroom-based assessments. The confidence interval of SBLALT was 0.07, which shows that the individual scores were not measured reliably by SBLALT. However, the classification dependability, the most important dependability measure for criterion-referenced measurements, was quite high (0.84). SBLAT was quite consistent in classifying students into masters or non-masters. From the item analysis of SBLALT, eight out of fifteen items have a recommended B-index* of [greater than or equal to] 0.20. This shows that those items did their job well in discriminating between the test takers who passed and failed the test. The rest of the test items (7 items) have a low positive B-index* ([less than or equal to] 0.20), which means that these items did not effectively discriminate between the test takers who passed and failed the test. SBLALT differs from other measures of AL/LAL because SBLALT was developed based on the latest developments in language assessment and on the specific needs of EFL teachers. SBLALT was also based on specific curriculum objectives, which means that SBLALT supports curriculum and instruction. It is developed based on a scenario that resembles the context the test-taker participants will likely encounter as future teachers in their teaching careers. The scenario also facilitated the scaffolded test development tasks, which enables teachers who use this test in their teaching to focus on specific knowledge, skills, and abilities their students still need to improve. This study contributes to the field of LAL by developing a formative, classroom-based language assessment literacy test. The psychometric properties of SBLALT suggest that SBLALT has the potential to be applied as a measure of LAL in the Indonesian EFL context, specifically in the Indonesian English teacher education context. Using scenario-based assessment design enables SBLALT to document what students have achieved, improve students' learning, and provide a useful learning experience for students. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]