ERIC Number: ED654460
Record Type: Non-Journal
Publication Date: 2023
Pages: 10
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Changing the Success Probability in Computerized Adaptive Testing: A Monte-Carlo Simultion on the Open Matrices Item Bank
International Society for Technology, Education, and Science, Paper presented at the International Conference on Research in Education and Science (ICRES) (Cappadocia, Turkey, May 18-21, 2023)
For efficiency, Computerized Adaptive Test (CAT) algorithm selects items with the maximum information, typically with a 50% probability of being answered correctly. However, examinees may not be satisfied if they only correctly answer 50% of the items. Researchers discovered that changing the item selection algorithms to choose easier items (i.e., success probability > 50%), albeit not optimum from a measurement efficiency standpoint, would provide a better experience. The current study aims to investigate the impact of changing the success probability on measurement efficiency. A Monte-Carlo simulation was performed on the Open Matrices Item Bank and simulated item bank. A total of 1500 examinees were generated. We modified the item selection algorithm with the expected success probability of 60%, 70%, and 80%. Each examinee was assigned to five item selection methods: maximum-information, random, p=0.6, p=0.7, and p=0.8. The results indicated that traditional CAT was 60-70% shorter than random item selection. Altering the success probability did not affect the estimation of the examinee's ability. Increasing the probability of success in CAT increased the number of items required to achieve specified levels of precision. Practical considerations on how to maximize the trade-off between examinees' experiences and measurement efficiency are mentioned in the discussion. [For the full proceedings, see ED654100.]
Descriptors: Success, Probability, Computer Assisted Testing, Adaptive Testing, Monte Carlo Methods, Matrices, Test Items, Item Banks, Algorithms, Item Response Theory, Test Wiseness
International Society for Technology, Education, and Science. 944 Maysey Drive, San Antonio, TX 78227. Tel: 515-294-1075; Fax: 515-294-1003; email: istesoffice@gmail.com; Web site: http://www.istes.org
Publication Type: Speeches/Meeting Papers; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A