NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
ERIC Number: ED641886
Record Type: Non-Journal
Publication Date: 2021
Pages: 397
Abstractor: As Provided
ISBN: 979-8-7621-8815-9
ISSN: N/A
EISSN: N/A
Improving Multiple Matrix Sampling Designs for Context Questionnaires in ILSAs
Yan Zhou
ProQuest LLC, Ph.D. Dissertation, Indiana University
As the international large-scale assessments (ILSAs) become more popular, policy makers and education practitioners are interested in collecting as much student background information as possible to better understand the learning context of their students. To collect such abundant information, administrators need to develop a lot of questions. However, assigning each question to every participant would likely exhaust participants. This dissertation is dedicated to balancing the conflict between the substantial information to be collected and the limitedly available logistics and resources in practical operations of large-scale educational assessments. This dissertation developed and compared various multiple matrix sampling (MMS) designs and missing data methods to plan and handle missing context responses. Based on multiple matrix sampling designs, I divided a long context questionnaire (CQ) into multiple short blocks that were not overlapped, and then developed a variety of forms with each form consisting of several blocks. Each participant was randomly assigned a form. Afterward, I adopted four missing data methods, including dummy coding, multiple imputation with Markov chain Monte Carlo (MCMC) algorithm, multiple imputation with predictive mean matching (PMM) algorithm, and regularized iterative principal component analysis (iPCA) method, to complete the missing context responses planned by each MMS CQ design. I conducted two simulation studies and one empirical study to investigate the performance of the various CQ designs and missing data methods on recovering the true or empirical values of the student population/subpopulation plausible values (PVs), Cronbach's alpha coefficients of the constructs, root mean square error of approximation (RMSEA) values of the confirmatory factor analysis (CFA) models for the context constructs. I also examined the CQ designs and missing data methods' performance on recovering the true/empirical values of the correlations across the PVs and context variables. I used generalized effect sizes estimated from the ANOVA tests and Cohen's "d" values from the "post-hoc" tests to evaluate the performance of the various CQ designs and missing data methods on recovering the true values of the related statistics. I considered moderate to large effect sizes or Cohen's "d" values as substantial effects in this study. The results of this dissertation indicate that the balanced incomplete block designs (BIBDs) together with the MCMC multiple imputation method were most likely to perform the best on recovering the true values of the statistics of interest, compared to the other CQ designs and missing data methods. In addition, the within-construct BIBDs performed better than the between-construct BIBDs on recovering the true correlations between the PVs and context variables, and both designs performed similarly on recovering the true student population/subpopulation PVs. Considering the primary emphasis of ILSAs on precisely estimating students' PVs and their predictors, I recommend using within-construct BIBDs and the MCMC multiple imputation method to plan and handle missing context responses. The empirical study results further verified the simulation study results, which indicate that the within-construct BIBDs and MCMC multiple imputation method performed well on recovering the empirical values of the related statistics except for the correlations between the PVs and context variables. The simulation and empirical studies comprehensively explored the CQ designs and missing data methods. The produced results provide guidance to the administrators of practical international large-scale assessments on how various CQ designs and missing data methods affect the estimation of student achievement as well as the other statistics of notable significance in the context of global education. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]
ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml
Publication Type: Dissertations/Theses - Doctoral Dissertations
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A