Empirically supported multimedia learning (MML) principles [1] suggest effective ways to design instruction, generally for elements on the order of a graphic or an activity. We examined whether the positive impact of MML could be detected in larger instructional units from a MOOC. We coded instructional design (ID) features corresponding to MML…

This study investigates consistent and inconsistent respondents to mixed-worded questionnaire scales in large-scale assessments. Mixed-worded scales contain both positively and negatively worded items and are universally applied in different survey and content areas. Due to the changing wording, these scales require a more careful reading and…

This study is part of a larger effort to investigate linguistic features of assessment items that might explain differential performance of English learners on science assessments. An analysis of a large set of multiple-choice items suggested two features that held promise in reducing the performance gap: the presence of contrast words and the…

In this paper, we undertake a content analysis of mathematics assessment tasks to understand how often graphical representations are embedded within high-stakes national and international tests. A total of 274 items were analysed, consisting of 160 Grade 9 UN items, 88 Grade 8 TIMSS items, and 26 PISA items. Analysis showed that all items in the…

Two test items that examined high school students' beliefs of sample size for large populations using the context of opinion polls conducted prior to national and state elections were developed. A trial of the two items with 21 male and 33 female Year 9 students examined their naive understanding of sample size: over half of students chose a…

Two techniques for detecting differential item functioning (DIF) of test items are compared: (1) item response theory (IRT), using the three-parameter model; and (2) Mantel-Haenszel chi square techniques (MHCS). The steps necessary for identifying differentially functioning items are identified for both approaches. Using data from the 1988…

Though item bias statistics are widely recommended for use in test development and analysis, problems arise in their interpretation. This research evaluates logistic test models and computer simulation methods for providing a frame of reference for interpreting item bias statistics. Specifically, the intent was to produce simulated sampling…

Four item bias methods were studied. The methods compared include the Mantel-Haenszel statistic, the plot method, the route mean squared difference method, and the total area method; the latter two methods are based on item response theory. The test consisted of item responses of 451 male and 486 female ninth graders to 75 test items on the 1985…

The validity of logistic test models and computer simulation methods for generating sampling distributions of item bias statistics was evaluated under the hypothesis of no item bias. Test data from 937 ninth-grade students were used to develop 7 steps for applying computer-simulated baseline statistics in test development. (SLD)

The cognitive structure of an algebra test was defined and validated using the linear test model (LLTM) and quadratic assignment (QA) techniques. The LLTM is an extension of the Rasch model with a linear constraint that describes item difficulty in terms of the cognitive operations required to solve the item. The model permits the specification of…

Six item bias detection techniques (distractor analysis, the Mantel-Haenszel procedure, Angoff's delta plot, Angoff's modified delta plot, Stricker's partial correlation index, and direct comparison of item difficulties) were compared for their ability to identify biased items at three stages of test development. The techniques were applied to…