Test Development with Performance Standards and Achievement Growth in Mind

State Assessment Program

This state assessment program includes assessments in reading, mathematics, and science in grades 3–8 and high school, as required by NCLB, as well as other content areas. This study focuses on the operational items in the mathematics tests at grades 3, 4, and 5. We selected only these grades (as opposed to using the entire grade span) because our main purpose is to evaluate the utility of using these coding frameworks to identify and illustrate coherence problems of proficiency standards (i.e., the cut scores and proficiency level descriptors) in typical state assessment programs that are intended to support inferences about achievement growth across grades. We did not set out to evaluate the coherence and articulation of inferences of achievement for the entire testing program.

This second edition of the state assessment program became operational in 2008, when within-grade operational scales were established. The mathematics tests contain only multiple choice items. The tests were scaled using the 3-parameter logistic IRT model with number correct scoring, independently, for each grade. Calibrations were based on items assigned to both operational and embedded field test slots (i.e., 55, 55, and 60 items in grades 3, 4, and 5) and more than 35,000 valid examinee response records in each grade. Small numbers of items were flagged for model misfit and differential item functioning. After expert review of flagged items, only one item in the grade 3 test was suppressed. Theta scales were converted to scale scores with target means and SDs of 150 and 10. After truncation of extreme theta scores (i.e., values beyond −4.0 and 4.0), final scale scores ranged between 88 and 183 for grade 3, 105 and 186 for grade 4, and 98 and 188 for grade 5. Performance standards were established on the scale score scales using the ID Matching method (Cizek & Bunch, 2007, chap. 11; Ferrara & Lewis, 2012).

Test items are closely aligned to content Competencies (i.e., the state's content strands) in each grade. Competencies are explicitly articulated across grade levels. In corresponding fashion, proficiency level descriptors are explicitly articulated across grade levels. The Proficient level descriptor explicitly represents attainment of grade level knowledge and skills, using verbs to indicate the cognitive processes expected of students (e.g., describe, analyze, compare, predict), as defined by the four Depths of Knowledge levels used in the Web Alignment Tool (see Webb, 2007). Table 1 contains the Proficient descriptors for grades 3–5 and the five Competencies. This state separates the proficiency level descriptors for the Competencies to aid interpretations of test scores and planning for instruction and to highlight the articulation of expectations for achievement growth across grades. Using the Number and Operations Competency to illustrate, students at the Proficient level in grade 3 are expected to “compose and decompose four-digit numbers,” 4th graders are expected to “compose and decompose five-digit numbers and decimal numbers,” and 5th graders to “compose and decompose seven-digit numbers and decimals through thousandths.” In this illustration, growth in achievement expectations across grades is defined by increases in the complexity of numbers, from four-digit numbers to five-digit and seven-digit numbers, and from “decimal numbers” to decimal numbers through the thousandths. Growth expectations are defined similarly in this Competency, including addition and subtraction, multiplication and division, and fractions and for the other four Competencies, as well.

In this testing program, knowledge and skills in the Number and Operations strand are emphasized over the other four Competencies in all grades and knowledge and skill demands increase slightly across grades in Geometry and Measurement.

Item writers were instructed to write items to align closely with the performance level descriptors for Basic, Proficient, and Advanced. (They did not target the Minimal level because more than enough existing items were located at this level.) Item writers were trained extensively on general principles for writing items, the mathematics Competencies, and the performance level descriptors. Cut scores had not yet been set for this assessment, so the training included guidelines for targeting each proficiency level descriptor rather than scale score ranges based on cut scores and anchor items for each level. Item writers were instructed to target nouns in the Basic, Proficient, and Advanced proficiency level descriptor in each grade that represent mathematics understanding, and verbs that represent cognitive skills and depth of knowledge. For example, the Geometry competency for Proficient performance in Table 1 requires 3rd graders to “explain and describe [the target verbs] the process [the target noun] of decomposing, composing and transforming polygons”; 4th graders are required to “identify [the target verb] transformations, reflections, and model translations [the target nouns]”; and 5th graders to “describe [the target verb] the characteristics of each type of transformation, reflections, and translations of two-dimensional figures [the target nouns].” The directions about targeting specific proficiency level descriptors were explicit about mathematics knowledge and skills, but did not include model items to emulate nor specified procedures to follow while targeting items at a specified proficiency level. For example, the training did not include discussing examples of items located in each proficiency level, as recommended in Hambleton and Jirka (2006, pp. 408–415), because cut scores for each level had not yet been established.

Nine to 23% of the items in the standard setting ordered item book that item writers targeted below the Proficient level actually were located below the Proficient cut score. Over 80% of the items in the ordered item book that item writers targeted at and above the Proficient level actually were located above the Proficient cut score in grades 3 and 4; only 17% in grade 5. Overall, the item writing targeting accuracy was 40% at grade 3 (i.e., 20 of 50 items were accurately targeted), 54% at grade 4 (29 of 54 items), and 14% at grade 5 (8 of 57). So, even though the item writer training lagged behind recommendations in the measurement research literature (an all-too-common observation), the results from the level of training that was provided represent a promising first attempt.

Content Standards

This state assessment program uses five familiar content strands to define its state content standards and guide curriculum and test development: Number and Operations, Algebra, Geometry, Measurement, and Data Analysis and Probability (see Table 1).

Cognitive Response Demands Coding Frameworks

Linguistic Demands Frameworks

Mathematics items like those in this study place linguistic demands on examinees as they process items, formulate an understanding of item demands, and select or generate responses. While reading load might be a more general cognitive-linguistic demand, linguistic demands are specific to the language used in items. In this study we use a framework from an item difficulty modeling study of grades 4, 7, and 10 mathematics items (Shaftel et al., 2006). The authors concluded that “ambiguous wording, item length, difficult vocabulary, syntactic complexity with longer sentences, and comparison problems may contribute to item difficulty” (p. 110) and that “difficult mathematics vocabulary had a consistent effect on performance for all students at all grades” (p. 105). These language features of mathematics items can be represented as five linguistic attributes: (a) number of ambiguous, slang, multiple meaning, and idiomatic words or phrases, such as change, feet, function, set, and (it) took; (b) number of words that may be unusual or difficult and specific to mathematics (i.e., vocabulary), such as complementary, coordinate, equation, likelihood, perimeter, quotient, reflection, and symmetry; (c) number of complex verbs (i.e., verb forms of three words or more), such as had been going, would have gone; (d) number of relative pronouns, specifically, that, who, whom, whose, which (sometimes), why; and (e) number of prepositional phrases, such as phrases beginning with about, above, after, by, during, except, for, from, inside, instead of, into, like, of, over, past, since, and through.

Item Response Demands Training and Coding Procedures

We examined the response demands of items that define Proficient performance. For each grade level assessment, we selected items with scale locations from the lowest obtainable scale score to the Proficient cut score plus the three items located just above the Proficient cut score. (We included three items above each cut score to account for the conditional standard error at the cut score.) In ordered item books used for item mapping standard setting methods, panelists make cut score decisions that indicate that students who are just barely Proficient (a) are likely to be able to respond successfully to these items (i.e., the Bookmark method), or (b) would be expected to respond successfully to these items because the item response demands match the demands described for the proficiency levels below Proficient (i.e., Item-Descriptor [ID] Matching). (See Cizek & Bunch, 2007, chaps. 10 and 11 for a description of these methods.) Examinees who just barely reach the Proficient level would be less likely to be able to respond to items located above the Proficient cut score because those items require knowledge and skills that those examinees are less likely to possess. Items in the analyses were selected based on their location on the theta scale and the cut score that defines Proficient. The numbers of items studied in grades 3, 4, and 5 are 35, 26, and 23, respectively.

After each of the frameworks was adapted for the study, we coded items one grade at a time. Three researchers coded items independently. A co-author of this paper adapted the frameworks for this study, served as a coder for all frameworks, and trained a second coder, also a coauthor, for the reading load, depth of knowledge, and question type frameworks. The trainer trained a different second coder (another co-author) for the mathematical complexity and linguistic frameworks. The trainer was a Ph.D. student in an educational psychology program, with a master's degree in educational psychology. The second coder for the reading load, depth of knowledge, and question type frameworks is a biology lab technician with a bachelor's degree in biology and who is training as a science item writer. The trainer and this second coder were interns at CTB/McGraw-Hill in summer 2008. The second coder, for mathematical complexity and the linguistic frameworks, is a psychometrician at CTB/McGraw-Hill. The trainer provided definitions and examples for each framework, explained in a training session how to code items, and practiced coding with the other coders.

The researchers coded all items independently, one framework at a time, and coded items in each grade, one grade at a time. After coding was completed, inter-rater agreement was calculated. Final codes for all items were determined in consensus meetings between the first and second coders. Table 2 displays exact agreement rates between two coders (before consensus discussions) for the cognitive demands and linguistic demands codes.

Exact rater agreement rates varied across frameworks and grades, but were generally high, with some notable exceptions. Agreement rates tend to be lower in grade 3, the first grade coded for each framework, and increase in grades 4 and 5 for reading load and depth of knowledge. The lowest cognitive demand agreement rate is for mathematical complexity for grade 5 (48%). The highest agreement rates are for depth of knowledge for grades 4 and 5 and question type for grade 4, all over 90%. Exact plus adjacent agreement rates for continuous coding frameworks (e.g., reading load) all are 90% or higher. We found higher disagreement rates in the linguistic codes. While exact agreement on complex verbs was reasonable, other rates were unacceptably low. In response, we used a consensus process to specify lists of countable ambiguous words, mathematics vocabulary, complex verbs, pronouns, and prepositional phrases to remove all ambiguity, and recoded all items. Using those lists, we counted instances of each category. (The lists are available from the authors.)

For reporting purposes, we collapsed some linguistic demands counts into ranges of frequencies, based on size and variability of the counts. Counts for ambiguous words, mathematics vocabulary, and prepositional phrases varied widely and are reported as 0, 1–2, 3–5, and greater than 5. Complex verbs are reported in their original counts, 0 or 1. Pronouns are reported in their original counts, 0, 1, or 2.

Psychometric Characteristics of the Tests and the Sets of Study Items

Table 3 indicates the range of the locations of the grades 3, 4, and 5 study items on the within-grade scales and related information. We display the RP50 item locations to be consistent with the final locations of the Proficient cut scores on the within grade score reporting scales. Table 3 also contains information about the complete test forms.

The Proficient cut scores on the theta scales (i.e., .08, −.06, and −.01 for grades 3, 4, and 5) are located relatively close to the average difficulty of the operational items in each grade level test. Standard-setting panelists recommended, during a moderation process, an across grade articulation pattern of decreasing impact data (i.e., percentages of examinees at/above the Proficient level) across grades: 55, 52, and 50% of examinees for grades 3, 4, and 5 (see the last line of Table 3). They adjusted cut scores to achieve this articulation goal.

Mathematics Content Demands

Table 4 indicates the numbers of items included in the study that correspond to each content strand in each grade and the total numbers of items in the full tests. The response demands of these items represent the mathematics knowledge and skills that examinees at the Proficient level would be expected to be able to display. In addition, they should be consistent with the knowledge and skills represented in descriptors for the Minimal, Basic, Proficient, and Advanced levels.

The differences in percentages of study items and items on the 2008 operational test forms in each mathematics competency are 5% or less for the grades 3 and 5 tests. Differences also are similarly small in Geometry, Measurement, and Data Analysis and Probability for grades 4 and 5 and Number and Operations for grade 5. Differences for the grade 4 test are approximately 9% lower for Number and Operations and 7% higher for Algebra. The study items that define Proficient cover the content sub-strands quite consistently with the total test blueprint.

Reading Load

The reading load for the grades 3, 4, and 5 tests is low (the reading load for 68% of all items is low), and lowest at grade 3, as expected. The reading load is high for almost no items (fewer than 10% of all items). Two thirds of these items demand that examinees read small amounts of text that is relevant only to the content area knowledge or skill that is required to respond; processing these items and selecting a response is obvious or explicitly stated; and stimulus materials (i.e., visual displays) in item stems are short, simple to process, and easy to understand. Almost no items (overall, fewer than 10%) require high reading load. Reading load increases at the moderate level at grade 5.

Depth of Knowledge

The depth of knowledge requirements that these items place on examinees are at the first two levels, recall and skill/concept. More than half of the items are recall items that require examinees to recall mathematics facts, definitions, and terms and apply simple procedures and rote algorithms. Another one-third of the items are skill/concept items that require examinees to decide how to approach problems and solve multi-step problems. Surprisingly, the percentage of recall items is lower at grade 3 than at grades 4 and 5 and the percentage of skill/concept items is higher at grade 3 than at grades 4 and 5.

Mathematical Complexity

The mathematical complexity of these items is primarily low. Over half of the items in all grades are low-complexity items, which are straightforward, single operation items that require examinees to recall a mathematical property or recognize a concept. Another third of the items are moderately complex, which requires examinees to make connections and display flexibility as they decide how to approach a problem. The complexity of these items does not increase across grades.

Question Type

The majority of the items (81%) are use/apply items, which require examinees to use information provided with the item in visual displays (e.g., graphs, numerical charts) or text, information generated as part of responding to the item (e.g., completing a calculation, using a mathematical formula to compute), or prior knowledge (e.g., an arithmetic fact, a computational procedure). Another 15% are analyze/categorize/hypothesize items, which require examinees to consider the components of a situation in order to group things according to identifiable features, observe and describe patterns in data (e.g., complete a numerical sequence), or compare and contrast (e.g., describe similarities or differences). The number of analyze/categorize/hypothesize items is lower on the grade 5 test than on the grades 3 and 4 tests. Answer and explain items, which require examinees to provide or select and explain an answer, typically are in constructed response format. Only three answer and explain type items appear among the study items.

Ambiguous Words

Approximately three quarters of these items contain 1–5 ambiguous or multiple meaning words or phrases (e.g., feet, [it] took…). Shaftel et al. (2006) found that ambiguous words influenced item difficulty at grades 4, 7, and 10 and especially for 4th graders. Grade 5 items contain 3–5 ambiguous words somewhat more often than do grades 3 and 4 items.

Mathematics Vocabulary

Almost two-thirds of the items in our study (64%) contain at least one or two mathematics vocabulary items that examinees are likely to find difficult (e.g., complementary, coordinate). The number of items with no mathematics vocabulary is lower at grade 5, where only one item contains more than five mathematics terms.

Complex Verbs

Overall, few of the studied items (13%) contain complex verbs (e.g., had been going) that would increase the syntactic complexity of the items.

Pronouns

Relative pronouns (e.g., that, which) can influence the syntactic complexity of mathematics items. Almost two thirds of these items (64%) contain one or two pronouns. The trend across grades actually is the opposite that might be expected. Ninety-one percent of the grade 3 items contain one or two pronouns, while only 30% of the grade 5 items contain one or two pronouns.

Prepositional Phrases

Similarly, prepositional phrases (e.g., phrases beginning with after, except, instead of) can influence the syntactic complexity of mathematics items. Almost one quarter of all items (23%) include more than five prepositional phrases; more than one third (36%) include 3–5; and another one third (35%) include 1–2 prepositional phrases. The percentage of grade 3 items with three or more prepositions (51%) is lower than at grades 4 and 5 (62 and 65%, respectively).

Results Related to Expectations for Achievement Growth

The developers of this state assessment program implemented a design and procedures to align the test development system with the test performance interpretation system. They articulated performance level descriptors across grades as a strategy to enable monitoring achievement growth. They trained item writers to write items whose empirical difficulty would be consistent—by design, if they were successful—with the performance levels they explicitly targeted. And standard-setting panelists articulated cut scores in the standard setting workshops by examining percentages of examinees at and above the Proficient level as part of the strategy for aligning the test development and performance interpretation systems. The item writers achieved some limited success in hitting proficiency level targets (40% and 54% at grades 3 and 4; only 14% at grade 5). Independent of the program developers’ efforts, in this study we evaluated the consistency of the hypothesized cognitive and linguistic demands that the items that define Proficient performance for these test forms place on examinees. In general, we found that cognitive and linguistic demands of the items that define Proficient performance for grades 3, 4, and 5 do not increase consistently, in a manner that reflects coherently increasing expectations for mathematics achievement.

We summarize the cognitive and linguistic demands of these mathematics items later. We expect cognitive demands that define Proficient performance to increase coherently across grade levels, and any linguistic demands that may be construct-relevant or irrelevant to increase across grade levels. We discuss response demands in the study items that increase consistently across grade levels and response demands that are disarticulated and undermine coherent expectations for demonstrating Proficient performance across grade levels. Table 7 summarizes the discussion and portrays patterns of cognitive and linguistic demands across grades.

Hypothesized Cognitive Demands

The hypothesized cognitive response demands of these items do not increase consistently across the grade level tests, as might be expected for items that represent Proficient performance at each grade. Specifically:

Hypothesized Vocabulary Demands

As with the cognitive demands, hypothesized vocabulary demands do not increase consistently across the grade level tests, in contrast to what might be expected. Specifically:

Hypothesized Influences of the Syntactic Complexity of These Mathematics Items

The hypothesized linguistic response demands related to syntactic complexity of these items do not increase consistently across the grade level tests, in contrast to what might be expected. Specifically:

Using the Coding Frameworks to Examine the Articulation of Response Demands Across Grade Levels

These hypothesized cognitive and linguistic demands are useful for examining across-grade articulation, specifically regarding assessing achievement growth across-grade. Specifically, testing program developers can examine item response demands and item scale locations during test forms assembly to select items that are more closely aligned with targeted performance level descriptors in each grade level. Doing so should improve parallelism of annual test forms within the same grade. And, because performance level descriptors define increasing expectations for Proficient performance across grades—to define growth expectations—doing so could improve the clarity and validity of inferences about achievement growth as examinees continue through their schooling careers.

Table 7 summarizes the response demands within and across grade levels and conclusions about the contributions of these response demands to coherence and disarticulation in describing Proficient performance across grade levels.

Table 7 indicates that the items in this study that define Proficient performance increase response demands in reasonable ways in terms of reading load, ambiguous words, complex verbs, and numbers of prepositional phrases. The items that define Proficient performance introduce disarticulation in defining growth in Proficient performance in terms of response demands related to depth of knowledge, mathematical complexity, question type, difficult mathematics terms, and numbers of relative pronouns.

Examining the Influence of Hypothesized Item Response Demands on Item Difficulty and Discrimination

Because targeting proficiency levels during item writing and test forms assembly is, in essence, predicting item difficulty, we also investigated the degree to which these item response demands frameworks predict item difficulty. We followed the tradition of item difficulty modeling studies (e.g., Gorin & Embretson, 2006) and used the item codings as independent variables to predict item p-values in linear regressions. In addition, because they represent the relationship of an item to the construct represented by all items in a test, we also used item discriminations (i.e., item-total test score correlations) as the dependent variable in separate regressions to conduct item discrimination modeling analyses. We conducted regression analyses using the study items only and all items (we coded items above the Proficient cut scores in order to conduct these analyses), and we conducted regressions within and across grades.

Item difficulty modeling aims to identify item features that are related to item cognitive processing and to estimate the impact of those features on item difficulty (Bejar, 1993; Bennett, 1999). Item difficulty modeling studies in a range of contexts have produced R² that are as low as .04 for mathematics items (e.g., Shaftel et al., 2006) and as high as .35 for listening comprehension and .58 for reading comprehension items on the paper–pencil version of the Test of English as a Foreign Language (TOEFL; see Huff, 2003, table 2.1). These results suggest that other factors determine the empirical difficulty of test items for an examinee population. (We hypothesize that opportunity to learn [e.g., Herman, Klein, & Abedi, 2000; McDonnell, 1995] is one of those factors.) Item discrimination modeling aims to evaluate the relationship between response demands of individual items and response demands of the achievement construct represented by the entire test. And, because item discriminations (i.e., item-total, point-biserial correlations) are an item quality indicator, it is valuable to know how well response demand codes predict discriminations.

Results

After conducting full model regressions (i.e., models that included all four cognitive and all five linguistic codings) to identify significant predictors, we estimated reduced models that retained only the significant predictors. Of the 16 full model regressions (i.e., difficulty and discrimination regressions × three within-grade and one across-grade regressions × study item versus all item regressions, or 2 × 4 × 2), six were significant. In all cases except one, the statistical power for significant models was greater than .92; power was low or moderate (i.e., .28 to .54 for all non-significant models). R² for significant full models ranged from .27 to .62, consistent with results from other studies, reported earlier.

The Role of Item Response Demands in Item Difficulty and Discrimination

We had hoped that modeling the relationship between item features that represent hypothesized response demands and item difficulty and discrimination statistics would illuminate achievement growth expectations and provide additional validation support for the item response demands coding frameworks. We found that reading load, ambiguous words, and mathematics vocabulary are consistently related to item difficulty and discrimination. These results provide empirical support for using these three coding frameworks for other item evaluations and research—when the goal is to predict item difficulty or discrimination—and item writer training. The results represent disconfirming evidence for the other coding frameworks, even though other studies demonstrate their usefulness for evaluating item response demands and their accuracy in predicting actual examinee cognitive processing observed in think-aloud protocols (Ferrara & Chen, 2011). These results suggest that training item writers on item response demands alone would provide only limited guidance in successfully targeting proficiency levels. And they suggest that Hambleton and Jirka's (2006) conclusion that anchor items and anchor item descriptions may contribute to making accurate item difficulty predictors.