College and university administrators, as well as faculty members, are more likely to take responsibility for student learning if they believe that the assessment data represent their students and suggest specific actions for improvement. This study examined whether it is feasible to develop scalelets (i.e., focused measures, usually consisting of…

This chapter serves as a guide for quantitative researchers who seek to approach their research questions critically.

Attachment insecurity and maladaptive coping are associated with depression in adolescence; however, it is unclear whether these links primarily reflect stable individual differences among teens (trait influences), experiential differences in their interactions with relationship partners (social influences) or both. In this study, teens (ages…

Using generalizability theory, this study examined both the rating variability and reliability of ESL students' writing in the provincial English examinations in Canada. Three years' data were used in order to complete the analyses and examine the stability of the results. The major research question that guided this study was: Are there any…

This paper discusses the benefits of using generalizabilty theory in lieu of classical test theory. Generalizability theory subsumes and extends the precepts of classical test theory by estimating the magnitude of multiple sources of measurement error and their interactions simultaneously in a single analysis. Since classical test theory examines…

Cross-validation, an economical method for assessing whether sample results will generalize, is discussed in this paper. Cross-validation is an invariance technique that uses two subsets of the data sample to derive discriminant function coefficients. The two sets of coefficients are then used with each data subset to derive discriminant function…

Where the accuracy of a measurement is important, whether for scientific or practical purposes, the investigator should evaluate how much random error affects the measurement. New research may not be necessary when a procedure has been studied enough to establish how much error it involves. But, with new measures, or measures being transferred…

Generalizability Theory can be used to assess reliability in the presence of multiple sources and different types of error. It provides a flexible alternative to Classical Theory and can handle estimation of interrater reliability with any number of raters. (SK)

A family of coefficients of relational agreement for numerical scales is proposed, generalizing to multiple judges the theory of F. E. Zegers and J. M. F. ten Berge (1985) of association coefficients for two variables, using the premise that choice of coefficient depends on scale type of the variables. (SLD)

Studied the appropriateness and implications of using a generalizability theory approach to estimating the reliability of scores from tests composed of testlets. Analyses of data from two national standardization samples suggest that manipulating the number of passages is a more productive way to obtain efficient measurement than manipulating the…

Provides a comprehensive and integrated treatment of both conditional absolute standard errors of measurement (SEM) and conditional relative SEMs from the perspective of generalizability theory. Illustrates the approach with examples from commercial standardized tests. Examples support the conclusion that both types of conditional SEMs tend to be…

Demonstrated the use of generalizability theory in analyzing existing faculty evaluation data. Three measurement conceptualizations representing different purposes of faculty evaluation were developed and variance components associated with these conceptualizations were estimated from an existing faculty evaluation using 30 teachers for each…

This study investigated the level of generalizability across a few high quality assessment tasks and the validity of measuring student writing ability using a limited number of essay tasks. More specifically, the research team explored how well writing prompts could measure student general writing ability and if student performance from one…

Possible integrated and independent tasks were pilot tested for the writing section of a new generation of the TOEFL[R] (Test of English as a Foreign Language[TM]). This study examines the impact of various rating designs and of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the…

In this article, we critically evaluate the expectations of assistant professors as perceived by junior and senior faculty (n=22) across seven disciplines at two universities. Our research shows that there are similarities and differences within Sociology departments, across disciplines, and across the teaching and research schools where we…