The present study was conducted to characterize (a) the features of the thousands of primary reports synthesized in 47 reliability generalization (RG) measurement meta-analysis studies and (b) typical methodological practice within the RG literature to date. With respect to the treatment of score reliability in the literature, in an astounding…

Investigated how dissimilar in composition and variability samples inducting reliability coefficients from prior studies were from the cited prior samples from which coefficients were generalized. Results from 20 articles show that citing reliability coefficients from prior studies as the basis for concluding new scores are reliable is only…

Responds to criticisms of some "Educational and Psychological Measurement" policies and the reliability generalization meta-analytic methods of T. Vacha-Haase. Explores consequences of misunderstanding score reliability. (SLD)

The program evaluation standards approved by the American National Standards Institute (ANSI) in 1994 that deal with reliability and validity accurately represent contemporary views of the psychometric community with regard to reliability and validity. As such, these standards move the field forward. The ANSI standards recognize that reliability…

T. Vacha-Haase (1998) proposed her "reliability generalization" methodology to characterize (a) typical score reliability for a measure across studies, (b) the variability of score reliabilities, and (c) what measurement protocol features predict the variability in score reliabilities across administration. The present article provides…

The reliability and construct validity of scores on the Personal Preferences Self-Description Questionnaire (PPSDQ) (B. Thompson) were studied. Classical psychometric analysis of data from 641 participants (item and alpha analyses and both oblique and orthogonal exploratory factor analyses) indicate that the PPSDQ has reasonable properties. (SLD)

Given the potential value of reliability generalization (RG) studies in the development of cumulative psychometric knowledge, the purpose of this paper is to provide a tutorial on how to conduct such studies and to serve as a guide for researchers wishing to use this methodology. After some brief comments on classical test theory, the paper…

Response error refers to the tendency to respond to items based on the perceived social desirability or undesirability of given responses. Response error can be particularly problematic when all or most of the items on a measure are extremely attractive or unattractive. The present paper proposes a method of (a) distinguishing among preferences…

Illustrates a number of ways in which confidence intervals for reliability coefficients can be estimated. Suggests that authors who submit articles to "Educational and Psychological Measurement" report confidence intervals for reliability estimates whenever they report score reliabilities and that they note the interval estimation…

Investigates two aspects of research analyses in quantitative research studies reported in the 1996 issues of "Journal of Counseling & Development" (JCD). Acceptable methodological practice regarding significance testing and evaluation of score reliability has evolved considerably. Contemporary thinking on these issues is described; practice as…

Surveyed AERA members regarding their perceptions of: statistical issues and statistical significance testing; the general linear model; stepwise methods; score reliability; type I and II errors; sample size; statistical probabilities as exclusive measures of effect size; p values as direct measures of result value; and p values evaluating…

Asserts that editors should declare their expectations publicly and expose the rationale for editorial policies to public scrutiny. Supports effect size reporting and the reporting of score reliabilities. Argues against stepwise methods. Also discusses the interpretation of structure coefficients and the use of confidence intervals. (SLD)

A procedure for estimating reliability in a factor analytic context, when reliability of the extracted factors is not an emphasis, is identified. The procedure is an extension of Dressel's work and might be applied in attitude measurement. It assesses how homogeneous the weighted original item responses are, when they are scored for pattern…

Investigated measurement characteristics of the Multidimensional Health Locus of Control (MHLC) Scales when the scales are used with elementary school children. The MHLC scales were administered to 876 fourth, fifth, and sixth graders. Results from four analyses of data strongly support the reliability and factorial validity of the revised MHLC…

After presenting a general linear model as a framework for discussion, this paper reviews five methodology errors that occur in educational research: (1) the use of stepwise methods; (2) the failure to consider in result interpretation the context specificity of analytic weights (e.g., regression beta weights, factor pattern coefficients,…