Trend scoring constructed response items (i.e. rescoring Time A responses at Time B) gives rise to two-way data that follow a product multinomial distribution rather than the multinomial distribution that is usually assumed. Recent work has shown that the difference in sampling model can have profound negative effects on statistics usually used to…

Teacher preparation programs are increasingly expected to use data on preservice teacher (PST) skills to drive program improvement and provide targeted supports. Observational ratings are especially vital, but also prone to measurement issues. Scores may be influenced by factors unrelated to PSTs' instructional skills, including rater standards.…

In this secondary analysis of a previously conducted systematic review, we analyze social validity assessments in intervention research for transition-age autistic youth. Social validity is concerned with the acceptability of the intervention goals, the acceptability and feasibility of the intervention procedures, and the perceived importance of…

Educational researchers often need to construct precise and reliable measurement scales of complex and varied representations such as participants' written work, videoed lesson segments and policy documents. Developing such scales using can be resource-intensive and time-consuming, and the outcomes are not always reliable. Here we present…

Even when raters demonstrate agreement in the use of a measure, limited score variability or violation of often-ignored statistical assumptions can result in lower reliability estimates than intuitively expected. This article uses data drawn from two randomized controlled trials of schema therapy and cognitive behavioral therapy for the treatment…

We describe best practices in reporting reliability estimates in counseling research with consideration to precision, generalization, and diverse populations. We provide a historical context to reporting reliability estimates, the limitations of past practices, and new methods to address reliability generalization. We highlight best practices…

This study aimed to provide meta-analytic reliability information of the Columbia-Suicide Severity Rating Scale (C-SSRS). We implemented systematic search procedures to 35 eligible studies (N = 23,247; Mage = 26.74 years) that reported reliability estimates. The synthesized average values of Cronbach's alpha were 0.88 (95% CI [0.85, 0.92]) for the…

Scoring in creativity research has been a central problem since creativity became an important issue in psychology and education in the 1950s. The current study examined the psychometric properties of 27 creativity indices derived from summed and averaged scores using 15 scoring methods. Participants included 2802 middle-school students. Data…

Background/Context: Confirmatory factor analysis (CFA) model is a commonly adopted framework to estimate and test a measurement model. Once a well-fitting final CFA model is selected, the selected model may be used to test structural relationships of the latent constructs with other variables, to construct a test with desired reliability and…

Background: Although the Depression Anxiety Stress Scales, Short Form (DASS-21) was developed for adults, its authors noted no compelling reasons to not use the measure with youth as young as 12 years. Despite increasingly widespread use with youth, psychometric evidence in support of its use with this population needs to be investigated to fully…

Comparative Judgement (CJ) is traditionally and primarily used to compare written texts. In this study we explored whether we could extend its use to comparing audio files. We used GCSE Music portfolios which contained a mix of audio recordings, musical scores and text documents. Fifteen judges completed two exercises: one comparing musical…

Background: Individuals with dementia have communication limitations resulting from cognitive impairments that define the syndrome. Whereas there are numerous cognitive assessments for individuals with dementia, there are far fewer communication assessments. The Profiling Communication Ability in Dementia (P-CAD) was developed to address this gap.…

This study aimed to examine the effects of reverse items created with different strategies on psychometric properties and respondents' scale scores. To this end, three versions of a 10-item scale in the research were developed: 10 positive items were integrated in the first form (Form-P) and five positive and five reverse items in the other two…

Objective: Brief Self-Control Scale (BSCS) is the most widely used instrument to assess self-control. The purpose of this reliability generalization meta-analysis was to examine the degree to which consistency reliability coefficients for scores on the BSCS generalize across age groups and languages. Method: We included studies using the BSCS and…

This study aims to investigate the validation of the Rapid Interactive Screening Test for Autism in Toddlers (RITA-T) in Turkish toddlers between 18 and 36 months of age. Children aged 18-36 months were referred to the department of child psychiatry for concerns of autism spectrum disorder, language disorder, developmental delay, and typically…