Although careless respondents have wreaked havoc on research for decades, the prevalence and implications of these participants has likely increased due to many new methodological techniques currently in use. Across three studies, we examined the prevalence of careless responding in participants, several means of predicting careless respondents,…

Computer-based administration of large-scale assessments makes it possible to collect a rich set of information on test takers, through analysis of the log files recording interactions between the computer interface and the server. This report examines timing and engagement indicators from the Survey of Adult Skills, a product of the Programme for…

In undergraduate research settings, students are likely to encounter anomalous data, that is, data that do not meet their expectations. Most of the research that directly or indirectly captures the role of anomalous data in research settings uses post-hoc reflective interviews or surveys. These data collection approaches focus on recall of past…

We revisit the merits of the retrospective pretest-posttest (RPP) design for repeated-measures research. The underutilized RPP method asks respondents to rate survey items twice during the same posttest measurement occasion from two specific frames of reference: "now" and "then." Individuals first report their current attitudes…

Short message service (SMS or text messaging) surveys can collect data quickly and inexpensively. However, SMS surveys have space constraints that pose difficulties to questionnaire designers: Questions can only be 160 characters or less, surveys must be short, and many respondents use phones with small screens. In this article, we compare SMS and…

Routing examinees to modules based on their ability level is a very important aspect in computerized adaptive multistage testing. However, the presence of missing responses may complicate estimation of examinee ability, which may result in misrouting of individuals. Therefore, missing responses should be handled carefully. This study investigated…

This chapter examines the impact of declining student response rates on surveys administered at small- and medium-sized institutions. The potential for nonresponse bias and its effects are addressed.

The August 16, 2011, "Chronicle of Higher Education" article "Want Data? Ask Students. Again and Again" by Sara Lipka posits that in higher education there is a culture of oversurveying students and too often relying on surveys as the main, or only, way of assessing the impact of programs and services on student satisfaction and learning. Because…

This article addresses the issue of falling response rates in telephone surveys. To better understand and maintain respondent goodwill, concepts of psychological contract and respondent expectations are introduced and explored. Results of the qualitative study show that respondent expectations are not only socially contingent but also…

Studies reporting sociometric assessments based on nominations have been characterized by important methodological inconsistencies when conducted in the middle school context. The purpose of this study was to examine (1) the possibility of a response bias when participants are provided with a long roster sorted alphabetically, (2) the impact of…

This review presents an overview of selected articles on the leniency hypothesis: the idea that students give higher evaluations to instructors who grade more leniently. Such articles comprise a small subset of the voluminous research on student evaluations of teaching (SETs). In this diverse literature, research methods and aims have frequently…

This study investigated self-presentation bias in the self-reports of teachers about their practices in six topic areas in educational technology (Instructional Design, Assessment, Learner-Centered Instruction, Curriculum Alignment, Attitudes about Computers, and Use of Computers with Students). Subjects were 50 middle-school teachers. Data were…