The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal…

In this work, we explore the differences between proctored and unproctored Internet administration for a Basque language low-stakes test considering demographic factors such as age, gender, and knowledge level in the subject. To this aim, we have developed an ad hoc application that allows us to establish a set of filters and techniques that…

In psychological and survey research, the prevalence and serious consequences of careless responses from unmotivated participants are well known. In this study, we propose to iteratively detect careless responders and cleanse the data by removing their responses. The careless responders are detected using person-fit statistics. In two simulation…

Participation in web surveys via smartphones increased continuously in recent years. The reasons for this increase are a growing proportion of smartphone owners and an increase in mobile Internet access. However, research has shown that smartphone respondents are frequently distracted and/or multitasking, which might affect completion and response…

This paper aims to explore the response-order effects for rating questions presented in item-by-item and grid formats. It was hypothesized that the primacy effect occurs for both formats of questions, and that this effect is dependent on age, education, and type of device used for responding to questions. Two randomized experiments were conducted…

Purpose: In survey methodology, it is well-known that interviewers can have an impact on the registered answers. This paper aims to focus on one type of interviewer effect that arises from the differences between interviewers in the systematic effects of each interviewer on the answers. In the first case, the authors evaluate interviewer effects…

The item wording (or keying) effect is respondents' differential response style to positively and negatively worded items. Despite decades of research, the nature of the effect is still unclear. This article proposes a potential reason; namely, that the item wording effect is scale-specific, and thus findings are applicable only to a particular…

In order to enhance the quality of education, school self-evaluation (SSE) has become a key strategy in many educational systems. During an SSE process, schools describe and evaluate their own functioning, often by administering questionnaires among teachers. However, it is unknown to what extent SSE questionnaire results are distorted by…

In low-stakes assessments, some students may not reach the end of the test and leave some items unanswered due to various reasons (e.g., lack of test-taking motivation, poor time management, and test speededness). Not-reached items are often treated as incorrect or not-administered in the scoring process. However, when the proportion of…

The present study used process data from a computer-based problem-solving task as indications of behavioural level of test-taking effort, and explored how behavioural item-level effort related to overall test performance and self-reported effort. Variables were extracted from raw process data and clustered. Four distinct clusters were obtained and…

Purpose: Cross-cultural comparisons based on ordinal Likert-type rating scales have been threatened by response style which is systematic tendencies to respond to items regardless of the item content. So, this study aimed to investigate the effect of extreme response style and acquisance response style on TIMSS 2015 data. Method: The sample of…

Self-reported measures are an easy, time-efficient, and low-cost way to collect research data compared to other assessment methods. They are, however, characterized by several limitations regarding the quality and the clarity of the data they produce, especially when they are implemented in education. The main purpose of the current study was to…

In surveys, proportions of individuals self-reporting as bisexual and homosexual are likely to suffer from misreports or refusal to answer when directly questioned. For more reliable information, the randomized response technique is often used. Therefore, to reduce social desirability bias, a multi-level randomized response technique has been used…

The aim of this study is to determine students' opinions about open-ended question exam practice during 2018-2019 academic year for the following programs of Anadolu University Open Education System: Economy, Hospitality Management, Philosophy, History, Sociology, and Turkish Language and Literature. The study was designed as a quantitative study…

Short message service (SMS or text messaging) surveys can collect data quickly and inexpensively. However, SMS surveys have space constraints that pose difficulties to questionnaire designers: Questions can only be 160 characters or less, surveys must be short, and many respondents use phones with small screens. In this article, we compare SMS and…