ERIC Number: EJ1367471
Record Type: Journal
Publication Date: 2023-Apr
Pages: 23
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0013-1644
EISSN: EISSN-1552-3888
Supervised Classes, Unsupervised Mixing Proportions: Detection of Bots in a Likert-Type Questionnaire
Ilagan, Michael John; Falk, Carl F.
Educational and Psychological Measurement, v83 n2 p217-239 Apr 2023
Administering Likert-type questionnaires to online samples risks contamination of the data by malicious computer-generated random responses, also known as bots. Although nonresponsivity indices (NRIs) such as person-total correlations or Mahalanobis distance have shown great promise to detect bots, universal cutoff values are elusive. An initial calibration sample constructed via stratified sampling of bots and humans--real or simulated under a measurement model--has been used to empirically choose cutoffs with a high nominal specificity. However, a high-specificity cutoff is less accurate when the target sample has a high contamination rate. In the present article, we propose the supervised classes, unsupervised mixing proportions (SCUMP) algorithm that chooses a cutoff to maximize accuracy. SCUMP uses a Gaussian mixture model to estimate, unsupervised, the contamination rate in the sample of interest. A simulation study found that, in the absence of model misspecification on the bots, our cutoffs maintained accuracy across varying contamination rates.
Descriptors: Likert Scales, Questionnaires, Artificial Intelligence, Identification, Computer Mediated Communication, Accuracy, Models, Item Response Theory, Classification
SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://bibliotheek.ehb.be:2993
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A