ERIC Number: EJ1318379
Record Type: Journal
Publication Date: 2021-Oct
Pages: 21
Abstractor: As Provided
ISBN: N/A
ISSN: ISSN-0142-6001
EISSN: N/A
Human versus Computer Partner in the Paired Oral Discussion Test
Ockey, Gary J.; Chukharev-Hudilainen, Evgeny
Applied Linguistics, v42 n5 p924-944 Oct 2021
A challenge of large-scale oral communication assessments is to feasibly assess a broad construct that includes interactional competence. One possible approach in addressing this challenge is to use a spoken dialog system (SDS), with the computer acting as a peer to elicit a ratable speech sample. With this aim, an SDS was built and four trained human raters assessed the discourse elicited from 40 test takers that completed a paired oral task with both a human and a computer partner. The test takers were evaluated based on the analytic operational oral communication rating scales which included interactional competence, fluency, pronunciation, and grammar/vocabulary. Repeated-measures ANOVA indicated that fluency, pronunciation, and grammar and vocabulary were scored similarly across the two conditions, while interactional competence was scored substantially higher in the human partner condition. A g-study indicated that the computer partner was more reliable in assessing interactional competence, and rater questionnaire and interview data suggested the computer provided a more standardized assessment. Conversely, raters generally favored the human partner, in part because of its perceived authenticity and naturalness.
Descriptors: Oral Language, Grammar, Language Fluency, Language Tests, Communicative Competence (Languages), Evaluators, Task Analysis, Computer Assisted Testing, Artificial Intelligence, Interpersonal Communication, Scores, Comparative Analysis, Standardized Tests, Discussion, Discourse Analysis
Oxford University Press. Great Clarendon Street, Oxford, OX2 6DP, UK. Tel: +44-1865-353907; Fax: +44-1865-353485; e-mail: jnls.cust.serv@oxfordjournals.org; Web site: http://applij.oxfordjournals.org/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A