ERIC Number: EJ1403138
Record Type: Journal
Publication Date: 2023
Pages: 21
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: EISSN-1877-6345
Digital Measurement of Hands-On Performance? Ecological Validation of a Computer-Based Assessment of Automotive Repair Skills
Hartmann, Stefan; Güzel, Emre; Gschwendtner, Tobias
Empirical Research in Vocational Education and Training, v15 Article 15 2023
We investigated the ecological validity of performance measures from a computer-based assessment tool that utilises scripted video vignettes. The intended purpose of this tool is to assess the maintenance and repair skills of automotive technician apprentices, complementing traditional hands-on assessment formats from the German journeymen's exams. We hypothesise that the ability to correctly judge repair actions shown in videos is a good predictor of the ability to perform corresponding actions in hands-on scenarios. Apprentices in the third year of vocational training carried out repairs on real cars or car systems, while experts rated their performance. After this, they worked on our computer-based tests, which utilise videos of very similar repairs. The correlation between video judgement and hands-on performance was lower than expected for most repair actions as well as for overall scores, indicating insufficient ecological validity of the test score interpretations. However, the findings are promising for developing future tests, as the results for some repair actions indicate it is generally possible to develop ecologically valid video-based items focusing on hands-on skills. We discuss the results in the light of a validation framework that combines validity evidence from different sources for the same assessment tool. Finally, we hope our findings contribute to a broader discussion about the psychometric quality of exams.
Descriptors: Performance Based Assessment, Computer Assisted Testing, Auto Mechanics, Job Skills, Equipment Maintenance, Motor Vehicles, Vocational Education, Video Technology, Scores, Test Validity, Test Interpretation, Ecology, Experiential Learning, Psychometrics
Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://bibliotheek.ehb.be:2123/
Publication Type: Journal Articles; Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: N/A
Grant or Contract Numbers: N/A