The automation of creativity measurement is a promising avenue of development, given that classic creativity assessments face challenges such as resource-intensive expert judgments, subjective creativity ratings, and biases in people's self-reports. In this paper, we present a construct validation study for CollaboUse, a test developed to deliver fully automated creativity assessments for individuals and teams, based on the testing logic of the Alternative Uses Task. The CollaboUse can be taken online, with test-takers prompted to think up many, diverse and original ideas in three minutes. A construct validation study has been hosted on Amazon Mechanical Turk (MTurk), yielding valid data from N = 103 participants. A comprehensive set of metrics was obtained via automated assessment tools (CollaboUse, Immune Defense, C-Tracer), classic expert ratings of creative performance (Alternative Uses Task, and CollaboUse test responses judged by human experts), as well as self-reports (Creative Achievement Questionnaire, BIG5 Personality Inventory). Results indicate that automated tests as available with the CollaboUse deliver similar results to expert-judged creative performance.