Computational Thinking (CT), entailing both domain-general and domain-specific skills, is a competency fundamental to computing education and beyond. However, as a cross-domain competency, appropriate assessment design and method remain equivocal. Indeed, the majority of the existing assessments have a predominant focus on measuring programming proficiency and neglecting other contexts in which CT can also be manifested. To broaden the promotion and practice of CT, it is necessary to integrate diverse problem types and item formats using a competency-based assessment method to measure CT. Taking a psychometric approach, this article evaluates a novel computer-based assessment of CT competency, Computational Thinking Challenge. The assessment was administered to 119 British upper secondary school students (M = 16.11; SD = 1.19) with a range of prior programming experiences. Results from several reliability analyses, a convergent validity analysis, and a Rasch analysis, provided evidence to support the quality of the assessment. Taken together, the study demonstrated the feasibility to expand from traditional assessment methods to integrating multiple contexts, problem types, and item formats in measuring CT competency in a comprehensive manner.