Two studies were conducted to examine and compare the construct validity of scores on the Junior Metacognition Awareness Inventory (JMAI) and problem-solving interview protocols. Participants consisted of 183 middle and high school students attending a university summer program for academically talented youth. Study 1 results indicated that JMAI scores were internally consistent and yielded an interpretable two-factor structure after the elimination of several items; however, the scores were not significantly or meaningfully related to GPA or current and future mathematics achievement. In Study 2 (n = 30), JMAI scores did not predict students' metacognitive behaviors during mathematics problem-solving tasks. In contrast, students' metacognitive behaviors observed during problem solving were meaningfully related to mathematics achievement with medium to high effect sizes. Findings support the predictive validity of metacognition with regard to academic achievement when operationalized with problem-solving interviews, but call into question the criterion-related validity of JMAI scores.