Multilingual summative assessments face significant challenges due to tensions that exist between multiple language provision and comparability. Yet, conventional approaches for investigating comparability in multilingual assessments fail to accommodate assessments that comprise extended responses that target complex constructs. This article discusses a study that investigated whether bilingual examiners could apply comparative judgment (CJ) to pairs of Literature essays across different languages (English and Spanish). Preliminary findings suggest that whilst there are some cross-language standardization benefits, bilingual CJ faces validity challenges when different language cohorts approach target constructs differently. Existing definitions of "inter-subject" and "intra-subject comparability" are insufficient when multilingual subjects share fundamental constructs but differ in academic approaches. It is therefore proposed that an overarching classification of "intra-­disciplinary comparability" be introduced to frame discussions around multilingual assessments of this nature. Finally, it is recommended that further research into bilingual CJ be carried out to determine how the method can most effectively support investigations into multilingual assessment comparability.