Collaboration is a complex skill, comprised of multiple subskills, that is of growing interest to policy makers, educators and researchers. Several definitions and frameworks have been described in the literature to support assessment of collaboration; however, the inherent structure of the construct still needs better definition. In 2015, the Organisation for Economic Cooperation and Development, in their Programme for International Student Assessment assessed 15-year-old students' collaborative problem solving achievement, with the use of computer-simulated agents, aiming to address the lack of internationally comparable data in this field. This paper explores what the data from this assessment tell us about the skill, and how these data compare with data from two other assessments of collaboration. Analyses enable comment on the extent to which the three assessments are measuring the same construct, and the extent to which the construct can be covered using computer-based assessments. These investigations generate better understanding of this complex and innovative domain.