Collaborative problem solving (CPS) is an important 21st-century skill for academic and career success, and as a result, there is increased interest among businesses and educational institutions in the assessment and development of CPS skills. CPS skills are difficult to measure using traditional forms of assessment, and that difficulty has led to the use of computer environments that allow individuals to interact in complex situations and capture all actions throughout the process. In the current paper, we describe the design of a collaborative online mathematics task and explore student perceptions of the task and the extent to which the task elicits CPS skills. Results revealed areas of difficulty for students and showed that the task mostly elicited skills associated with the social dimension of CPS. We describe modifications to the task to provide better opportunities for students to display a broader range of CPS skills. An upcoming larger scale pilot study will explore whether the redesigned task can elicit the less observed CPS skills.