The current study explores students' collaboration and problem solving (CPS) abilities using a human-to-agent (H-A) computer-based collaborative problem solving assessment. Five CPS assessment units with 76 conversation-based items were constructed using the PISA 2015 CPS framework. In the experiment, 53,855 ninth and tenth graders in Taiwan were recruited, and a multidimensional item response analysis was used to develop CPS scales and represent the students' collaboration and problem solving performance. The results show that the developed H-A approach is feasible for measuring students' CPS skills, and the CPS scales are also shown to be reliable. In addition, the students' CPS performance scores are further explored and discussed under the PISA CPS framework.