Statistical mechanics has received limited attention in physics education research and remains a relatively underrepresented topic even in research on upper-division physics courses. The purpose of this study was to explore potential challenges that physics students encounter when they solve statistical mechanics problems in groups. Adopting a grounded approach, we video recorded and analyzed nine small student groups engaging in collaborative problem solving on the topic. The analysis involved iterative thematic coding, which gave rise to ten emergent categories of challenges. These were later divided into two broad groupings: "challenges with concepts" and "challenges with problem-solving strategies." In the first grouping, we list seven identified categories related to the concepts of macrostates and microstates, distinguishable and indistinguishable particles, temperature, entropy, energy, equilibrium, heat bath, the Boltzmann distribution, and the partition function. In the second grouping, we list three categories related to the inappropriate application of common relations, difficulty managing tensions between calculated results and qualitative reasoning, and coming up with definitions of new and inconsistent concepts. Some of our findings are supported by existing research on the topic, and others are previously unreported. Based on our findings, we propose that future research should investigate the relations between the identified challenges on one hand, and students' epistemological framing, reasoning, and use of multiple representations on the other. Finally, we suggest that teachers should spend time engaging students in a conceptual discussion of the central ideas of statistical mechanics, motivating the choice and pointing out limitations of commonly used toy models, and linking course content to real-world phenomena.