School improvement depends, fundamentally, upon collective agency--a group capability to work productively together and solve problems. Unfortunately, many schools operate in contexts of adversity that can pose considerable challenges with developing collective agency. Schools serving high-poverty communities of color often face chronic resource shortages, difficulties to reach their students, and negative reputations. Research has shown how such experiences of adversity can invite destructive tendencies that interfere with collective agency--including defensiveness, learned helplessness, and fragmenting conflict. However, prevailing approaches to researching school improvement have obscured insight into how collective agency may develop in adverse contexts. To study this, this paper draws on over 70 hours of participant observation and more than 50 reflective conversations conducted over 1 year with a Californian middle school facing adversity. Drawing on literature about group development and work teams, the article traces interaction patterns in three work groups, including one I led. The study finds clear efforts to develop collective agency at times, but it is a fragile emergence. Across all groups, collective agency becomes enabled when initiative to address a problem combines with manageable tasks, simple solutions, and group affirmation. However, these processes do not enable groups to fully address the complex problems they face, leaving groups vulnerable to recurrent experiences of inefficacy and overwhelm that quash collective agency. The findings offer a new understanding of school improvement amid adversity as a struggle to improve at "the next level of work," calling for reforms designed to sustain a foundation of collective agency.