One approach to reducing persistent racial/ethnic achievement gaps is to tackle their social-psychological dimensions, including the negative consequences of stereotype threat and other identity threats in school. Initial research suggested that a particularly promising approach is brief self-affirmation writing exercises for 7th grade students; studies in individual schools reported this strategy reduced racial achievement gaps in grade point average by as much as 40%. However, subsequent evidence has been mixed. It is important to distinguish between two potential sources of variability in self-affirmation treatment effects: (1) explainable heterogeneity based on the delivery, population served, and context of these interventions; and (2) heterogeneity is fundamentally not explainable, such as would be expected from random differences due to sampling variability. The practical implications of these two explanations of effect variability are quite different. The key problem with differentiating these two explanations is that it is difficult to isolate relevant differences between different research efforts. Many of the theoretical moderators of treatment impacts--including features of implementation, student population, and school context--are unmeasured and confounded across research projects. The contribution of this paper is to consider heterogeneity in a setting that controls many of these factors by assessing differences in a within-study replication across two implementations of the intervention. The research was conducted in the regular 11 middle schools in Madison, Wisconsin in 2011-2014. The study includes 2,109 individuals (939 in cohort 1, and 1,170 in cohort 2) who were 7th grade students in 2011-2012 or 2012-13. Students were individually assigned to complete self-affirmation or comparison activities. Randomization was blocked so that treatment-comparison proportions were equal within each school. Student outcome data were collected from administrative records through the end of 8th grade (the year following the intervention). Key implications of this study include: (1) Analyses demonstrate the value of tests of moderators to assess theory about where, and ultimately how, specific interventions are successful; (2) Results point to the need to develop the theory and evidence about how and where self-affirmation works; and (3) Results imply practical limitations of self-affirmation as a tool to improve student performance and close achievement gaps. One figure and one table are appended. [SREE documents are structured abstracts of SREE conference symposium, panel, and paper or poster submissions.]