The education measurement community has centered the idea of test fairness in both theory and practice. Yet, racial justice advocates in education research and practice (the racial justice community) have consistently critiqued that assessments are hardly fair and play a critical and outsized role in contributing to racial and social inequities in the educational system and larger society. We attempt to unpack two communities' different perspectives and different conclusions about fairness and assessments. We argue that these differences are rooted in the historical makeup of these communities, how they bound the issue of fairness, how they evaluate fairness, and how they consider the consequences of assessment both contemporaneously and historically. We conclude by contending that progress with respect to equity and justice will require an appreciation of and grappling with the nature of these differences and attention to boundary spanners who have long identified with both communities.