Standardized testing and accountability are currently unavoidable components of Texas Public Education. Through years of push-back, parents and educators have demanded that Texas consider alternative testing options that would reduce the high-stakes testing burden on students and schools. In 2015, the State of Texas passed legislation requiring the Texas Education Agency (TEA) to undertake a study of authentic writing assessment. This paper draws on data from a larger qualitative study to illustrate the complexity of teacher decision-making in the assessment process, provides further consideration into the influences of scoring calibration such as teacher knowledge, and highlights the need for intentionally designed professional learning about scoring as a means to mitigate differences and ultimately improve inter-rater reliability.