Understanding the effects of second language writers' topic familiarity on their subsequent writing performance has been a subject of great interest to researchers in second language acquisition and TESOL. However, prior studies have tended to suffer from multiple methodological limitations (e.g., no interrater reliability, limited or no control over important variables), which may call into question their results. In this conceptual replication, we highlight some of these issues by revisiting an influential study by He and Shi (2012), which explored the effects of ESL students' topic familiarity on their writing performance as assessed by measures of text quality and complexity, accuracy, and fluency (CAF). Apart from highlighting the differences in our studies' designs (e.g., the CAF measures and statistical analyses used), we highlight the similarities and differences in our findings. Finally, we close with a call for increased replication research in this area.