NotesFAQContact Us
Collection
Advanced
Search Tips
Back to results
Peer reviewed Peer reviewed
Direct linkDirect link
ERIC Number: ED658531
Record Type: Non-Journal
Publication Date: 2022-Sep-23
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Can Writing Samples of Applicants Predict Teacher Outcomes on the Job?
Yujia Liu; Emily K. Penner; Sabrina M. Solanki; Xuehan Zhou
Society for Research on Educational Effectiveness
Background: Identifying high-quality teachers at the point of hire can reduce later costs in the form of teacher attrition and minimize the attrition impact on school organizations and student learning. There have been many studies paying attention to the topic of improving the system for selecting and hiring teachers (Bruno & Strunk, 2019; Goldhaber et al., 2017; Jacob et al., 2018). During the educator hiring process, one low-cost and low-effort way to screen applicants is through their writing samples. The writing sample can provide a holistic picture of the applicant, such as their beliefs and values, pedagogy and teaching style/strategies, their knowledge, and their priorities. This can inform districts about unobservable teacher characteristics and the potential educational environment teachers will create. Prior studies show that common themes identified from applicants' writing samples are significantly related to their demographic characteristics and the school characteristics they applied (Penner et al., 2019). However, there is limited research linking text data from writing samples with teachers' application results and their future teaching outcomes. Although one recent study considered rating scores of writing samples (Bruno & Strunk, 2019), other dimensions of writing samples, including writing quality and common themes, remain to be examined. Research Questions: The present study using the educator application essays in a large school district in California, comparing different machine-learning techniques (e.g., indicators of writing quality and supervised and unsupervised text-analysis), seeks to understand teacher characteristics (such as which schools they apply to) and their future teaching outcomes (such as their retention and effectiveness). Particularly, this study asks two research questions: 1. What can we know about the writing quality and common themes from application essays? 2. Compared to the standard application data (e.g., demographic characteristics and work experience), does the text-analysis data provide additional explanatory power to identify the applicants' likelihood of being hired and their future teaching outcomes (e.g., retention and effectiveness)? Setting: This study uses applicant data obtained from the Human Resources Department of a large urban school district in California from March 2009 to October 2015. This school district serves a diverse student body and struggles to attract and retain educators. Participants and Data: The district received approximately 218,196 applications for various certificated positions, including classroom teachers, principals, therapists, and counselors from 17,066 unique individuals. Of these, 11,267 completed the required essays, which are included in our analytic sample. See Table 1 for the descriptive statistics of applicant characteristics by hired status. Practice All applicants to certificated positions in the district applied through a proprietary online interface. The district certificated application includes many elements that describe applicants' qualifications and experiences, such as applicants' prior teaching and work experience, highest degree and institution, and type of credential(s) they possess. In addition to many standard application elements, the district application asks applicants to respond to three short-answer essays detailing how they would address particular social issues and problems of practice that are relevant for teaching in the district's context. The three essays ask about applicants' perspectives on measuring their success, addressing the achievement gap, and working with misbehaved students. Research Design and Data Analysis: We measure several characteristics of writing quality, including sentiment, readability, lexical density, number of misspelled words, and text length (see Table 2 for details). We then apply the structural topic modeling (STM) method to determine the common themes of the applicant essays (Roberts et al., 2014). Our STM estimation models control for whether an applicant previously worked in the district, their total years of K-12 teaching experience, and credentials they obtained. Two human coders examine the STM outputs to label the topics and interpret the themes. To address the second research question, I run ordinary least squares models with fixed effects to test the relationship between the writing sample characteristics and the outcomes of interest. We include a set of applicant characteristics to control for differences in the evaluation standards or labor market opportunities. To allow for the possibility that teachers have systematically different outcomes in different years and in various job positions, we include year fixed effects and job position fixed effects. Standard errors are clustered at the teacher level. Findings: Controlling for specific years and positions, we observed significant differences in most writing quality indicators between teachers from different backgrounds, employment results, and teaching outcomes (see Table 3 for details). For example, compared to teachers who do not get hired, teachers who get hired on average write longer essays and have fewer spelling mistakes; the readability of their texts is significantly higher; and the sentiment of their essays tends to be more positive. When compared to white applicants, applicants of color, however, write shorter essays, and their essays, on average, tend to be less positive. We identify 55 themes for three applicant essays (see Table 4 for details). These results are robust with STM models using various K settings, different sub-samples, and with or without covariates. We observed a range of associations between essay themes and teacher characteristics and outcomes. For example, the theme 'Culturally and community embedded social justice' is more likely to be talked about by applicants of color compared to the white applicants; and it is positively associated with value-added scores in English Language Arts. Conclusions: This study is the first to tie a comprehensive measure of applicant writing samples with a holistic measure of teacher real world outcomes. Compared to studies with similar text-as-data methods, this study is the first to use multiple writing samples from an individual, which creates a rich text dataset for exploring latent classes of educators' beliefs. Priorities and desired teacher qualities might vary by districts. This text-as-data method can test whether recruiting based on those priorities helps teacher retention and student performance. Limitations We could not observe everything about hiring process, especially about who got interviews and who declined offers. We also observed that applicants put various effort in writing samples. Lastly, it is hard to draw causal conclusions from our data.
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Identifiers - Location: California
Grant or Contract Numbers: N/A