ERIC Number: ED663653
Record Type: Non-Journal
Publication Date: 2024-Sep-19
Pages: N/A
Abstractor: As Provided
ISBN: N/A
ISSN: N/A
EISSN: N/A
Why We Need Better Reports of Teacher Professional Development Interventions to Find out What Works Where
Adrie Visscher; Marta Pellegrini; Natasha Dmoshinskaia; Veerle van Luppen
Society for Research on Educational Effectiveness
Why we need better reports of teacher professional development interventions for finding out what works where: Worldwide, billions of dollars are invested yearly in the professional development of in-service teachers ($14 billion in the USA alone, as of 2014; Bill & Melinda Gates Foundation, 2014). This makes the question of what effects TPD has on student performance and under what conditions very salient. The effects of teacher professional development (TPD) interventions have been and likely will continue to be the subject of an enormous number of studies. However, the findings of an individual study will always be very context-dependent. Metaanalyses do have much potential for investigating what TPD programs work for what TPD goals (e.g., improving teacher content knowledge, classroom management), teachers, students and contexts (e.g., Basma & Savage, 2018; Blank & de las Alas, 2009; Didion et al., 2020; Egert et al., 2018; Korpershoek et al., 2014; Kraft et al., 2018; Sims et al., 2021). We recently conducted a meta-analysis of studies into TPD interventions (Visscher et al., under review) and observed that many primary TPD studies are poorly reported. We used the MUTOS framework (Aloe & Becker, 2009; Cronbach, 1982) to study all the factors that play a role in TPD intervention research in an integrated way: the research Methods used in primary studies (e.g., the study design), the Units studied (e.g., the grade levels), the TPD Treatments (the TPD program features), the study Outcomes (e.g., teacher and/or student effects) and the study Settings (e.g., country, student SES). However, basic information about each category of the MUTOS framework is too often omitted or reported ambiguously in primary TPD studies. In the worst cases, the publications about the TPD studies included in our meta-analysis did not tell much more than that "something of which the characteristics were unclear" was done to make teaching quality better and that it was (not) effective, which is like reporting that a COVID-19 vaccine, the precise formula for which is unknown, proved to be (in)effective. Missing and unclear information in publications about interventions is problematic, first of all when selecting the studies to be included in a meta-analysis based on study inclusion criteria. Contacting researchers/developers for extra information is very time-consuming and the researchers/developers may not have the information needed. If, despite all that effort, there is not enough information about studies to determine if they meet the inclusion criteria or not, this will likely mean that some high-quality studies are not included in the meta-analysis, and that we will not learn anything from what was found in those studies. Actually one could ask why we conduct costly, primary studies on TPD interventions if we do not report them in ways that make them suitable for meta-analyses, as the results of individual studies are so context-dependent (e.g., Kaplan et al., 2020), and thus do not by themselves provide evidence about what in general works when. The missing and unclear information is also a major problem for the moderator analyses that researchers may want to conduct across a set of included TPD studies, to find out which TPD interventions, implementations and context characteristics go together with TPD effectiveness. A final negative effect of unreported data about TPD interventions is that it is a barrier for practitioners in schools and districts when determining which TPD treatments may be applicable and effective in their own specific context, and for designers of TPD interventions when designing better TPD programs. A proposal for tackling the problem: We propose that guidelines be developed for reporting TPD programs, and that as many scientific journals publishing TPD articles and research organizations as possible commit to these guidelines. We as a community can benefit greatly from guidelines for how to report TPD programs. Ideally, representatives of scientific journals, research organizations, other professional organizations, and policymakers at various levels of educational systems (including schools and school districts, as they may have quite different reporting desires) will work together on developing such guidelines to make sure that interests of all parties are addressed. Categories to be reported: To give the reader an idea of what we have in mind, we present here a framework with five interconnected categories. Four categories describing TPDs influence the fifth category that presents the effects of the TPD programs on students (and possibly teachers). The four categories are: (1) TPD program features (including goals, methods, content and context), (2) TPD Implementation features (e.g., fidelity of implementation), (3) TPD Context features (e.g., teacher characteristics, school context), (4) Study and test features (e.g., research design and test alignment with the TPD content). The precise guidelines for what to report for each of those categories, and possibly some additional categories, is what we as a community should work on to reach a consensus. In addition to these categories, the guidelines could also include reporting of the costs of TPD interventions, as we need TPD interventions that are not only effective, but also affordable for schools and districts. Effective but extremely costly interventions are not likely to be implemented at scale. In our SREE presentation we will elaborate on the content of each of the framework categories and also present our ideas about how to proceed with the development of the reporting guidelines. If we succeed in reporting our research better we can develop better TPD theories, which can help us to design better TPD interventions and TPD policies and thereby improve the impact of TPD research on teaching quality and student learning.
Descriptors: Faculty Development, Intervention, Publications, Educational Research, Meta Analysis, Data Collection, Program Implementation, Teacher Characteristics, Research Design, Educational Quality, Teaching Methods, Learning Processes
Society for Research on Educational Effectiveness. 2040 Sheridan Road, Evanston, IL 60208. Tel: 202-495-0920; e-mail: contact@sree.org; Web site: https://www.sree.org/
Publication Type: Reports - Research
Education Level: N/A
Audience: N/A
Language: English
Sponsor: N/A
Authoring Institution: Society for Research on Educational Effectiveness (SREE)
Grant or Contract Numbers: N/A