Background/Context: School principals have long played a central role in managing building operations and supervising school employees. Over the past two decades, however, policy developments--including site-based management, external accountability measures, and teacher evaluation systems--have increased expectations that principals also improve school climate, instructional practices, and student outcomes. Mounting evidence highlights that principals may exert substantial influence on these outcomes. For instance, principals appear to have important impacts on students' test scores (Branch et al., 2012; Coelli and Green, 2012; Dhuey and Smith, 2014, 2018; Laing et al., 2016), disciplinary consequences (Bacher-Hicks et al., 2019; Sorensen et al., 2021), and attendance rates (Bartanen, 2020). Despite this evidence, however, there remain important methodological questions about how to best examine principals' contributions to student and teacher outcomes. Obstacles include mis-attributing persistent trends in school performance to principals (Chiang et al., 2016), mis-attributing principal effectiveness to principal-school match quality (Dhuey and Smith, 2018), or uncertainty about the substantive interpretation of principal effects due to instability in their estimated magnitude across various plausible statistical-modeling strategies (Grissom et al., 2015). Objectives: Reliable, valid and unbiased measures of principal effects are critical to human resource management policies for school leaders and to guide future research that measures the effect of efforts to improve principal practices. In this paper, we study whether extant measures accomplish these goals and propose novel strategies to model principal effects on student and teacher outcomes. Our core concerns center on three questions. First, to what extent are estimated principal effects on near-term student outcomes reliable and stable across estimation strategies? Second, what tradeoffs between measurement error and external generalizability are imposed when comparing principals to those who have served in the same schools or connected network of schools? Third, to what extent are estimates of principal effects on contemporaneous student outcomes biased by fixed school characteristics, by students sorting to principals, or by principals sorting to schools on different performance trajectories in non- random ways? Data and Sample: We draw on three distinct longitudinal data sources for our analyses that permit inferences across a diverse set of educational systems with unique policy and practice contexts. We leverage 13 years of statewide administrative data from Oregon and Tennessee and 19 years of data from New York City to estimate principals' effects on student outcomes for just shy of 19 million student-year observations. Analytic Strategy: The standard approach in the principal effects literature (e.g., Branch et al., 2012; Dhuey and Smith, 2014; Grissom et al., 2015; Chiang et al., 2016) is to leverage panel data with repeated student observations to estimate what Grissom and co-authors refer to as the Relative Within-School Effectiveness model of principal effects. The critical proposed source of identification in this model comes from the inclusion of school and principal fixed effects. Parameters on principal fixed effects can be interpreted as measures of relative within-school principal effectiveness. They compare lagged-performance-adjusted student achievement during the time when the principal is responsible for a particular school to the mean effectiveness of other principals leading the same school at other times. We explore four central issues related to the reliability and validity of principal effect estimates: annual measurement error, strategies for modeling fixed characteristics and prior achievement trajectories of schools, external generalizability, and sorting-induced bias. We first seek to understand whether principal effects are sensitive to different approaches to account for annual measurement error, as well as the extent to which principal effects differ by estimation strategy or across outcomes. Next, we explore the validity of principal effects. For the purposes of this paper, we define this term to reference two distinct concepts. First, we examine what assumptions about principal effects are embedded in different estimation strategies and whether these differences may encode bias into the estimates. In particular, we explore how to best model how persistent attributes of schools can be disentangled from principal effects and how typical principal effect analytic methods produce estimates that generalize to the full population of principals. Second, we explore the extent to which principal sorting between schools threatens the claims that principal effect models causally identify the contributions of principals to student and teacher outcomes. Results: In this proposal, we present preliminary findings from the Oregon setting. Our results suggest that principals have meaningful impacts on student learning and attendance outcomes (Table 1), though the magnitude of these impacts varies across estimation approach. In particularly, we note substantively meaningful differences in the magnitude of estimates that employ different approaches to account for annual measurement error (Figure 1). We also document empirical concerns about the validity and generalizability of principal effect estimates that threaten the interpretation of these findings. One such concern is that estimates of principal effects rely on comparisons across principals who have ever served in connected networks of schools. We show in Figure 2 that while the largest network in Oregon includes between one-third and one-half of all principals in the state, the smallest networks include as few as two schools. Another concern is that different approaches to model the annual contribution principals make to students' outcomes when they are responsible for the same group of students over multiple years return substantively different results (Table 2). Additionally, we find evidence that principals who enter schools which were higher-performing before they arrived have students who experience higher test score gains under their leadership Table 3. Conclusions: Our findings suggest that researchers should undertake more validation work to detect potential sources of instability and bias in principal effect estimates prior to their use for policy purposes. We hope that in our conference paper we will be able to suggest strategies to resolve some of these concerns.