Background/Context: As students increasingly enroll in online postsecondary courses, instructors and students need new strategies to maintain engagement, motivation, and sense of belonging to have academic success, particularly for historically disadvantaged students. This study focuses on improving student success in online postsecondary STEM courses through improved technological applications and is situated in a larger collaborative initiative of federally funded research and capacity-building center, the Postsecondary Teaching with Technology Collaborative. As part of the initiative, instructors participated in rapid-cycle evaluations (RCE) to test one or more specific technology-based strategies in their courses to promote students' Self-Directed Learning (SDL) skills. Intervention/Program/Practice: The RCEs were conducted in three waves, and this paper focuses on the first two, which occurred in Fall 2022 and Spring 2023. Interventions were co-developed with faculty and were designed to be easily incorporated into the LMS systems used by faculty. During the first two waves, we tested two SDL-promoting strategies -- instructional videos on SDL skills and self-reflection prompts. Instructors chose which of these to implement or whether to implement them in combination. The videos were designed by a group of education researchers in collaboration with faculty at partner institutions to support students' sense of belonging, time management, and growth mindset. Each video follows a consistent structure with an overview of the content, an introduction to the focal SDL skill or mindset, 2-3 concrete strategies to develop it, additional resources, and a reflection activity for students to self-rate their focal skill or mindset and select and apply strategies that can help them further develop those skills and mindsets. The prompts are short-answer questions focused on goal setting, task planning, and reflection. Four groups of SDL prompts were provided to participating instructors, along with a suggested timeline of administering them weekly. Instructors were encouraged to adapt the provided prompts and adjust the timeline according to their course contexts. Setting: In the first two waves of the RCEs, researchers partnered with a total of 12 instructors who taught fully online or hybrid STEM courses across four broad-access community colleges. These institutions are geographically diverse, representing New England, the Pacific Northwest, and the Southern US. Population/Participants/Subjects: Students participating in the first two waves of the study are from four broad access institutions from across the United States. Demographic characteristics of students varied across data sources, with the administrative data including the most comprehensive sample. Among students included in the administrative data sample, 9 percent of students were Asian, 18 percent Black of African American, 25 percent were Hispanic, and 32 percent were White. About half (48 percent) were full time students, and 85 percent were employed during the semester, with 40 percent working 30 hours a week or more. All analyses are restricted to students who were 18 or older at the start of the semester. Purpose/Objective/Research question: This paper includes findings from the first two waves of this ongoing study on the effectiveness of technology-based interventions in improving students' SDL skills and will discuss the challenges and opportunities of researching interventions designed to improve students' SDL skills in a post-secondary setting. Each data source used in this evaluation provided insight into how best to leverage technology-based interventions in online STEM courses, but each had important limitations. Nonetheless, a holistic evaluation that incorporates these diverse data sources can provide evidence on the effectiveness of interventions aimed at improving students' SDL skills. The presentation will address the following research questions: 1. Adjusting for baseline differences in student characteristics, what are the impacts of these technology-based strategies on students' SDL skills and end-of-course grades? 2. Do descriptive analyses from high-frequences LMS data support these impact findings? 3. Do qualitative data support a conclusion of positive impacts on students SDL skills and end-of-course grades? Data Collection and Analysis: Surveys were administered to students both at the beginning of each semester -- prior to the start of intervention activities -- and at the end of the semester. Administrative records, including student demographics, prior achievement, and end-of-course grades were collected at the end of each semester. LMS log data capturing each "click" made by a student in the LMS system were collected at the end of the semester. LMS log data were codified to determine which clicks capture interaction with interventions, and actions measuring students' comprehension monitoring, time management, and other SDL processes. Qualitative data were collected through interviews with a purposeful sample of instructors and students. Research Design: Each participating instructor taught multiple sections of the same course. At the beginning of the semester, they chose one section for each course to receive the intervention and another to serve as a comparison group, and additionally chose which version of the intervention (video, prompt, or both combined) to implement. Analyses pooling all interventions indicate baseline differences in academic achievement and most SDL skills were less than 0.10 SD. Regression analyses were used to adjust for baseline differences. Findings: Estimates of the impact of technology-based strategies on students' end-of-semester grades and students' perceptions of SDL skills are imprecise and insignificant, but a Bayesian interpretation of impact findings indicates the probability of a positive impact on students' end-of-course grades is 82%, on comprehension monitoring 64 percent, and on self-reflection 78 percent. Findings from analyses of LMS data support these conclusions, indicating strong positive and statistically significant correlations between use of the technology-based strategies and students SDL skills. Qualitative analyses indicated that students meaningfully engaged with reflection questions and prompts, and they outlined concrete strategies for improving SDL skills and mindsets. Qualitative data also indicated that instructors were better informed about students' understanding after reviewing responses to prompts, indicating a possible channel through which the interventions may have improved student outcomes. Conclusions: Impact findings from this study indicate that technology-based interventions may have positive effects on students' SDL skills and academic achievement, though impact estimates were generally imprecise. These findings our supported by both correlational analyses of LMS data and qualitative analyses. These findings will be used to inform development and testing of additional technology-based strategies.