Measures of academic engagement and disruptive behavior in students are strong predictors of social, academic, and lifelong career outcomes. Educators need a measurement tool that is feasible, accurate, and cost-effective in order to identify students who may be in need of additional behavioral support. Direct Behavior Ratings (DBRs) offer a solution, but are prone to rater effects, threatening the psychometric properties of the tool. Two methods are proposed for the mitigation of such rater effects: rater training and statistical adjustment. Previous studies have demonstrated mixed results regarding the effectiveness of rater training on DBRs, and no studies have investigated the impact of statistical adjustment on the mitigation of rater effects and DBR accuracy over time. In the current study, 23 participants were recruited to complete DBRs on video recordings of middle school students at two time points. One group received training while the other received no training. Additionally, many facet Rasch measurement (MFRM) was used to investigate the impact of statistical adjustment on DBR accuracy and rater effects. Results demonstrated that across both behaviors, neither rater training nor MFRM adjustment significantly improved DBR accuracy, and rater effects were stable over time. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]