In today's era of large-scale data, academic institutions, businesses, and government agencies are increasingly faced with heterogeneous datasets. Consequently, there is a growing need to develop effective methods for extracting meaningful insights from this type of data. Quantile, expectile, and expected shortfall regression methods offer useful tools to deal with heteroscedasticity in data. Beyond traditional metrics like mean and variance, quantile, expectile, and superquantile (also known as the expected shortfall or conditional value-at-risk) analyses can capture nuanced effects of predictors on the extremes of the response distribution. The importance of these regression methods is evident in the numerous publications on their applications and theories in both statistical and scientific literature. Our specific contributions to this field of regression methods, which consider the tail of a distribution, are outlined below. First, we introduce a novel approach, termed robust expectile regression (retire), which tackles heteroscedasticity in high-dimensional data. We focus on iteratively reweighted l[subscript 1]-penalization which reduces the estimation bias from l[subscript 1]-penalization and leads to oracle properties. Theoretical analysis establishes its statistical properties in both low and high-dimensional regimes, demonstrating oracle properties and efficient convergence rates. Empirical evaluations showcase its superior performance compared to existing methods, offering a promising tool for robust and efficient estimation. Second, we propose a unified algorithm for penalized convolution smoothed quantile regression with various commonly used convex penalties, overcoming computational challenges inherent in fitting penalized quantile regression models in high-dimensional settings. This algorithm, implemented in an R-language package conquer, exhibits superior statistical accuracy and computational efficiency, demonstrated through extensive numerical studies and exemplified by a fused lasso additive quantile regression model applied to real-world happiness data. Third, we investigate the impact of distance learning on academic outcomes in STEM courses, particularly focusing on underserved and lower-performing students. Utilizing a large dataset spanning several years, the study employs expected shortfall regression to analyze disparities in academic outcomes between different student groups during distance and in-person learning. Findings underscore the challenges of online education, highlighting the effectiveness of targeted instructional interventions in narrowing academic disparities, thereby emphasizing the importance of equitable strategies in higher education. The first two chapters contribute novel methods for high-dimensional expectile and quantile regression, which represent alternatives to traditional least squares regression by allowing for the evaluation of the entire distribution of the response variable rather than solely focusing on its conditional mean. The third chapter extends the application of these advanced regression techniques by utilizing expected shortfall regression to investigate the impact of distance learning on academic outcomes in STEM courses. By applying sophisticated statistical methods to address real-world challenges in educational research, this chapter demonstrates the practical relevance and versatility of high-dimensional regression approaches. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]