Vision classification tasks, a fundamental and transformative aspect of deep learning and computer vision, play a pivotal role in our ability to understand the visual world. Deep learning techniques have revolutionized the field, enabling unprecedented accuracy and efficiency in vision classification. However, deep learning models, especially supervised models, require large amounts of labeled data to learn effectively. The acquisition of large-scale datasets meets many difficulties considering the dynamics in real-world applications. Collecting and annotating data is a time-consuming and expensive process, which sometimes requires domain-specific expertise to provide a sufficient quantity of high-quality labeled data. Meanwhile, privacy and ethical concerns hinder data acquisition in certain domains, such as healthcare or finance. Learning with limited supervision addresses these challenges by developing techniques that allow models to learn and make predictions with only a partial or a small number of supervision. In this presentation, we will introduce our research, which encompasses several advancements within the domain of learning with limited supervision. Initially, we introduce a novel fine-tuning method tailored to enhance the efficiency of few-shot learning, particularly in cross-domain scenarios. Building upon this, we extend our comprehension of few-shot fine-tuning into the transductive setting. Here, we present innovative weighting techniques to harness the potential of unlabeled data during the testing phase. In addition, we confront the intricate balance between data quality and quantity when leveraging unlabeled training data in semi-supervised learning. To address this challenge, we introduce the SoftMatch method, which allows for the adaptive integration of unlabeled data during training. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://bibliotheek.ehb.be:2222/en-US/products/dissertations/individuals.shtml.]