This study investigates the effectiveness of a Low-Cost Single-Channel BCI system in improving mathematical learning outcomes, self-efficacy, and alpha power in university students. Eighty participants were randomly assigned to either a BCI group receiving real-time neurofeedback based on alpha rhythms or a sham feedback group. Results showed that the BCI group had significantly higher mathematical performance, self-efficacy, and alpha power compared to the sham feedback group. Mathematics performance, alpha wave intensity, and self-efficacy showed significant positive correlations after training, indicating that neurofeedback training may have promoted their interaction and integration. These findings demonstrate the potential of BCI technology in enhancing mathematical learning outcomes and highlight the importance of considering pre-test performance and self-efficacy in predicting learning outcomes, with implications for personalized learning interventions and the integration of BCI technology in educational settings.