Purpose: Digital technology's integration into education has transformed learning frameworks, necessitating the exploration of factors influencing students' engagement in digital informal settings. This study, grounded in self-determination theory (SDT), proposes a model comprising artificial intelligence (AI) competence, chatbot usage, perceived autonomy (PA), digital informal learning (DIL) and students' engagement. Design/methodology/approach: The study collected survey data from 409 participants at Saudi Arabian universities, ultimately using 387 valid responses for analysis. This dataset was subjected to a thorough examination to confirm the validity of our proposed model. To decipher the complex interactions within our model, we utilized partial least squares structural equation modeling (PLS-SEM). The study adopted a disjoint two-stage method to formulate a reflective-formative higher-order construct (HOC). Findings: The study's findings showed that cognitive learning (CL), metacognitive learning (MCL) and social and motivational learning (SML) are the essential components of DIL. Significantly, the study determined that AI competence, chatbot usage, PA and DIL markedly affect students' engagement. Moreover, the R[superscript 2] value of 0.592 for student engagement indicates the model's robustness in explaining 59.2% of the variance, highlighting its effectiveness in identifying key drivers of student engagement in DIL contexts. Originality/value: This research enhances understanding by detailing the intricate relationships among AI competence, chatbot usage, and students' engagement in informal digital learning. It extends SDT to emphasize intrinsic motivations and AI capabilities, introducing reflective-formative HOCs for comprehending educational intricacies. It provides practical strategies for enhancing AI abilities and chatbot use in education, promoting personalized, engaging and autonomous digital learning spaces, thereby advancing educational theory and practice.