The integration of augmented reality (AR) and game-based learning could increase interaction in educational games and provide more learning guidance in higher education courses. This study developed an AR educational board game that featured cognitive scaffolding, immediate feedback, and collaborative interaction. The study also analyzed and compared the competition- and collaboration-oriented mechanisms in the game. Participants of the study were 60 university students, who were divided into a competitive group and a collaborative group. We analyzed learners' psychological and cognitive differences in their learning effectiveness, motivation, flow, technology acceptance, learning anxiety, and gaming anxiety in the game. The results showed that both groups significantly improved in learning, and the competitive group had better learning effectiveness than the collaborative group. As for learning motivation, the competitive group improved in certain sub-dimensions, whereas the collaborative group improved in all sub-dimensions. Moreover, the collaborative group had higher scores in flow antecedent and perceived ease of use than the competitive group. The students had more smooth gaming experiences and perceived more clear feedback. Both groups showed lower gaming anxiety. These findings may have important implications for AR educational board game design.