We propose a method of generating teaching policies for use in intelligent tutoring systems (ITS) for concept learning tasks [1] , e.g., teaching students the meanings of words by showing images that exemplify their meanings la Rosetta Stone [2] and Duo Lingo [3] . The approach is grounded in control theory and capitalizes on recent work by [4] , [5] that frames the teaching problem as that of finding approximately optimal teaching policies for approximately optimal learners (AOTAOL). Our work expands on [4] , [5] in several ways: (1) We develop a novel student model in which the teacher's actions can partially eliminate hypotheses about the curriculum. (2) With our student model, inference can be conducted analytically rather than numerically, thus allowing computationally efficient planning to optimize learning. (3) We develop a reinforcement learning-based hierarchical control technique that allows the teaching policy to search through deeper learning trajectories. We demonstrate our approach in a novel ITS for foreign language learning similar to Rosetta Stone and show that the automatically generated AOTAOL teaching policy performs favorably compared to two hand-crafted teaching policies.