The design, implementation, and outcome of educational robots in special education have not been sufficiently examined in a systematic way. In particular, learner-based and contextual factors, as well as the essential roles played by various stakeholders have not been addressed when robots are used as a learning tool in special education. Therefore, a systematic review using Activity Theory was conducted to analyze 30 studies in robot-assisted special education. Content analysis of the studies reported relevant information with respect to each activity component -- (a) "subject (learners with disabilities), (b) "technology" (robots supported by instructional design), (c) "object" (target skills or behaviors), (d) "rules" (implementation procedure and performance measures), (e) "community" (learners with disabilities, special education professionals, and parents), (f) "division of labor" (among learners, professionals and parents), and (g) "outcome" (performance of target skills or behaviors). Furthermore, the study identified existing gaps from the robot-assisted special education studies (e.g., lack of parental engagement), challenges (e.g., difficulty with standardizing performance measures due to heterogeneity of learner profiles), and contradictions (e.g., opposing views among experts on the role of robots in social interactions). Finally, recommendations were made under each activity component. The study concluded that both general and domain-specific guidelines should be created for each disability category proposed in this review to assist practitioners who wish to use robots to assist special education.