Testing language abilities is common in the context of migration. It has been observed that approximately 6.2 million adults in Germany are low literate and that approximately 47% of this group does not have German as their L1. Similar findings have been reported for other modern Western societies with compulsory schooling and a well-developed educational system. However, in most testing situations no exemptions are granted for low literate learners and the same tests are used irrespective of the level of reading proficiency. In this review we focus on brain imaging research showing that reading-related neural networks in the brain differ between (L1) low literate adults and adults with normal reading abilities. We argue that these differences in functional brain anatomy influence cognition in general and should form the basis for changes in the practice of granting exemptions in language testing involving low literate adults. Possible consequences for language assessment for different purposes are discussed. Furthermore, the reported influence of literacy on functional brain organization should be considered for decisions in the context of granting exemptions for low literates in language assessment.