Background: Face-identity processing declines with age. Few studies have examined whether face-identity processing abilities can be measured independently from general cognitive abilities in older adults (OA). This question has practical implications for the assessment of face-identity processing abilities in OA and theoretical implications for the notion of face processing as a specific ability. The present study examined the specificity of face memory and face matching abilities in OA aged 50+. Methods: Performance of younger adults (YA) and OA was measured on face tasks: Cambridge Face Memory Task (CFMT), the Glasgow Face Matching Task (GFMT), holistic processing; and tasks of general cognition: fluid intelligence, selective attention, and mental rotation. Data were analyzed using multiple regression models encompassing: (1) the CFMT/GFMT and measures of general cognition; and (2) all face processing tasks. Results: Across the two age groups, models encompassing all face tasks were significant and accounted for more variance in the data than models encompassing the CFMT/GFMT and measures of general cognition. General cognitive abilities accounted for 17% of variance for the GFMT (p < 0.01) and 3% for the CFMT (p > 0.05). Discussion: Our results suggest that face memory can be measured independently from general cognition using the CFMT in OA. Implications for the notion of a general face processing factor across the adult lifespan are discussed.