This study investigates variation in listeners' accuracy in accent identification of native and non-native speakers of English. Thirty native speakers of New Zealand (NZ) English completed a free identification task with stimuli extracted from naturalistic conversations of several speakers from three native and two non-native English language backgrounds (German, Korean, New Zealand English, Southern British English, and Standard American English). The listeners were found to be generally accurate at identifying native English varieties, most often confusing them with other native accents (average accuracy over 75%). They were much less accurate at identifying the provenance of German first language speakers (mean = 13%) and even less so of Korean ones (mean = 4.7%), despite both of these language groups being two of the largest ones in NZ. This difference is explained through a step-wise process of accent identification whereby the speaker is first categorised as a native or non-native speaker; their broad regional provenance is identified next, with more specific identifications coming last, with potential reliance on both linguistic and social knowledge.