Understanding how language modality (i.e., signed vs. spoken) affects second language outcomes in hearing adults is important both theoretically and pedagogically, as it can determine the specificity of second language (L2) theory and inform how best to teach a language that uses a new modality. The present study investigated which cognitive-linguistic skills predict successful L2 sign language acquisition. A group (n = 25) of adult hearing L2 learners of American Sign Language underwent a cognitive-linguistic test battery before and after one semester of sign language instruction. A number of cognitive-linguistic measures of verbal memory, phonetic categorization skills, and vocabulary knowledge were examined to determine whether they predicted proficiency in a multiple linear regression analysis. Results indicated that English vocabulary knowledge and phonetic categorization skills predicted both vocabulary growth and self-rated proficiency at the end of one semester of instruction. Memory skills did not significantly predict either proficiency measures. These results highlight how linguistic skills in the first language (L1) directly predict L2 learning outcomes regardless of differences in L1 and L2 language modalities.