Self-efficacy, or the beliefs people hold about their ability to succeed in certain pursuits, is a long-established construct. Self-efficacy for science communication distinguishes scientists who engage with the public and relates to scientists' attitudes about the public. As such, self-efficacy for public engagement has the potential to serve as a key indicator in the evaluation of scientist training and public outreach programs. To date, most evaluation scales have been designed for public audiences, rather than scientists. This study used think-aloud methods and Item Response Theory to develop a scale to measure scientists' Self-Efficacy for Public Engagement with Science. The results from this study support the use of a 13-item self-efficacy scale, and provide initial validation evidence to support its use with scientists who engage with the public. The findings are presented in relation to the continued study of public engagement through both research and evaluation.