This paper describes a framework for measuring student learning gains and engagement in a Computer Science 1 (CS 1) / Information Systems 1 (IS 1) course. The framework is designed for a CS1/IS1 course as it has been traditionally taught over the years as well as when it is taught using a new pedagogical approach with Web services. It enables the new approach to be compared with the traditional way of teaching the courses in terms of student self-assessment of learning gains, student assessment of their engagement with the subject matter, and researcher assessment of student learning gains as measured by performance on a researcher-designed examination. The framework includes a comprehensive pre-test and post-test for students in the control and treatment sections to complete, a common assessment exam module for all students to take, and a faculty survey for the instructors to complete. This enables the researchers to answer many questions regarding the effectiveness of the Web service approach, including "Do students using the Web service approach perform better in the common assessment exam module?" and "Do students and faculty members find the Web service approach more engaging?" Results from the first semester of a 3-year multi-university study are discussed. (Contains 2 tables and 1 figure.)