The main goal of this study was to illustrate and provide some direction for dealing with the complexities of propensity score matching within different multilevel contexts. Special attention is given to how procedures typically applied in a non-hierarchical setting may be modified to properly reduce the expected bias in the estimated treatment…

This presentation summarizes a pilot study conducted after the May 2011 administration of the AP Environmental Science Exam. The study used analytical methods based on scaled anchoring as input to a Performance Level Descriptor validation process that solicited systematic input from subject matter experts.

The Many-Facet Rasch (MFR) Model is traditionally used to evaluate the quality of ratings on constructed response assessments; however, it can also be used to evaluate the quality of judgments from panel-based standard setting procedures. The current study illustrates the use of the MFR Model by examining the quality of ratings obtained from a…

Presented at the Advanced Placement Annual Conference (APAC) in Las Vegas, NV in July 2013. This presentation reviews new research examining the AP® experience of Hispanic graduates over the past decade. Topics include an in-depth look at the AP Spanish Language and Culture gateway hypothesis and trends in family characteristics such as parent…

Presented at the Advanced Placement Annual Conference (APAC) in Lake Buena Vista, FL in July 2012. This presentation reviews concepts central to achieving equitable AP access and success for all willing and academically prepared students. We analyze trends in participation and performance by race/ethnicity from the AP Report to the Nation and…

In this session presented at the Middle States Regional Forum (February 2013), experts working with national data presented trends in higher education from a forward-looking perspective. Senior researchers from the College Board provided trends using SAT, AP, and Net Price Calculator data. The research director of the National Student…

Presented at the AP Annual Conference (APAC) in Las Vegas, NV in July 2013. This analysis takes a close look at first-generation Advanced Placement® (AP®) test takers to better understand the needs and challenges they face on their path to college. The analysis focuses on college-bound AP test takers who also took the SAT and examines a variety of…

There are many changes occurring in higher education, and often it is difficult to see trends nationally. In this presentation, experts working with national data present trends in higher education from a forward-looking perspective. Senior researchers from the College Board provide trends using College Board SAT and AP data, as well as new…

Presented at the Annual Meeting of National Council on Measurement in Education (NCME) in San Diego, CA in April 2009. This presentation provides an overview of ECD. In addition, the presentation describes the benefits of, as well as the challenges that were faced, in implementing ECD in the Advanced Placement Program.

Developing a cognitive model of task performance is an important and often overlooked phase in assessment design; failing to establish such a model can threaten the validity of the inferences made from the scores produced by an assessment (e.g., Leighton, 2004). Conducting think aloud interviews (TAIs), where students think aloud while completing…

On 9/13/12, the Workshop on Developing Assessments to Meet the Goals of the 2012 Framework for K-12 Science Education was held at the National Academies of Science. The workshop was organized and led by the NRC Committee on Developing Assessments of Science Proficiency in K-12 (co-chaired by James Pellegrino and Mark Wilson) and targeted to state…

Presented at the Annual Meeting of National Council on Measurement in Education (NCME) in San Diego, CA in April 2009. This presentation describes the methodology that was used with subject-matter experts (SMEs) to articulate the content and skills important in the domain, and then the iterative processes that were used to articulate the claims…

Presented at the College Board National Forum, October 26, 2011. Choosing a college major is challenging enough, without stopping to consider the impact it has on a student's college experience and career choice. To provide support during this major decision, participants in this session will develop strategies to facilitate students in making an…

Presented at the quarterly meeting of the New York Area SAS Users-Group in December 2008. In virtually any research context, there is likely to be opportunity to apply mixed-models--of which repeated-measures, multilevel, hierarchical and random- and fixed-effect models are specific types-- that improve upon their traditional linear or generalized…

For educational achievement tests that employ multiple-choice (MC) items and aim to reliably classify students into performance categories, it is critical to design MC items that are capable of discriminating student performance according to the stated achievement levels. This is accomplished, in part, by clearly understanding how item design…