The COVID-19 pandemic negatively affected the quality of data from educational testing programs. These data were previously used for many important purposes ranging from placing students in instructional programs to school accountability. In this article, we draw from the research design literature to point out the limitations inherent in "disrupted" educational testing data, suggest questions and criteria to be considered in evaluating the use of such data for decision making, and indicate how such data may be valid or invalid for specific purposes. Six criteria are proposed for evaluating the degree to which educational testing data are valid for specific decisions. These criteria suggest data from COVID-disrupted school years are not likely to be valid for accountability purposes, but may be valuable for making decisions at the individual student level. Thus, we encourage researchers and policy makers to focus on how decisions derived from such disrupted data affect children.