A Framework for Policies and Practices to Improve Test Security Programs: Prevention, Detection, Investigation, and Resolution (PDIR)

Test security is not an end in itself: test security is important because we want to be able to make valid interpretations from test scores and take appropriate actions based on those interpretations. We can make valid interpretations and uses of test scores only if test data have integrity. So, test security is an important means to an important end.

In this article, I propose a comprehensive framework for thinking about, planning, implementing, and operating test security systems. The framework is intended for testing program managers and their vendors, though anyone involved in the testing process should be concerned about test security and may find the framework helpful. The framework has four parts: prevention, detection, investigation, and resolution (PDIR).1 The comprehensiveness of test security systems is important especially because it appears that, in practice, prevention and detection efforts often do not address all security risks, investigations are not always conducted rigorously (e.g., Judd, 2012) or at all, and test security violations may not be resolved fully (e.g., Office of the Inspector General, 2014; Toppo, 2013). Designing comprehensive and effective security systems requires systems thinking2 (e.g., Nichols, 2016). So, in addition to addressing the four parts of test security thinking, the framework also addresses all parties in the testing process before, during, and after test administrations. Anyone involved in the testing process can pose a threat to test security. Everyone involved in the testing process is responsible for protecting test security and minimizing risks of security violations. Finally, the PDIR framework implies and guides remediation of weaknesses to security threats and failures when security breaches are discovered.

The framework also identifies threats to test security, countermeasures, and available detection measures. I also summarize the best thinking from several sources on preventing test security violations and detecting violations that may have occurred. Recommendations on these two parts of the framework have appeared in multiple publications, especially in recent years, and in numerous conferences, including the annual Conference on Test Security (see https://cete.ku.edu/2016-conference-test-security). I do not propose frameworks for investigation and resolution. Little has been published and discussed publicly about investigation and resolution—and practice in these two areas is inadequate (Ferrara, 2014). Such frameworks should be developed by experts in professional investigation techniques; educational policy and law; and teacher certification, employment, and legal requirements. I describe some professional investigation principles and propose that education and testing managers, policy makers, and authorities collaborate to develop best practices for investigation and resolution, to ensure comprehensive frameworks for test security.

As part of preparing to write this article, I conducted a survey of state educational testing programs on their PDIR policies and practices. The three-minute, 15–selected response item survey was administered online, using Survey Monkey, for 2 weeks in February 2016.3 I sent the survey to 51 program directors; 13 responded, for a response rate of 25%. Because the survey responses were anonymous, I had no means of evaluating the diversity or representativeness of the responders. Survey results appear in the appendix; I refer to selected results in several sections of the article. Given the small sample of respondents, I use the results as anecdotes rather than generalizable results. Those survey responses are supplemented by results from the 2015 test security survey of the Association of Test Publishers (Association of Test Publishers [ATP], 2015). Test publishers and their vendors (e.g., testing center and online platform providers) responded to a lengthy online survey, for a 13.5% response rate. The majority of respondents were from certification, licensure, and vendor organizations (ATP, 2015, pp. 11–12).

Background

A test security system includes rules, guidelines, requirements, and procedures—including training—to protect security, avoid breeches, and enforce security. Rigorous test security systems by themselves are insufficient. Once a test is printed and distributed or released to examinees in technology-based environments, secure test material no longer is controlled by testing program managers and their vendors. At that point, responsibility for test security has shifted to testing site managers and test administrators—and, for that matter, examinees. So, producing data with integrity relies on an effective test security system and a local culture of professional ethics and general ethical behavior.4 While it is not necessary that local testing site managers and test administrators buy into the culture, it is necessary that they abide by it. The same applies to examinees as well, of course. A culture of professional ethics encourages professionally ethical thinking and behavior, reporting of irregularities, and cooperation with investigations when irregularities may have occurred (Ferrara, 2012).

Prevention

Preventing security violations is the first and perhaps most effective countermeasure to test security threats. Minimizing threats can be supported by clear guidelines and requirements for reducing those threats and avoiding security breaches, communicating about the guidelines and requirements, training all professional staff and examinees on the guidelines and requirements, and reinforcing and enforcing them. A strong and public declaration that test security violations will not be tolerated and will be reinforced by effective detection, investigation, and resolution actions is a good starting place. Even with the most rigorous prevention measures and efforts, test security violations always are a threat. So, a realistic goal is a combination of maximizing prevention, relying on rigorous detection and investigation procedures, and intervening forcefully when possible security violations arise. To my knowledge, no one has conducted test security practice efficacy studies. So, I base these recommendations on experience and communication with colleagues.

Security guidelines (as opposed to strict rules) are necessary because, inevitably, situations arise that require professional judgment where rules may not apply directly or explicitly (e.g., applying a no camera phones rule to camera pens; administering a test just outside of the testing window because of a significant testing site disruption at the end of the window). In other cases, decisions may be clear, simply by abiding by test security rules (e.g., by locking up secure test materials and guarding the chain of custody). Guidelines and requirements should address managing secure test materials and content at all times, conducting orderly test administrations, guidance on how to intervene when necessary, training test administrators and testing site staff, and ensuring that all security guidelines and requirements are implemented rigorously. Regular communication about the responsibilities of all roles in the testing process (see Table 1 above) to protect test security and behaviors that are prohibited is crucial for effective test security systems. Training the people who serve in those roles about their responsibilities and test security guidelines and requirements builds on that basis. As above, I am not aware of systematic research on the efficacy of test security training. I have observed in a range of testing programs that training can be cursory and superficial, that test administrators and their supervisors vary widely in how closely they appear to pay attention during training, and how rigorously they implement the guidelines and requirements in local testing sites. My observations indicate that, often, training involves a cursory review of test security guidelines in test administration preparation sessions. Often, little attention is devoted to threats to test security, how the people in roles in the testing process can avoid and counter those threats, and how their actions are part of a larger system of test security and data integrity. The judgment, rules, guidelines, and requirements necessary to protect test security suggest the need for test manuals that supplement test administration manuals and for rigorous training.

The main responsibilities of testing program managers and their vendors are to develop guidelines and requirements to protect test security, to communicate those guidelines and requirements to train test administrators and other local testing site staff, to monitor implementation, and to build and implement countermeasures for threats that seem likely for a testing program. Table 2 summarizes the main test security responsibilities of people serving in key roles in the testing process. Wollack and Case (2017) address protecting test security in the context of maintaining fair, or comparable, testing conditions for all examinees. A study of a single licensure examination indicates no deleterious effects on item and test form characteristics of publicly releasing a large number of items (Buckendahl & Gerrow, 2016), a strategy often considered as a way of undermining motivation to reconstruct or otherwise compromise test item content.

Detection

Detection of possible cheating has benefited from a surge of attention in the psychometric and testing communities, due to efforts of the US Department of Education (e.g., the Symposium on Data Integrity; see US Department of Education, 2013), Association of Test Publishers publications,6 Council of Chief State School Officers’ publications (e.g., Council of Chief State School Officers and Association of Test Publishers, 2013; Olson & Fremer, 2013), annual Conference on Test Security (see https://cete.ku.edu/2016-conference-test-security), and researcher publications like the Handbook of Test Security (Wollack & Fremer, 2013), Test Fraud: Statistical Detection and Methodology (Kingston & Clark, 2014), and the Handbook of Quantitative Methods for Detecting Cheating on Tests (Cizek & Wollack, 2017). Table 4 summarizes statistical and other methods for detecting possible test security breaches. The table addresses the roles and responsibilities in the testing process from Table 2 and test security threats posed in Table 1. Table 4 includes only those observational and statistical detection methods with logical promise or some empirical support for efficacy or effectiveness in detecting possible test security violations, where statistical methods have acceptable reliability and accuracy under specified conditions.

Before considering the contents of Table 4, it is important to remember that a statistical flag alone may not represent adequate evidence that a test security violation has occurred. For example, explanations for inordinate numbers of wrong-to-right answer changes include cheating (e.g., educators tampering with examinee responses, test administrators telling examinees to change answers) and innocent ones (e.g., distributing the wrong test form, erasing responses, and restarting with the correct test form). The same is true for verbal reports of possible security violations. For example, observing a test administrator providing an approved read aloud accommodation for students with disabilities could be mistaken for assisting examinees with responding to test items. As a general principle, no single piece of evidence is adequate to support a claim that a test security violation has occurred and that sanctions may be warranted. Concerns and single pieces of evidence about possible test security violations almost always require additional investigation and evidence.

Developing detection methods for the many threats to test security will benefit from continuation of current individual efforts plus organized, systematic research and development programs that address all threats, rather than limited types of threats in Table 4.

So detection methods research, development, and refinement are emerging, but the state of the art in detection is still quite limited. Detection methods are limited to selected response items, even though short constructed-response items, performance tasks (Ferrara, 1997), and essay prompts (Lane, 2013, chapter 5) are increasingly common in state testing programs. Similarly, I have not yet found publicized attention to widely used next-generation assessment approaches like technology-enabled items (e.g., drag-and-drop and hot spot items) and technology-enabled features like item sets associated with animated, sometimes manipulatable, simulations. And the array of assessment approaches encouraged in the Every Student Succeeds Act (ESSA; i.e., projects, portfolios, and locally designed formative assessments to create a summative test score; see http://www.ed.gov/essa?src = rn) will expand the need for prevention and detection methods. These limitations put current security practices in operational testing programs at risk, at least in educational testing. As things stand now, perhaps only half of state educational testing programs use statistical detection analyses (e.g., Bello & Toppo, 2011; Government Accountability Office, 2013).

Investigation

When an apparent test security violation is suspected or alleged8 as a result of detection analyses, it may be necessary to undertake an investigation. The goal of investigations is to gather evidence to support or refute allegations. Of course, the evidence may be insufficient to support allegations, even if suspicions linger. Because results of investigating possible test security violations may be tried in federal, state, or local administrative courts (e.g., for actions against a teacher's license), civil courts (e.g., infringement of copyrighted material; M. Croft, personal communication, July 6, 2016), and even criminal courts (e.g., as in the Atlanta Public Schools case), the evidence produced in an investigation must be able to withstand courtroom scrutiny and must follow rules of evidence. Investigation is a profession, just like education and psychometrics. It is a profession with principles of practice, practical and scientific methods, regulations and guidelines, prepractice training and continuing education, professional societies, journals, and licensure requirements.

Who conducts investigations for educational testing programs when suspicions, allegations, or evidence warrants it? In practice, it seems to be primarily principals, local assistant superintendents, and state assessment directors and staff. Three of the 13 states that responded to the survey for this article reported that the state department of education (only) conducts investigations, five reported that local school system staff (only) conduct investigations, and five reported that both state and local system staff conduct investigations. Two of the states also reported that they would hire an outside contractor to conduct investigations. Contrast that with other widely reported test security cases. The Atlanta School Board hired the auditing firm, KPMG, and the test security company, Caveon, to investigate widely publicized cheating in the Atlanta Public Schools (Kingston, 2013, p. 302).10 In addition, the District of Columbia Public Schools system hired Alvarez & Marsal, a “global professional services firm specializing in…performance improvement and business advisory services” (see http://www.alvarezandmarsal.com/) and Caveon (e.g., Gillum & Bello, 2011)11 to investigate allegations of wrong-to-right answer changing.

Most state and local educational administrators have not been trained in professional investigation techniques. In addition, the knowledge, skills, and abilities (KSAs) and job requirements for investigators seem incompatible with the KSAs and job requirements required of educational administrators. This is evident in the Occupational Information Network, O*Net, a free, online database sponsored by the U.S. Department of Labor/Employment and Training Administration, which contains standardized and occupation specific descriptors for 974 occupations in the United States.12 A search on the O*Net category Investigators yielded 184 occupations (on November 5, 2016; see http://www.onetonline.org/find/result?s=investigators&a=1). An O*Net search for Fraud Examiners, Investigators and Analysts (http://www.onetonline.org/link/summary/13-2099.04) and Education Administrators, Elementary and Secondary School (at http://www.onetonline.org/link/summary/11-9032.00) reveals some similarities. Both jobs require similar types of knowledge, skills, and abilities; for example, knowledge of human psychology and behavior, active listening and judgment and decision making skills, and “problem sensitivity… the ability to tell when something is wrong or is likely to go wrong.” However, required work activities reveal how different these jobs and their requirements are, as indicated in Table 5.

As Table 5 indicates, work activities required of investigators include gathering and presenting facts and evidence about and from people who are under suspicion. The focus is on finding credible evidence that supports or disconfirms suspicions about unprofessional, unethical, and even criminal behavior for the purpose of convicting or clearing those under suspicion. In contrast, tasks required of school administrators focus on helping, counseling, and collaborating with people to help them improve professional learning and performance. O*Net lists three work values for educational administrators and investigators, two that are common to both professions: independence and working conditions. Relationships is another work value for educators; “Occupations that satisfy this work value allow employees to provide service to others and work with coworkers in a friendly noncompetitive environment.” In contrast, another work value for investigators is achievement; “Occupations that satisfy this work value are results-oriented and allow employees to use their strongest abilities, giving them a feeling of accomplishment.” These values conform with stereotypes about kindly teachers and steely, objective detectives.

Should educators conduct investigations of possible test security violations? Clearly, some could be trained to be effective investigators. However, state and local educational administrators also face conflicts of interest. A conflict of interest is “a set of circumstances that creates a risk that professional judgment or actions regarding a primary interest will be unduly influenced by a secondary interest” (quoted and cited at https://en.wikipedia.org/wiki/Conflict_of_interest). Local school system administrators investigating possible test security violations in a school in that school system face a conflict of interest: On the one hand, their job as the investigator is to discover evidence that may support the suspected violation and protect the integrity of the school system's testing program and data; on the other hand, their job as a school system administrator is to protect the school system's reputation and avoid unfavorable publicity. Because this conflict of interest can be identified and defused before impropriety occurs, the appearance of a potential of conflict of interest should be avoided. Testing program vendors face a conflict of interest for similar reasons. This is not a new insight (e.g., see G. Cizek quoted in Kingston, 2013, p. 302); it is one that is often not acknowledged and acted upon. My view is that when a test administration irregularity is reported or suspected—one that appears not to be a security breach, such as an examinee discovered to have a cell phone during a testing session but with no indication of using it—a school system official could follow up. However, when evidence suggests the need for investigation of alleged test security violations, states or school systems should hire professionals to conduct those investigations—at least when the most damaging and egregious offenses are suspected. I recognize that state and local budgets may not be able to afford professional investigation services. Perhaps, states could form consortia to share the cost of a retainer fee for investigative service needs that may arise in the future.

And, in fact, that probably is the most common practice. In that case, the people assigned to conduct investigations should be trained in professional investigation techniques and should come from a situation that enables them to be free from any real or perceived conflict of interest in the outcome of an investigation. Or, perhaps, the USDE should create a fund to support states’ investigative service needs.

Evidence that exists about educational test security investigation practices around the country is not positive. For example, the Atlanta Journal-Constitution examined more than 130 cheating cases in several major cities with high-profile cheating cases and surveyed state education departments (Judd, 2012). Under the headline “School Cheating Thrives While Investigations Languish,” Judd cited the “haphazard manner in which many states and school districts handle reports of cheating on high-stakes achievement tests,” “investigating allegations of cheating remains a low priority in many states, despite high-profile scandals,” and that states badly underbudgeted for test security investigations even though Georgia spent $2.2 million in Atlanta and Dougherty County.

Resolution

Resolving a suspected, alleged, or confirmed test security violation involves determining to the extent possible, or failing to determine, that test security was violated, the extent of damage to data integrity and test score validity and usefulness, and who did or may have committed the violation(s).

While the sample of states in this audit is small and the time period is limited (the 2007–2008 through 2009–2010 school year administrations), most likely it represents current practice in the United States. In addition, three questions in the survey of state assessment programs conducted for this article are relevant to this discussion. Eleven of the 16 states responded yes to the question, “Does your state have legislation, state board of education regulations, or other rules that…protect test security or test data integrity?” However, only 7 of the 12 responding states answered yes that they have legislation, regulations, or rules to “guide sanctions if test security violations are detected and supported by evidence.” In response to the question “In your opinion, if a possible test security violation is detected and supported by sufficient evidence, how likely is it that sanctions on individuals, schools, or local school systems will be enforced?” only three states responded highly likely, seven responded somewhat likely, and two responded somewhat unlikely. Further, (a) a USA Today article cites a PBS Frontline broadcast report by education journalist John Merrow and suggests that then DC Schools Chancellor Michelle Rhee did not follow up on evidence of widespread erasures and wrong-to-right answer changes in the DC CAS program at multiple schools (Toppo, 2013); and (b) a recent Baltimore Sun article documented that two Maryland 10th grade students had posted test questions from a statewide English test on Twitter and that “any disciplinary actions against the students would be decided by the schools” (Bowie, 2015, p. 1), even though posting the material is a violation of protections for copyrighted test material.

In some cases, public opinion seems to oppose resolving some test security violations. For example, bloggers and tweeters supported a Columbia University professor's posting of three PARCC Grade 4 test questions. PARCC's strong efforts to have the posters remove the items were considered by some to be infringing on free speech rights to discuss test flaws (see Lewin, 2016). Publicized opinions in some instances opposed monitoring social media as a violation of privacy rather than an extension of classroom monitoring for cheating (e.g., Strauss, 2015). Such opposition may seem justified in conjunction with concerns about school systems forcing students to reveal their passwords to social media accounts and state laws (in Louisiana,, Maine, Michigan, Rhode Island, and Utah) give schools that access in order to monitor social media posts for possible exposure of secure test content (see Herold, 2015).

Other considerations come into focus when suspicions have been raised and investigations undertaken. Specifically, the reputation of anyone suspected or accused of a violation may suffer irreparable harm, and the public may come to distrust test data. Likewise, when sanctions are imposed or overturned (e.g., by a state board of education or law court), public confidence in the sanctioning body may suffer, and that body could be liable to retaliation through court cases. It is difficult and risky to try to do the right thing. Investigations conducted unprofessionally and weak evidence exacerbate the risk.

Consequences of not following through can be dire, however, either because the “failure of districts and states to adopt and enforce effective test security policies can result in litigation involving parents, employees, administrators, whistleblowers and others adversely affected by falsified scores” (Phillips, 2011, p. 6) or because it undermines the culture of professional ethics, test data integrity, and test score validity.

Resolving a suspected test security violation requires an authorized body to weigh evidence and make judgments about people who are responsible for the violation, the degree of responsibility in the violation, the extent of loss and damage to the testing program and to examinees and others, and to determine the types and levels of sanctions to impose on those who are responsible. The authorized body cannot act capriciously. Its decisions will be accepted by the public and be able to withstand legal challenge if those decisions are based on well publicized (a) policy, practices, guidelines, and procedures for prevention, detection, and investigation; and (b) sanctions that correspond to various types and degrees of violations. Sanctions calibrated to the severity of violations would not be unlike sentencing guidelines that criminal court judges may follow.

Aside from the evidence cited above, we do not know a great deal about the extent to which state testing programs follow through with sanctions when security violations are supported by evidence. We do know a fair amount about prescribed sanctions for test security violations. Croft (2014) found guidance in an examination of 38 states’ test security “state statutes and regulations” (p. 3), where statutes and regulations have the force of law (M. Croft, personal communication, June 6, 2016.) Croft found that (a) a greater number of states provide information on penalties for violating test security than they do on preventing and investigating violations; (b) educator penalties range from requiring additional professional development to suspension or revocation of certification; (c) 12 states penalize school districts in which violations occur; and (d) six states permit civil penalties for violations, while seven states’ statutes specify criminal penalties. It is not clear that the information on penalties for violations includes guidance on calibrating those penalties with the severity and impact of the violations. Respondents to the ATP survey, most of whom represent certification, licensure, and testing vendor organizations, reported that the most common actions in response to a security breach were invalidating test scores and conducting security integrity investigations (ATP, 2015, p. 79). Very few respondents reported more severe actions, such as personnel actions, referral to law enforcement, and pursuing civil legal actions.

I have written elsewhere that the U.S. Department of Education, which mandates state testing and accountability requirements through the Every Student Succeeds Act, and state departments of education have the responsibility and authority to make and enforce test security policy (Ferrara, 2014). In fact, they, local school systems, schools, and licensure and certification boards have an inherent interest in the integrity of test data. Guidance in resolving test security incidents is essentially nonexistent in previously cited, key publications on test security beyond the models of state statutes and regulations in Croft (2014).

Discussion and Recommendations

In the wake of the shocking cheating scandal in the Atlanta Public Schools (see, for example, Kingston, 2013), the Atlanta Journal-Constitution conducted analyses of reading and mathematics test scores for 69,000 schools nationwide. The analyses identified “suspicious test scores in roughly 200 school districts” (see Perry, 2012).13 I cannot attest to the rigor and appropriateness of the analysis methods and interpretation of these results. However, the pervasiveness of inordinate increases and decreases in school performance is alarming—and points to the need for rigorous, comprehensive test security systems in K-12 and licensure and certification testing programs.

The U.S. Department of Education's Office of the Inspector General found in March 2014 (Office of the Inspector General, 2014), among other things, that “The Department Could Strengthen Its Monitoring of States’ Test Results and Test Administration Procedures” (finding number 1) and “SEAs Could Strengthen Their Oversight of Statewide Test Administration and Security” (finding number 2). Finding number 2 implies that states should improve monitoring of test administrations, use of forensics, and resolution of security problems. The Department of Education agreed with these recommendations. Efforts in improving test security have come from an array of interested organizations: the U.S. Department itself, the Association of Testing Publishers, Council of Chief State School Officers, annual Conference on Test Security, individual states and school districts, testing contractors, and individual researchers through conference papers and journal articles.

The review here suggests that these disparate responses are making solid contributions to improving test security policies, practices, and perhaps outcomes—and suggests that a coordinated and more systematic approach would be more efficient and probably more effective. For example, rather than 51 separate state assessment programs (plus those of U.S. territories) working independently to develop test security policies and practices, a committee of state stakeholders could develop guidelines for PDIR for adaptation and implementation. The detection subcommittee could include statisticians and psychometricians who could conduct additional reviews of test security data forensics methods (e.g., in presentations from the 2014–2016 Conference), make recommendations for use for specific types of security threats, and develop a research agenda for additional development of forensics in the research community. The investigation subcommittee could include expert investigators, while the resolution subcommittee could include legal, legislation, and regulatory experts. Many state and other assessment programs do not have staff and financial resources to develop comprehensive security systems—and will remain at risk for significant security breaches until they do. Pooling resources—and getting significant support from the federal government, which requires state testing programs and demands trustworthy accountability data—is essential. Testing contractors and the measurement research community bear some responsibility here, as well. But the U.S. Department of Education and the state departments of education are the testing program sponsors who are most responsible for ensuring test security and data integrity, and are at risk of failing to provide adequate protections. The committee could be selected and convened by any one or a combination of the organizations that support the interests of educational, certification, and licensure testing programs: the Association of Test Publishers, Board on Testing and Assessment of the National Research Council, Council of Chief State School Officers, and the National Council on Measurement in Education. Other organizations could provide specialized guidance; for example, the National Association of State Boards of Education, on reasonable expectations and sanctions for educators.

The U.S. Department of Education has taken an appropriate first step. The revised peer review guidance for state assessment programs includes two critical elements for preparing assessment peer review submissions: critical elements 2.5—Test Security and 2.6—Systems for Protecting Data Integrity and Privacy (see U.S. Department of Education, 2015). Providing the means for states to collaborate and develop guidance for PDIR is the critical next step. Department of Education funding should support a coordinated effort among the testing programs and other professional communities with stakes in large-scale testing. The support for using federal education dollars is in the public record: A colloquy between Senators Barbara Mikulski and Lamar Alexander during hearings on December 8, 2015, on the Every Student Succeeds Act confirmed that states are provided the “flexibility to use [federal education] funds to preserve and maintain the integrity and validity of these important [state] assessments” (Assessment Security, 2015, p. S8469). The bipartisan, bicameral bill was signed into law by President Obama two days later.