Test takers should have a voice in testing practices (Mathew, 2004). However, when incorporating their input, systematic processes to ensure the validity of testing practices must be followed. Such processes allow for test development to be a more democratic process (Shohamy, 2001), without sacrificing the value of the resulting inferences made from the test scores. This article describes a case study that incorporated the desires of test takers to change the procedures of a group oral discussion test in a university English as a Foreign Language program. A study was designed to determine the extent to which the proposed changes would threaten the validity of the testing process. Specifically, the procedures for the group oral were altered to investigate the effect of interlocutor familiarity. Students were randomly assigned to class-familiar (n?=?146) and class-unfamiliar (n?=?159) groups to identify to what extent group familiarity affected test takers' scores in the four assessed categories: pronunciation, fluency, lexis and grammar, and communication skills. For the two groups, no statistically significant difference in scores was found, and score reliability estimates were similar. The implications of the findings are addressed in terms of recommendations for using stakeholder input in the assessment design process.