Many medical schools have adopted multiple mini-interviews (MMI) as an advanced selection tool. MMIs are expensive and used to test only a few dozen candidates per day, making it infeasible to develop a different test version for each test administration. Therefore, some items are reused both within and across years. This study investigated the influence of item reuse and familiarity with test structure on scores for a high-stakes non-cognitive test administered every year to hundreds of medical school applicants. The results show no evidence that reusing items makes the test easier or less differentiating. Data are consistent across years, test versions and item types. Two additional aspects of familiarity with test structure and test items were examined: whether retaking the test has an effect and how candidates prepare for the test. It was found that retaking the test in and of itself does not result in higher scores. Notwithstanding the results, in the age of the Internet, test developers must assume that all items included in any particular test version will be fully exposed. Hence, in order to preserve test validity, reliability and fairness, it is important to rethink and carefully preplan test versions, as well as decide what information to divulge to candidates.