Test development is a crucial, yet difficult and time-consuming part of any educational system, and the task often falls all on teachers. Automatic item generation systems have recently drawn attention as they can reduce this burden and make test development more convenient. Such systems have been developed to generate items for vocabulary, grammar, reading, and writing. However, the literature lacks such a system for idioms. Informed by corpus linguistics, we aimed at developing a system that can generate items on idioms using the Corpus of Contemporary American English. The result was a system capable of generating two types of multiple-choice items, that is, fill-in-the-blank and definition. To validate the system output, we asked ten experts of Teaching English as a Foreign Language to evaluate 400 generated items for content issues. Also, a sample test of 40 items was administered to 110 upper-intermediate and advanced EFL learners, and then validated through the Rasch model. The results manifested that a large number of generated items were acceptable based on both experts' judgments and Rasch analysis. The study is an example of the application of corpus linguistics to language testing and a step toward automatic generation of items for assessment purposes.