Playing board games is known to improve pre-school student numeracy and many board games are used to motivate and explore concepts in probability. This study investigates whether college students in liberal arts mathematics courses assigned to play the board game "Settlers of Catan" develop a stronger grasp of basic probability concepts than those assigned more traditional homework problems via an analysis of pre- and post-assessments and final exam questions.