Background: The concept of reliability is central to conducting--and understanding--research in Psychology. Students' understanding of concepts are strengthened when they learn by applying concepts. Objective: This article describes initial evidence of an activity for teaching reliability. Method: Students watched a short video of a staged bank robbery. They then tested the reliability of two different forms of police instructions for eyewitness recall. In so doing, they gained practice at calculating and interpreting inter-rater reliability and test-retest reliability. Results: Data collected from N = 191 students indicates that the exercise has a statistically significant positive effect on student understanding of and confidence about reliability concepts contributes to a roughly 20% increase in performance when comparing responses on pre- and post-exercise multiple choice questions. Conclusion: The activity gives students practice with the concept of reliability in a way that is engaging and memorable insofar as it demonstrates the implications of reliability for the real world.