Many colleges and universities now require faculty search committees to use rubrics when evaluating faculty job candidates, as proponents believe these "decision-support tools" can reduce the impact of bias in candidate evaluation. That is, rubrics are intended to ensure that candidates are evaluated more fairly, which is then thought to contribute to the enhanced hiring of candidates from minoritized groups. However, there is scant -- and even contradictory -- evidence to support this claim. This study used a multiple case study methodology to explore how five faculty search committees used rubrics in candidate evaluation, and the extent to which using a rubric seemed to perpetuate or mitigate bias in committee decision-making. Results showed that the use of rubrics can improve searches by clarifying criteria, encouraging criteria use in evaluation, calibrating the application of criteria to evidence, and in some cases, bringing diversity, equity, and inclusion work (DEI) into consideration. However, search committees also created and implemented rubrics in ways that seem to perpetuate bias, undermine effectiveness, and potentially contribute to the hiring of fewer minoritized candidates. We conclude by providing stakeholders with practical recommendations on using rubrics and actualizing DEI in faculty hiring.