Retrospective confidence refers to the phenomenological experience of the level of certainty that retrieved information is, in fact, correct. Retrospective confidence judgments are examined across a range of sub-disciplines in psychology from perception to memory research, and in education and legal applications. This paper focuses on retrospective confidence judgments directed at memory. Typically, retrospective confidence judgments are explained by direct-access models. Direct-access models postulate that people have direct access to the strength of the retrieved memory. In contrast, inferential models posit that people use accessible heuristic cues to determine their retrospective confidence judgments. This paper outlines existing models from both the direct-access approach and the inferential approach. I then present the outcomes of studies that support the need to include inferential models in any explanation of retrospective confidence judgments. These heuristics include cue and encoding fluency, retrieval fluency, retrieval of related information, vividness of the retrieval, and self-consistency. I then present an integrative model to account for how retrospective confidence judgments are made.