In recent years, lifelong learning has gained prominence, necessitating a continuous commitment from learners to enhance their skills and knowledge. During the lifelong learning process, it is essential to precisely assess the cognitive states of lifelong learners, as this will provide a learning report and further support subsequent learning activities. In the literature, researchers have proposed various cognitive diagnosis models (CDMs) to estimate the cognitive states based on learners' responses. However, learners' responses are noisy for different reasons, including guessing, slipping, accidentally clicking, and network issues. Rashly fitting the CDMs with noisy responses would yield imprecise cognitive state estimation. To tackle this problem, we first unify all types of noise underlying learners' responses. Then, we propose a novel diffusion-based cognitive diagnosis framework (DiffCog) to extend existing CDMs and enhance their effectiveness and robustness. DiffCog does so by addressing the following two technical challenges in denoising: 1) the hard-to-track problem and high computational cost in discrete and sparse responses and 2) the unknown extent of noise underlying responses. Specifically, DiffCog tackles these technical challenges by: 1) introducing responses encoders to project responses to continuous cognitive states for case of adding easy-to-track noise and reducing computation cost and 2) incorporating a time extractor and a denoise module to trace the noisy cognitive states back to the noise-free ones in a personalized way. We conduct extensive and sufficient experiments on three real-world datasets, and the results demonstrate that our proposed DiffCog not only elevates the performance ceiling of existing CDMs but also enhances their robustness to noise.