Dementia patients may have language barriers and decreased comprehension ability. Their family caregivers can feel frustrated when communicating with them. Poor communication hinders family caregivers from obtaining accurate health information about patients, and may also increase their emotional burden, affecting patient care quality. The present study developed a virtual reality-based simulated communication training (VRSCT) system and applied it to a training course for family caregivers of dementia patients. It allowed family caregivers to simulate real-world situations in a VR environment, experience the daily communication barriers and stress with dementia patients, and apply their acquired knowledge and skills to solve related problems. This study used a randomised control experimental design with mixed analysis methods. A total of 63 family caregivers were recruited and randomly divided into the experimental group (N = 32) learning with the VRSCT system to interact with virtual dementia patients and practice communication skills, and the control group (N = 31) using the traditional role-playing method for practice. Quantitative data were analysed to determine participants' knowledge of dementia care, attitudes, communication confidence and skills. In addition, the qualitative method was used to analyse the participants' discussion records. The results showed that by using the VRSCT approach, participants significantly improved their knowledge of dementia care, attitudes, communication confidence and communication skills compared to the control group. In addition, participants reported that through the real-time feedback of the VRSCT system, they could recognise their previous incorrect communication approach. As a result, they adjusted their communication strategies and increased their self-confidence.