Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England
BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b3851 (Published 29 September 2009) Cite this as: BMJ 2009;339:b3851
- Martin Roland, professor of health services research1,
- Marc Elliott, senior statistician2,
- Georgios Lyratzopoulos, clinical senior research associate1,
- Josephine Barbiere, research assistant1,
- Richard A Parker, research assistant1,
- Patten Smith, director of research methods3,
- Peter Bower, reader in health services research4,
- John Campbell, professor of general practice and primary care5
- 1Department of Public Health and Primary Care, University of Cambridge, Institute of Public Health, Cambridge CB2 0SR
- 2RAND Corporation, 1776 Main Street, Santa Monica, CA 90401-3208, USA
- 3Ipsos MORI, London SE1 1FY
- 4National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL
- 5Peninsula Medical School, Smeall Building, Exeter EX1 2LU
- Correspondence to: M Roland mr108{at}cam.ac.uk
- Accepted 16 September 2009
Abstract
Objective To assess the robustness of patient responses to a new national survey of patient experience as a basis for providing financial incentives to doctors.
Design Analysis of the representativeness of the respondents to the GP Patient Survey compared with those who were sampled (5.5 million patients registered with 8273 general practices in England in January 2009) and with the general population. Analysis of non-response bias looked at the relation between practice response rates and scores on the survey. Analysis of the reliability of the survey estimated the proportion of the variance of practice scores attributable to true differences between practices.
Results The overall response rate was 38.2% (2.2 million responses), which is comparable to that in surveys using similar methodology in the UK. Men, young adults, and people living in deprived areas were under-represented among respondents. However, for questions related to pay for performance, there was no systematic association between response rates and questionnaire scores. Two questions which triggered payments to general practitioners were reliable measures of practice performance, with average practice-level reliability coefficients of 93.2% and 95.0%. Less than 3% and 0.5% of practices had fewer than the number of responses required to achieve conventional reliability levels of 90% and 70%. A change to the payment formula in 2009 resulted in an increase in the average impact of random variation in patient scores on payments to general practitioners compared with payments made in 2007 and 2008.
Conclusions There is little evidence to support the concern of some general practitioners that low response rates and selective non-response bias have led to systematic unfairness in payments attached to questionnaire scores. The study raises issues relating to the validity and reliability of payments based on patient surveys and provides lessons for the UK and for other countries considering the use of patient experience as part of pay for performance schemes.
Footnotes
We acknowledge the assistance from a wider team involved in development of the survey at Ipsos MORI, including Beccy Maeso, Helen Rowley, Juliet Brown, Lisa Valade-DeMelo, and Sonja Nissen. We are also grateful to Guy Watkins for his description of the GP payment system.
Contributors. All authors were involved in designing the analyses, reviewing intermediate analyses, and contributing to the final paper. MR is the guarantor of the paper.
Funding. The study was funded with a grant from the Department of Health. The opinions expressed are those of the authors and not of the department. All authors had full access to the data and take responsibility for the accuracy of the data analysis. The Department of Health was not involved in the analysis of the data.
Competing interests: PS is an employee of Ipsos MORI, which developed and delivered the GP Patient Survey for the Department of Health. MR and JC act as academic advisers to Ipsos MORI for the survey.
Ethical approval: Not required.
Data sharing: The results of the GP Patient Survey are publicly available at www.gp-patient.co.uk/results
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.