Using patient experience within pay for performance programmes

“Pay for performance” schemes are being introduced in several countries to incentivise improvements in the quality of health care. In the United Kingdom, the pay of general practitioners is partly based on their performance under the quality and outcomes framework,1 which includes indicators of the process and outcome of care and a limited number of measures of patients’ reported experiences of care. It is important that schemes to improve the quality of health care take account of patients’ priorities and experiences. In the linked study (doi:10.1136/bmj.b3851), Roland and colleagues assess response rates and non-response bias in a national survey of patient experience.2

Since 2007, national postal surveys have been conducted to assess patients’ views of NHS general practice. In 2009 a more extensive survey was conducted, which included questions on a range of topics known to be important to patients.3 The results of just two questions were linked to payment; both were related to access—the ability to get an urgent appointment and to book an appointment in advance. The questions were intended to measure patients’ actual experiences rather than their subjective satisfaction, because satisfaction is consistently related to sociodemographic characteristics, such as age and deprivation.4

The publication of the survey results has led to a drop in payment for many general practices.5 Some practices are challenging the reliability and validity of the results,6 on the basis of the low response rate to the survey (38%) and the small number of patients who responded. This survey and its consequences have important lessons for the assessment of patient experience within pay for performance schemes internationally.

Unsurprisingly, Roland and colleagues’ study found differences between the sociodemographic characteristics of respondents and non-respondents.2 Practices with lower response rates tended to have lower questionnaire scores, but after adjusting for patient sociodemographic characteristics this correlation largely disappeared. This led the authors to conclude that there was little evidence of systematic unfairness as a result of poor response rates or selective non-response bias.

However, the practice scores used to calculate payments are not adjusted for the sociodemographic characteristics of the practices’ patients. Roland and colleagues’ data suggest that practices that serve particular patient groups are more likely to have lower survey response rates and poorer scores, which might not reflect their actual performance. The paper does not specify which sociodemographic characteristics were important, but previous research suggests that age, deprivation, and ethnicity probably have the greatest effect.4

The small number of respondents may also be a problem. The number of patients sent questionnaires was intended to produce scores on the access questions within confidence intervals of plus or minus 7% for each practice.3 6 Because of the way in which payments are graded in relation to the score, with this level of imprecision an individual practice might receive anything from 15% to 85% of the maximum payment simply because of random sampling error.7 In fact, the published results show that for one of the access questions almost half of all practice scores had a confidence interval exceeding 7%, which introduces even greater variability.6

What are the implications for countries wishing to incorporate measures of patient experience within pay for performance schemes?

Firstly, although the NHS general practice survey asks questions about patients’ experiences rather than their satisfaction, responses seem to be influenced by patients’ sociodemographic characteristics. Surveys are a good way to measure subjective views, but if the aim is to measure system performance objectively, other methods may be more reliable. For example, waiting times can be measured using data extracted from electronic appointment systems and appointment availability assessed using simulated patients or independent audit.8

Secondly, if surveys are used, the sample size needs to be related to the payment mechanism. The narrow range in payment thresholds in the UK will lead to large year on year fluctuations in the payments due to individual practices because of sampling error, which undermines confidence in the pay for performance scheme. More precise estimates of practice scores are needed, based on much larger samples of patients.

Thirdly, consideration should be given to adjusting scores according to the sociodemographic characteristics of practice populations.9 We need more research to investigate whether the lower scores consistently reported by certain sectors of the population reflect less good care or different perceptions.10

Fourthly, it is important to consider the potential adverse consequences of incentives. The focus on access in the UK system is likely to encourage practices to prioritise this rather than encourage continuity of care and good interpersonal care, which are equally or more important for many people. Ironically, one of the current access targets (ability to book in advance) has been introduced because an earlier incentive scheme resulted in many practices restricting advance bookings.11

The most important finding from the recent survey is that most NHS general practices are providing a good service. Overall, 84% of patients could be seen with two days and 91% were satisfied with the care they received.5 The reduction in income that many general practices have experienced is mainly related to increased payment thresholds rather than problems with the survey, meaning that pay goes down even if performance remains the same. This may be a way of driving up quality, but in the UK it may also be a way of manipulating doctors’ pay in response to perceived overpayment in the past.12