Reliability of patient responses in pay for performance schemes: analysis of national General Practitioner Patient Survey data in England

Dear editor,

We shelved this response written in October after the editorial of
Professor Salisbury, which set out the caveats in much clearer detail. (1)
However, this article is now cited in a publication issued to practices
and our observations might be of some use to some practices.

Roland et al demonstrated that the low response rate to the GP
Patient Survey was not responsible for the unexpected low achievement of
many practices in the QOF. (2) They demonstrate that the response rate is
a function of age profile of the practice.

However some basic problems are identified that were not immediately
obvious to us and we fear other uninitiated readers may overlook important
messages in the report. We will try to clarify the findings in the report
of Roland et al at our – simplistic – level for the benefit of other
readers.

If response rate is a function of the practice age profile and socio-
economic deprivation, could demographics be a source of inequity in the
performance scheme? Roland et al report a considerable response bias in
the survey relating to the age profile. They report that response bias is
not responsible for discrepancies per se, but the question remains could
the underlying cause, age profile and deprivation, cause systematic
inequity?

Low response rates have been recognised as a problem in previous
years and have been ‘corrected’ by over-sampling of the affected
practices. How this solution may work is demonstrated in tables 1 and 2.
This is a simplistic example of two imaginary identical practices with an
identical population, but a different age profile of the patients.

In the imaginary population, patients over 45 are easily pleased and
rate access as ‘good’ whilst patients under 45 are more demanding and rate
the same service as ‘poor’. The response rate of patients over 45 is 60%
and that of patients under 45 is 20% (average fictional response rate
40%).

Tables 1 and 2 show that age profile is the cause or the low
satisfaction in Practice A. Associated with this is a low response rate.
Over-sampling of the population for this practice to correct the low
number of responses does not alter the age profile of the practice and
therefore does not alter the low satisfaction rating (table 2, practice
A).

† Patients under 45 have a 20% response rate, patients over 45 a 60%
response rate.
‡ Patients under 45 rate care as poor, patients over 45 rate care as
good.

† Patients under 45 have a 20% response rate, patients over 45 a 60%
response rate.
‡ Patients under 45 rate care as poor, patients over 45 rate care as
good.

Mead et al recently discussed whether practices should deliver
services to meet the demands of their population or whether a correction
in the league tables should be applied. (3) In the current reward system
for patient access, practices offering identical access do not receive
identical rewards.

The message seems to be that practices with a younger and more
deprived population are systematically flagged as delivering lower quality
care and receive lower funding for patient experience.

We eagerly anticipate a more substantial analysis of these factors,
announced in a recent publication of Campbell et al, trusting this will
shed more light on these issues. (4) Until then, it is probably wise to
exercise a degree of caution when interpreting the GP Patient Surveys.

(1) Salisbury C. Using patient experience within pay for performance
programmes BMJ 2009;339:b4224
http://www.bmj.com/cgi/content/full/339/oct20_2/b4224

(2) Roland M , Elliott M, Lyratzopoulos L, Barbiere J, Parker R,
Smith P, Bower P, and Campbell J. Reliability of patient responses in pay
for performance schemes: analysis of national General Practitioner Patient
Survey data in England. BMJ 2009;339:b3851.
http://www.bmj.com/cgi/content/full/339/sep29_3/b3851

(3) Mead N and Roland M. Understanding why some ethnic minority
patients evaluate medical care more negatively than white patients. BMJ
2009;339:b3450. http://www.bmj.com/cgi/content/full/339/sep16_3/b3450

(4) Campbell J, Smith P, Nissen S, Bower P, Elliott M and Roland M.
The GP Patient Survey for use in primary care in the National Health
Service in the UK – development and psychometric Characteristics. BMC
Family Practice 2009, 10:57.
http://www.biomedcentral.com/content/pdf/1471-2296-10-57.pdf

Technical endnote:

One of the problems when analysing the results of the survey is the
multiple collinearity between the variables. We repeated the analysis of
Roland et al, using variables available to us, which differ slightly from
those available to the authors. We could reproduce the ‘volatility’ of the
response rate in multiple regression models (table 1 of Roland et al;
Predictors of patient response). We interpret this volatility as a
possible multicollinearity problem amongst the explanatory variables. When
there is multicollinearity, it is very difficult to identify the variable
that is truly responsible for a given result.

Statistics reveal relationships, but not causation. Causation is a
matter of interpretation and this is subjective. Looking at the variables
in table 2 of Roland et al; Comparison of weighted demographic
characteristics, we interpret that age profile could be partially or
mostly responsible for ethnicity, response rate, deprivation and
satisfaction with practice performance. The listed satisfaction variables
interrelate moderately to strongly, which raises the question how much
distinction is made by the public between the various aspects of care. It
could be that reported satisfaction with opening hours is ‘contaminated’
by having confidence in the doctor or by a generally higher satisfaction
with the practice. That would systematically link satisfaction to patient
demographic factors.

Our attention was drawn to this when we compared our practice to
neighbouring practices and noticed that there was a degree of discrepancy
between the proportion of patients reporting having to wait long for their
appointment after arriving in the office (16-30minutes) and the rating of
this (‘I have to wait a bit too long’). The proportion of patients rating
waiting as too long is related to a higher proportion of patients rating
all aspects of the service as poor. It would seem counterintuitive that
doctors that are rated as giving sufficient time are also the ones running
the surgery to schedule.

Table 3: The relationship (beta) between response rate and
satisfaction with access can be altered from +0.34 to +0.03
(insignificant), or even to –0.12 (significantly negative, p<_0.001 if="if" more="more" covariates="covariates" are="are" added="added" to="to" the="the" regression="regression" model="model" not="not" shown.="shown." age="age" profile="profile" and="and" deprivation="deprivation" explain="explain" _68="_68" of="of" variation="variation" in="in" response="response" rate.="rate." p="p"/>Competing interests:
None declared