Use of standardised patients to assess quality of healthcare in Nairobi, Kenya: a pilot, cross-sectional study with international comparisons

Case development and presentation

We worked with the Advisory Committee of Kenyan doctors and nurses to develop tracer conditions, construct medically relevant checklists and assign correct case management protocols, with review from medical professionals. Generally healthy individuals were selected for case presentations so that physical examination would not lead providers to alternative diagnoses, and SP responses to relevant history questions were scripted such that appropriate questioning and examinations by a healthcare provider should have led to correct diagnoses for all patients.

The case presentations and correct treatments are detailed in table 1. The four cases were: (1) unstable angina, in which a 45-year-old man complains of being awoken suddenly by severe chest pain in the morning; (2) diarrhoea, in which the mother of an 18-month-old complains that her child has diarrhoea and requests medicines; (3) asthma, in which the patient complains of severe shortness of breath with specific triggers and, if asked, a history of similar attacks; and (4) tuberculosis (TB), in which the patient first complains of persistent cough, later revealing the duration and admitting to night sweats and weight loss.

The choice of these tracer conditions reflects their relevance in the Kenyan setting along with the availability of established medical protocols for triage, management and treatment. For example, among children aged 5 years or under, diarrhoea is the third most common cause of mortality and morbidity in Kenya.30 In Nairobi, rates of specific symptoms associated with asthma in children were as high as 25% — twice as frequent as in surrounding rural areas.31 TB is the fourth most lethal infectious disease in Kenya, with an incidence of 233 cases per 100 000 population and a high mortality rate of 20 deaths per 100 000 population; in addition, one in three patients with TB whose HIV status is known is HIV-positive, posing an especially challenging comorbidity profile.32 Finally, ischaemic heart disease is now the tenth leading cause of death in Kenya, just after malaria and TB, and may become an increasingly prevalent condition as the population ages.33

For asthma, diarrhoea and TB, the Kenyan Ministry of Health had already developed and disseminated its own diagnostic and treatment guidelines.34 For unstable angina, guidelines had not yet been established in Kenya, and the Advisory Committee consulted the European Society of Cardiology and the American College of Cardiology and American Heart Association Joint Guidelines. These SP cases had previously been developed and deployed in various studies in India and China, therefore allowing for comparability with this study. In all instances, we use a lenient definition of ‘correct management’, where a condition is judged to be correctly managed as long as the necessary management was undertaken — regardless of whether other unnecessary medicines such as antibiotics were given as well. As in samples from other settings, adopting a more strict definition of correct management that accepts only correct management without any unnecessary medicines produces a proportion of ‘fully correct management’ near zero.

Together with the Advisory Committee, SP scripts were redeveloped and contextualised for the Kenyan setting, including adjustments to both the patient’s personal backgrounds and the vernacular manner of speaking. Standardised Swahili translations of scripts described symptoms and responses as patients themselves would express them. For example, when asked to describe the chest pain, an angina SP would say, “It felt a little heavy, like something was sitting on my chest.” The SPs themselves recommended revisions as broad as the occupation of the patient and as focused as the patient’s physical presentation, such as that tuberculosis SPs should wear loose clothing to suggest weight loss. These recommendations were agreed on as part of the final phase of development, in which SPs presented their cases to Advisory Committee members and doctors for review.

SP cohort selection and training

From an initial list of 50 potential candidates recruited with the support of a local survey firm, 26 candidates were invited to attend 150 hours of training over 4 weeks after an extensive screening based on age, gender, education, work experience and health history. During training, SPs memorised medical details, symptoms and risk factors for each case with a qualified doctor and nurse, building their understanding of the background story and presentation. SPs were also extensively coached during training to avoid potentially dangerous situations that could arise during fieldwork. While they were allowed to undergo all basic vitals, triage and physical examinations offered by providers, SPs were coached to avoid taking medication or undergoing blood tests, injections, X-rays and other invasive or intrusive procedures during their encounters. Following the training, SPs presented the cases to doctors and nurses who were part of the study, and based on their feedback 14 SP candidates were selected for field implementation (see the online supplementary technical appendix for further details; the complete methodology and materials are available from the authors by request).

Study location and sample description

Health facilities were approached in a convenience sample designed to include low-income, middle-income and high-income neighbourhoods in various Nairobi neighbourhoods. Of 49 health facilities approached, 46 agreed to participate and SP interactions were completed in 42 facilities, with 4 randomly selected facilities held as reserves in case a sampled facility was closed or otherwise inaccessible. Of 168 potential interactions in these 42 facilities, we completed 166 for a completion rate of 98.5%. Of those facilities, 14 were public and 28 were privately owned and operated, with the private sample including 1 community clinic, 5 clinics (of 28) operated by faith-based organisations and 4 clinics operated by social franchise operations (table 2). In analysis, facilities are classified only as ‘public’ or ‘private’ due to the small sample size.

Kenyan health facilities are also classified by ‘level’, reflecting their size and technical capacity.35 Levels 2 and 3 include smaller facilities offering basic primary care and preventive services and are typically staffed by clinical officers or trained nurse practitioners. Levels 4 and 5 cover facilities offering integrated care and inpatient services with increased specialisation, and will typically also be staffed by medical officers (the equivalent of an MBBS in the UK). We excluded level 4 and higher facilities from sampling, as the integrated service offered there could introduce complications to the SP interactions, such as the unstable angina case being immediately triaged into inpatient care.

In comparison with the universe of healthcare facilities present in Nairobi (table 2) according to the government roster, our final facility mix undersampled publicly owned facilities relative to privately owned facilities and oversampled level 3 facilities relative to level 2 facilities. We do not apply any weighting to our outcomes, since our sample is non-random and not intended to be representative of the city’s provider mix, but we report outcomes for public facilities separately in our main results and for private not-for-profit facilities separately in online supplementary appendix table A2.

Facilities were provided a consent form with details on the study, which included both a description of the process (‘In the following 6 months, you will be visited by someone who has been trained by us to act as a patient. These patients are called ‘standardized patients’ and this approach has been used to assess quality of medical care. You will not know exactly when this standardized patient will visit you, but please note the date and time if/when you think you saw this standardized patient. No later than one month after this visit, our research team will contact you to find out if/when you saw our standardized patient’.) and informed consent (‘The standardized patient who visits you for a consultation will pay your usual consultation fees. So, you will not suffer any economic loss due to participation in this study. While you will not directly benefit from the research, we hope that the information from this study will help us understand how the standardized patient approach can be used to evaluate quality of primary care in Kenya.’). There were no direct refusals to participate; however, 3 of the 49 facilities initially approached requested additional time before signing the consent form to obtain further authorisation from a main office that oversees the facility, and they were dropped from the sample.

Between 3 April 2014 and 9 June 2014, SPs were randomly allocated to different facilities with a visit schedule that was optimised over travel and wait times. For instance, multiple SPs could visit a busy facility the same day of the week, but low-load facilities were capped at two SP interactions per day.

Adverse events and detection of SPs

There were no adverse events reported by SPs during the fieldwork. The most common invasive treatment that SPs had to avoid was being given an injection, which was offered in 38 of 126 non-diarrhoea interactions (30%; 95% CI 23 to 39). To assess detection rates, each provider was administered a structured questionnaire within 2 weeks of the completion of fieldwork asking whether an SP had visited the clinic, and, if so, the characteristics of the SP. Providers claimed to have detected an SP in nine instances, but on further elicitation of the characteristics and presentations of the suspected cases, there was no match between the SPs we had sent and the presentation of the patient that the provider suspected to be an SP. We therefore conclude that detection rate for the SPs was zero in this study.

Time burden for providers

The burden on providers was the time spent with SPs, which averaged 7.2 min per interaction (95% CI 6.1 to 8.2) for a total of 28.8 min per facility across the four cases. In all cases, SPs recorded the duration by checking watches or cellphones at the beginning and end of the interaction, for which they were specifically trained. In private facilities, the provider was fully compensated for this time, as the SP paid whatever fee was charged, according to regular clinic procedures. In public facilities, consultations were free and therefore the time spent is a true additional burden on the provider. In these facilities, average consultation time was 4.2 min per interaction (95% CI 2.6 to 5.8) for a total time burden of 16.8 min per facility. This time burden could be substantial if the delay affected seriously ill patients; however, in those circumstances, SPs were trained to allow others to bypass the queue and see the doctor immediately, as is typically customary; this situation did not arise during fieldwork.

Impact of SP characteristics on interactions

A primary concern for the SP validation is that individuals presenting the cases are not actually ill. It is therefore important to ensure that case presentations are convincing, in the sense that continued questioning and examination of the SP will not lead the provider to conclude that the patient is healthy. We therefore examined associations between the likelihood of correct case management and checklist adherence in a multiple regression model with dummies for each case and SP. A 100% completion rate for the essential history question checklist (detailed in online supplementary appendix table A1) for each case versus no questions asked is associated with a 27 percentage point (pp) increase in the likelihood of correct treatment (p=0.123; 95% CI −8 to 62), an 8 pp increase in the likelihood of giving any medication (p=0.662; 95% CI −28 to 43) and a 32 pp increase in the likelihood of any verbal diagnosis (p=0.067; 95% CI −2 to 66). The likelihood of completing at least one of these three behaviours increases by 24 pp over the interval (p=0.041; 95% CI 1 to 46). Although many of the verbal diagnoses given are incorrect (in particular, the angina case and the TB case were often verbally diagnosed with pneumonia or chest infections; see online supplementary appendix table A4 for the full list), the fact that the provider is not led to believe that the patient is presenting false symptoms when asking case-appropriate questions is a strong validation of the methodology in this context.

Since SPs were randomised across facilities, we can also assess whether SP characteristics affected clinical practice. To do so, we enter the body mass index, the systolic blood pressure, gender and age of the SP directly into regressions for various outcomes, controlling for case-specific dummies (online supplementary appendix table A3). SPs who were older and had a higher systolic blood pressure (although none was hypertensive) had higher consultation times and better checklist completion rates, but these characteristics did not affect the likelihood of correct treatment or the type and number of medications received. These differences are of interest in their own right, but do not present challenges to the use of the SP methodology or the accuracy of case depiction beyond documenting variation in care due to naturally occurring variation in the patient population. They do imply, however, that as SP characteristics need to be adjusted for when comparing across providers and ideally, studies should ensure that different providers are visited by similar SPs.

Aggregate results

The typical SP interaction lasted 7.2 min (95% CI 6.1 to 8.2), with providers completing 38% of the condition-specific checklist of history questions and examinations (95% CI 34% to 42%) and correctly managing 88 of 166 cases (53%; 95% CI 45 to 60) (see table 3 for full details). SPs were not instructed to request a diagnosis from the provider, and a verbal or written diagnosis was given to the patient in 53 of 166 cases (32%; 95% CI 25 to 39). These varied widely and are reported in online supplementary appendix table A4. Medication was offered in 118 of 166 cases (71%; 95% CI 64 to 77), with antibiotics, unnecessary in every case, offered in 82 of 166 interactions (49%; 95% CI 42 to 57). Unnecessary steroids, however, were used in only 3 of 166 interactions (2%; 95% CI 1 to 5). While we later present stratified results on differences between public and private providers, we note that the average wait time was 49 min (95% CI 40.6 to 57.7) and that the average interaction costs the SP KSh425 (equivalent to US$5.10 at the exchange rate during the study) in out-of-pocket expenses.

Clinical practice varied substantially across conditions in terms of correct case management. For instance, the asthma case was treated with appropriate medication in 34 of 42 interactions (81%; 95% CI 67 to 90) compared with only 4 of 42 instances for angina (10%; 95% CI 4 to 22), with child diarrhoea and TB falling in between these two extremes. Notably, the use of medicines and (unnecessary) antibiotics was high across all cases, ranging from a low of 32% in child diarrhoea to 60% for the angina SP.

Public–private variation

We observed few statistically significant differences in either preferred case management or the use of medicines across public and private sector clinics (table 4; figure 1), although public clinics were significantly more likely to correctly order a sputum test for the TB case. It is noteworthy that public clinics were just as likely to give unnecessary antibiotics as private clinics across all cases. In contrast, there were large and significant differences in patient experience, with private providers spending 4.5 more minutes with SPs when compared with public clinics (p<0.05) and completing 20 pp more of the required checklist (95% CI 16 to 24). Public-sector SPs waited an additional 68.2 min (p<0.001, 95% CI 53.1 to 83.3) and paid KSh426 less than in private clinics. These estimates are robust to alternate functional form assumptions, such as the logistic, as shown in online  supplementary appendix figure A3.

Cross-country comparison

We compare results from the Nairobi pilot against similar SP presentations from three prior studies: rural India (the state of Madhya Pradesh), urban India (Delhi) and rural China (Shaanxi province), shown in figure 2 and online supplementary appendix table A5.7 17 18 For three conditions — asthma, TB and diarrhoea — the performance of the providers in the Nairobi sample is strikingly better. For example, in the child diarrhoea case, no provider in the sample from China gave oral rehydration salts (ORS) compared with 18% of Indian and 73% of Nairobi providers, and Nairobi providers gave fewer antibiotics than observed in the other settings. Providers in the Nairobi sample were also significantly more likely than those in the China sample to provide the correct treatment, but dispensed antibiotics at about the same rate. Similarly, Nairobi providers were significantly more likely to use microbiological testing for TB than those observed in Delhi. These trends in appropriate management are consistent with the significantly greater time that Nairobi providers spent with patients, which is typically twice as high as the other studies.

This advantage does not extend to all cases: In the Chinese and Indian samples, providers managed the angina case correctly for 63% and 41% of SPs, respectively, compared with only 10% of Nairobi providers. Further, despite wide variation in the proportion of correctly managed cases, there is little to no difference in the use of unnecessary antibiotics. In asthma, for instance, Nairobi providers were 23% more likely to correctly manage the case, but were equally likely to use unnecessary antibiotics. Finally, purchasing power-adjusted comparisons show that out-of-pocket expenses were twice as high in Nairobi relative to China and two to six times higher than in India for the same cases, even though the Nairobi sample includes both public and private providers, with free consultations in the public sector. Waiting times of 95 min in public and 27 min in private clinics were also much higher in Nairobi, compared with 9.5 min for the average wait time in an Indian private clinic. Finally, similar to the 32% in Nairobi, only 32% of cases in rural Indian and 13% in urban India received any verbal diagnosis, which is much lower than the 80% of cases that received a diagnosis in the sample from China.