Evidence gap map of performance measurement and management in primary healthcare systems in low-income and middle-income countries

Introduction

Forty years after Alma Ata, the Astana Declaration of 2018 reaffirmed calls for placing primary healthcare (PHC) at the centre of service delivery in the era of the Sustainable Development Goals (SDG) and universal health coverage.1 The availability of high-performing PHC systems that serve as the first point of contact for the delivery of comprehensive, people-centred health services is an essential component for the achievement of these and other global health priorities. PHC systems also play an important role as a source of preparedness and response to disease epidemics and natural disasters.2–4 Furthermore, a recent report highlighted that quality-driven health systems could save up to 8 million lives annually in low-income and middle-income countries (LMICs) and that structural reforms are needed to improve the performance of individual providers, organisations and entire health systems.5

This paper reports one of several evidence gap maps (EGMs) commissioned to inform the launch of a PHC Research Consortium that delivers policy-relevant research in support of improving the performance and quality of health systems in LMICs.6 Informed by systematic approaches to evidence synthesis and review, EGMs were developed to systematically map evidence and research gaps on broad topic areas, describe the characteristics of the available evidence and inform the design of future research.7 In this paper, we summarise the framework and scope for a performance measurement and management (PMM) EGM, the methods and results, and implications for research.

PMM framework

For the purposes of this study, we defined PMM systems as the set of strategies, resources and capabilities for systematically measuring and improving the performance of healthcare delivery systems at the level of healthcare workers, patients, organisations and populations. The proximal aim of a functioning PMM system is to generate and sustain improvements in the behaviours and competences of healthcare workers and organisations alike, and increase the supply of high-quality services; their distal aim is to contribute to socially valued, population health and equity outcomes.

Based on this conceptualisation, we developed a PMM framework that integrates an existing public administration model of PMM systems8 and a framework developed by our team in the course of evaluating PMM strategies in LMICs.9 The resulting PMM system can be articulated as a series of iterative cycles with inter-related elements that include the policy and organisational context in which the healthcare system is embedded; cyclical measurement of performance at individual-wide, organisational-wide and system-wide levels; PMM strategies or interventions; the process of transformation of raw data into performance information; sense-making and purposive use of performance information by health system actors; design and implementation of improvements; and the outcomes arising at the various levels of the PHC system (figure 1).

Figure 1

Request permission

Performance measurement and management (PMM) framework.

Drawing on organisational science, development economics, behavioural science, health systems research, public-sector administration and the sociology of organisations, we developed a high-level theory of how the PMM cycle described above may be operationalised to improve outcomes. Figure 2 maps the resulting theory of change of how PMM strategies can be linked to various outcomes.

Figure 2

Request permission

Performance measurement and management (PMM) theory of change.

The effective delivery and performance of a PHC system can be influenced by priority health policies and programmes, by PMM strategies and interventions, and by contextual conditions. The implementation of healthcare priorities and PMM strategies, and the use of iterative PMM cycles at multiple levels in a PHC system (individual, organisational and system-wide), can lead (or not) to proximal effects such as healthcare workers increased motivation to perform and to the use of performance information to experiment with operational innovations. The adoption (or not) of those innovations by system actors and the repetition of iterative cycles of measurement and process improvement (operational improvement loop in figure 2) can normalise the PMM cycles within the organisational culture and reinforce the adoption of new routines by growing numbers of system actors.10–12

The regular recurrence of improvement cycles through time may lead to the generation (or not) of intermediate outcomes like health workers’ retention, increased efficiency and productivity, among others, and to the emergence of new organisational capabilities for coordination of care, quality improvement, policy-making and regulation and others.13–16 Through time, these routines and practices can be further normalised and assimilated as new standards and operating procedures. The latter are necessary conditions for the generation of patient-level outcomes such as increased utilisation, effective coverage, improved perception of quality and patient satisfaction. As patients’ perceptions of quality increase, so can PHC service utilisation by households and communities. Assuming that the system’s governance invests in the development of resources and capabilities to maintain gains in performance (organisational learning loop in figure 2), the recurrence of PMM cycles through time can lead (or not) to the questioning of long-standing practices and the emergence of new collective norms about patient and community care. The emergence of these capabilities for system learning and improvement are necessary conditions for the sustained delivery of high-quality PHC services and for PHC system performance (sustaining performance improvement loop in figure 2).16

As with any theory of change, a number of assumptions need to hold for PMM interventions to effectively change distal, population-level outcomes. For example, performance information needs to be available and used to trigger operational improvement or organisational learning. Also, the resulting performance management innovations need to be both well designed and implemented. In addition, PMM effects may be moderated by system antecedents and contextual factors, such as favourable organisational environments and leadership, and reforms to existing policies and regulations, among others.17–22 Ultimately, demand of quality services by households and communities is a necessary condition for PHC system performance. This is a function of sociocultural and economic conditions, public policies such as use of incentives to households and/or healthcare providers, and the effective delivery of health promotion and outreach programmes, among others.

Methods

This study systematically characterised the research done of the interventions and effects of PMM strategies and identified gaps in evidence. This EGM covers a broad body of literature covering PMM strategies that aim to change supply-side and, to a lesser extent, demand-side approaches like social accountability. Our methods differ from systematic reviews that usually address more narrow research questions. In completing this research, we followed the evidence-mapping methods developed by the International Initiative for Impact Evaluation (3ie).26 27 Such methods were outlined in detail in the published study protocol7 and are summarised below.

We systematically mapped the evidence on PMM strategies in PHC systems in LMIC settings that were published in any language between 1 January 2000 and 30 June 2018. Based on the theory of change and the taxonomy described above, we developed inclusion and exclusion criteria, and designed a comprehensive search strategy. The search strategy was developed to identify studies in Medline (PubMED), Embase (Ovid), CAB Global Health (Ovid), CINAHL (Ebsco), Cochrane Library, Scopus (Elsevier) and Econlit (Ovid). We also searched institutional databases and repositories of impact evaluations and systematic reviews. Search strategies are presented in online supplementary file 1.

We included studies in LMICs, as defined by the World Bank, designed to assess the interventions and outcomes listed in table 1 and systematic reviews of such studies. We used the PROGRESS-Plus framework to assess the extent to which intervention outcomes addressed equity and social differentiation into categories such as place of residence, race/ethnicity, occupation, gender, religion, education, socioeconomic status and social capital, age and disability.28 We developed coding tools to extract meta-data from all impact evaluations included and appraised the methodology of systematic reviews using a standardised checklist29 (online supplementary file 2). The latter assesses the methods used to identify, include, appraise and synthesise studies in the review, and gives an overall rating to each review of high, medium or low confidence in review methods.

Included impact evaluations used experimental or observational data to measure the effects of a programme relative to a counterfactual. We included randomised controlled trials (RCTs), regression discontinuity designs, controlled before-and-after studies using appropriate methods to control for selection bias and confounding, instrumental variables estimation or other methods using an instrumental variable, and difference-in-difference study designs. Cross-sectional or panel studies were included that had an intervention and comparison group, and interrupted time series studies that used at least three observations before and three after the intervention. Studies in high-income countries, efficacy trials and systematic reviews of efficacy trials were excluded. We used an online systematic review application (EPPI-Reviewer V.4) to manage the literature database, screen for eligibility and extract data30 and developed and refined data extraction coding tools.

Four reviewers were trained on screening, eligibility criteria and data extraction. All reviewers first screened the same set of 100 studies, and we discussed and resolved any disagreements between reviewers. This process ensured a common interpretation of inclusion criteria within the team. In addition, we also operated with a safety-first approach31 whereby any studies where the first screener was uncertain about inclusion/exclusion were screened by a second, more senior reviewer.

A total of 38 088 articles were uploaded unto EPPI-Reviewer including 38 056 from the systematic searches and 32 from the grey literature and citation tracking. From this original set, we manually excluded 6277 records (16.5%); an additional 21 543 (83,5%) were excluded using EPPI-Reviewer’s machine-learning capabilities, a text-mining feature that allows automatic term recognition and document clustering.32 The resulting 555 studies were screened at full text, after which 200 were excluded with reasons. The most common reasons for exclusion were study design and intervention criteria. The final EGM included 137 impact evaluations and 18 systematic reviews. Figure 3 summarises the screening process.

We conducted descriptive analysis of the database of literature included in EPPI-Reviewer V.4 across geography, time, interventions and outcomes, study design and equity considerations. Resulting data were uploaded to 3ie’s website to create a visual representation of the evidence.

Results

Between 2000 and mid-2018, the total number of studies in the PMM field increased, from 2 in 2000 to 23 in 2017; impact evaluations were conducted in 42 LMICs; two-thirds of the studies were completed in Sub-Saharan Africa (SSA) (56%) and South Asia (19%); no systematic reviews were published prior to 2010. RCTs were the most frequently used study design (71%), followed by controlled before-and-after studies (21%). Thirty studies adopted mixed-methods study designs. The most common methods of analysis were multivariate regression, difference-in-difference estimation, and comparison of means. Few studies used fixed-effects or random-effects models, instrumental variables or propensity score matching. RCTs were less frequent in Latin America and the Caribbean (LAC).

The most frequently studied PMM approaches were implementation strategies such as in-service training, supervision and continuous education, and on financial arrangements such as performance-based financing (PBF). These types of strategies accounted for more than 90% of studies in SSA and LAC, more than two-thirds in South Asia and Middle East and North Africa (MENA), and 65% in East Asia and the Pacific (EAP). PBF and incentives were under-represented in South Asia (4%) and EAP (12%). We identified a small cluster of continuous quality improvement studies in SSA (5%) while all other regions showed absolute gaps in this type of intervention. Similarly, supervision interventions were studied in South Asia (31%), SSA (8%) and EAP (6%), but no studies were found in LAC or MENA. While the category of accountability arrangements was the least studied overall, 24% of studies in South Asia addressed public release of performance information, and 8% in SSA evaluated social accountability interventions. Other than a few studies in SSA (2%), the largest gaps were in the area of audit and feedback.

The most frequently assessed outcomes were effects at the provider level, followed by patient-level and population-level health. Among the former, the acquisition of knowledge and skills and adherence to guidelines were the most frequently measured effects; the least assessed were work morale; providers’ attitudes, beliefs and perceptions; turnover and retention; workload; stress; burnout; and sick leave. Patient behavioural and mental health outcomes and adherence to treatment were infrequently studied. Two-thirds of studies addressed more than one outcome.

At the organisational level, the most frequently addressed outcomes were adherence to recommended practice or guidelines, process of care improvements, patient satisfaction and perceived quality of care. Population-level outcomes included utilisation or coverage of health services, and, less frequently, access to services. Overall, the least studied outcomes were unintended effects, harm, and social and equity outcomes. No studies measured changes in organisational culture.

Twenty-eight impact evaluations addressed equity to some extent. The majority of studies analysed subgroups or targeted populations. Very few studies considered an equity-sensitive research question, assessed equity outcomes, adopted an equity-sensitive method to assess how and why effects were produced, or followed an equity-informed research process. No study adopted equity-sensitive analytical frameworks or theories of change. Among the studies that addressed equity, the most commonly addressed dimensions were place of residence and socioeconomic status. Few studies considered other dimensions such as gender; vulnerable groups; race, ethnicity, culture and language; education; or age. No studies considered disability, occupation, religion or social capital. Equity considerations included in the systematic reviews were almost absent.

Although we only found 18 systematic reviews, their quality was reasonably high. Nine reviews were rated as high confidence, two as medium confidence and the remaining seven as low confidence. Of the high-confidence reviews, five were published in the Cochrane Collaboration Library, three were published in peer-reviewed journals and one in the Campbell Collaboration Library. Reviews rated as high confidence had clear inclusion criteria, conducted comprehensive searches that included sources of grey and unpublished literature, undertook independent double-screening of included studies, and undertook and clearly reported risk of bias assessments using sensible criteria. They also typically undertook at least partial independent double data extraction, clearly reported the characteristics and results of included studies, combined the results of included studies appropriately and explored heterogeneity in results, including by risk of bias of the included studies where appropriate. The reasons for the systematic reviews being downgraded to low confidence included a combination of conducting a limited search that did not include grey literature, selective reporting of results of included studies, lack of independent screening of full-text papers, not assessing the risk of bias or quality of included studies, and using vote counting by direction of effect and statistical significance as a method of synthesising findings.

Reflecting the IE evidence base, systematic reviews also focused on individual-level PMM interventions such as in-service training, continuous education and supervision. Nine systematic reviews addressed PBF, six of which focused on healthcare workers’ behaviours and three on organisational-level change. Several reviews considered more than a single intervention. Specific outcomes addressed included adherence to guidelines and provider knowledge and skills. Seven systematic reviews addressed patient-level mortality. Population-level outcomes included increased access, coverage and utilisation of specific services along with changes in patient knowledge, adherence, satisfaction and perceived quality of service. Only one review measured unintended patient, organisational, population and social outcomes, and two more addressed changes in work morale and provider attitudes, beliefs and perceptions. Three of the 18 systematic reviews had an equity focus.

A summary of results is presented in table 2. The resulting map (not included here due to its size) is available in 3ie’s website (http://gapmaps.3ieimpact.org/evidence-maps/performance-measurement-and-management-primary-care-delivery-systems). In the map, readers will find interventions on the y-axis and outcomes on the x-axis. Included studies are presented in the form of bubbles according to the interventions and outcomes addressed. Bubbles indicate the relative size of the evidence base; grey bubbles show impact evaluations and coloured bubbles identify systematic reviews and protocols. The confidence rating assigned to the systematic reviews in the critical appraisal is also indicated by colours (red, low confidence; orange, medium confidence; green, high confidence). Linkages to all studies are available online and are listed in online supplementary file 3.

Discussion

We conducted an EGM of the evidence on the effects of PMM system interventions in PHC systems in LMICs to characterise existing research and identify evidence gaps. Overall, our main finding is that at a global level, the evidence is concentrated on implementation strategies such as in-service training and continuing education, and financial arrangements such as PBF and financial incentives. More specifically, there are PMM strategies for which we found few or no studies. These absolute gaps are concentrated in the area of accountability arrangements and, specifically, on interventions such as audit and feedback, public release of performance information and social accountability. In the category of implementation strategies, there was an absolute gap in the area of continuous quality improvement.

To date, the evidence base is mainly focused on individual-level outcomes among healthcare workers such as the acquisition of knowledge and skills and the adherence to practice guidelines. Patient-level outcomes are also frequently measured, in particular the utilisation and coverage of specific services or biomedical interventions. Globally, gaps in outcomes are concentrated on organisational-level and on the social-level and equity-level effects of most PMM strategies and interventions. There is also an almost-complete absence of studies that address unintended and harmful effects. The patterns identified for impact evaluations are largely mirrored in the systematic reviews available.

Research to date has prioritised the evaluation of PMM strategies that appear to operate, in the short term, through changing the behaviours of healthcare workers and patients. The underlying programme theories appear to assume that changing PMM inputs and financing processes would somehow ensure population-level health outcomes. However, our theoretical framework suggests that PMM strategies behave like complex systems whose functioning requires coordinated and sustained efforts operating at multiple levels going beyond improving individual providers’ skills, knowledge or incentives. Implementation of under-theorised PMM strategies and interventions may contribute to this, but the prevalence of ‘single method’ studies originating from a single discipline is also likely to be an important factor. We found that a majority of the identified impact evaluations adopted experimental designs, and while such studies generate reliable evidence on effects, the infrequent use of mixed-method impact evaluations also means that important research questions such as how and why change happens (or not), for whom and at what costs are not addressed.33 34

Given the strategic role that PHC systems can play in ensuring that differentials in health outcomes are reduced, the lack of attention to equity in the evidence base is a major concern as countries face the challenge of attaining the SDGs. Furthermore, unsafe practices, harmful effects and unintended outcomes have not been systematically addressed in the literature. This is also relevant for the SDG agenda, given that PMM interventions can create perverse incentives and undesirable effects with potential for worsening or further deepening existing inequities.

Despite differences in aim and scope, a recent systematic review (not included in our dataset) of the effectiveness of strategies to improve the practice of healthcare providers in LMICs35 reached similar conclusions. In particular, the authors found that the strategies used had low-quality evidence and that comparisons between studies were difficult given contextual and methodological heterogeneity. Our findings also echo those reported by Shojania and Grimshaw more than a decade ago in their review of the evidence on quality improvement interventions.36 Similar to their findings, we find a lack of convincing theories underlying the selection and design of PMM strategies studied in the literature. Furthermore, much of the literature fails to meet commonly accepted standards for the conduct and reporting of research. Evidence on effects is necessary, but not sufficient, for informing policy and practice. The field requires approaches to impact evaluation that are informed by well-articulated theory and that effectively integrate methods for assessing effects with qualitative and systems science approaches that are better suited to address the interactions between context and implementation processes.37 38

Conclusions

In view of the recent Astana Declaration1 and calls for high-quality health systems for achieving the SDGs,5 the limited evidence available on PMM strategies is a major concern. The evidence that does exist is clustered around relatively ‘simple’ interventions that are unlikely to achieve the transformational change in health outcomes that will be required to reach the SDGs and save and improve the lives of the millions of people without access to high-quality health services, particularly in LMICs. The evidence gaps identified suggest that routinely used PMM strategies are being implemented without sufficient knowledge of their effects, intended and otherwise. We call on researchers, commissioners and decision-makers to get behind an ambitious research agenda which can ensure future efforts at strengthening or redesigning PHC systems are informed by evidence on the most effective approaches for using PMM strategies to improve outcomes.