The WHO-INTEGRATE evidence to decision framework version 1.0: integrating WHO norms and values and a complexity perspective

Step 1: Development of the framework

In step 1a we analysed WHO norms and values and, rooted in these norms and values, proposed a structure for the WHO-INTEGRATE framework and derived preliminary criteria. A universally agreed normative theory for health does not exist, but most rivalling theories converge on a set of principles.31 As the use of these principles is less restrictive than the choice of one theory over another, we pursued a principles-based approach,31 32 and used WHO norms and values as the guiding principles for developing a new EtD framework. Given the complexities of normative orientation in modern pluralistic and globalised societies, we believe that WHO norms and values represent a useful foundation: they are rooted in the universally recognised concept of human rights and receive their legitimacy from having been agreed on by all 194 Member States of the WHO. To identify WHO norms and values of relevance to the process of guideline development and implementation, we used the WHO Constitution33 and chapter 5 ‘Incorporating equity, human rights, gender and social determinants into guidelines’ of the WHO Handbook for Guideline Development22 as a starting point. Given the emphasis in these two documents on human rights, equity and non-discrimination, social determinants of health and the role of health systems, we retrieved and analysed relevant related documents,34–40 including several public health ethics frameworks.16 18 41–50 We also reviewed the SDGs51 in view of their likely impact at global and national levels and as WHO is mainstreaming these throughout the organisation’s work.52–54

From these documents and sources, we derived principles and concepts. The structure of the WHO-INTEGRATE framework was developed via an iterative process among coauthors. We explored the meaning of different principles and concepts and assessed overlap and redundancies, making rearrangements to derive preliminary criteria. In doing so, we used a structure and wording as close as possible to the existing GRADE EtD framework to build on its strengths and to maximise potential synergies. On several occasions, we also consulted with members of the WHO Guidelines Review Committee as well as other WHO staff considered experts on selected principles or concepts (see Acknowledgements).

During the development process, we focused on substantive criteria or what decisions are based on (eg, cost, acceptability) rather than procedural criteria or how the decision-making process is conducted (eg, composition of guideline panels, participation, transparency). This is consistent with the approach promoted by the WHO Handbook for Guideline Development,22 whose overall purpose is to specify procedural rules for an objective, transparent and acceptable guideline development process. Embedded in these procedural rules, the current WHO EtD framework (table 10.1 of the WHO Handbook for Guideline Development)22—and, by extension, the WHO-INTEGRATE framework presented here—is concerned with how to facilitate the use of evidence in decision-making in a structured and comprehensive manner. It is important to note that a distinction between structural and procedural aspects is widely practised in guideline development and several other health decision-making processes,5 6 19 55 but is not commonly seen in the public health ethics literature.43 44 46 56

In step 1b we refined the preliminary criteria derived from WHO and other related documents and supplemented them with a comprehensive set of subcriteria; we also developed definitions for criteria and example questions relating to each of the subcriteria. We conducted an overview of systematic reviews of criteria used in decision-making, priority setting and resource allocation processes for health to derive a comprehensive set of health-relevant criteria (Stratil et al, forthcoming). We then compared the preliminary criteria developed in step 1a against this comprehensive set of criteria and subcriteria. To do so, one author (JMS), in a discussion with a second author (EAR), allotted the subcriteria obtained from the overview of systematic reviews to the preliminary criteria within the WHO-INTEGRATE framework. Subcriteria that did not fit were kept in a separate category. Any uncertainties were resolved in discussion with a third author (RB).

We then prepared definitions for each of the criteria using the above-described source documents for health norms and values, existing health decision frameworks (Stratil et al, forthcoming), and any definitions or descriptions provided in the publications included in our overview of systematic reviews of criteria (Stratil et al, forthcoming). Where appropriate, we also drew on additional key documents (eg, Scott et al 57 for the definition of acceptability, Hultcrantz et al 58 for the conceptualisation of quality of evidence, and Maeckelberghe and Schröder-Bäck59 for details on the subcriteria for human rights and sociocultural acceptability and health equity, equality and non-discrimination). Each definition (1) provides an overall definition of the criterion, (2) offers details and explanations regarding the subcriteria, and (3) gives guidance on how the criterion in question influences the recommendation.

As we prepared definitions, we also examined the extent to which the criteria and subcriteria relate to the intervention itself versus the health system and the broader context, in which an intervention is implemented. For example, the same label (eg, equity) may be employed to describe different underlying concepts, relating to process versus outcome (an intervention can either be implemented taking equity principles into account, or it can increase or decrease equity in health outcomes) and the point in time when these criteria apply (eg, equity before, during or after intervention implementation). To enable better access to sometimes abstract constructs, we also developed example questions for each of the subcriteria, drawing on the same set of documents as above.

Step 2: assessment of framework value

In step 2a we explored whether the WHO-INTEGRATE framework would be able to accommodate different types of health interventions and different features of complexity. We assessed to what extent the WHO-INTEGRATE framework would be able to accommodate features of distinct types of health interventions.60 We broadly distinguished between interventions targeting individuals (eg, diagnosis, treatment or preventative measures addressed at individuals), interventions targeting populations, and interventions targeting the health system or other systems. Population-level interventions encompass those concerned with whole populations or population groups as defined by their age, sex, risk factor profile or other characteristics; they are often implemented in specific settings or organisations (eg, school health programmes). System-level interventions specifically redesign the context in which health-relevant behaviours occur; they are often implemented through geographical jurisdictions from national to local levels (eg, laws and regulations regarding the taxation, sale and use of tobacco products). Health system interventions represent a specific type of system-level intervention and often result in complex rearrangements across multiple health system building blocks (eg, task shifting as a process of delegating specific health service tasks from medical doctors or nurses to less specialised health workers). Interventions implemented at any of these levels can be conceptualised and analysed from a complexity perspective. To do so, we mapped core and additional components of complex interventions as defined in the iCAT_SR tool61 and sources of complexity in systems reported in another paper in this series28 against the included criteria.

In step 2b we examined the usefulness and relevance of the WHO-INTEGRATE framework and its criteria among those developing WHO guidelines. We conducted key informant interviews with individuals who had recently participated in a WHO guideline development process. In consultation with the Secretariat of the WHO Guidelines Review Committee, we purposively selected three ongoing or completed guidelines that had applied the current WHO EtD framework,29 62 63 seeking to cover distinct types of health interventions and positive as well as more difficult experiences with the application of the framework. For each guideline, we interviewed the WHO staff coordinating the guideline, the Chair of the guideline development group and the methodologist. The interviews were semistructured and used a pretested interview guide concerned with practical considerations (eg, understandability, operationalisability), as well as an assessment of missing and redundant criteria of the WHO-INTEGRATE framework. Interviewees were also asked to reflect on the implications of the WHO-INTEGRATE framework for evidence collection and guideline formulation. Interviews were held between June and November 2017 either face-to-face at the WHO Headquarters in Geneva or by telephone (JMS). Interviews were audiotaped and transcribed; data were then analysed by two researchers (JMS and IBS) using qualitative content analysis.64 We employed a combination of deductive (based on the guiding research questions) and inductive approaches using the software MAXQDA (VERBI Software, Berlin).

Step 3: facilitation of framework uptake

We critically examined how to enable use of the WHO-INTEGRATE framework as intended, and generated a table linking the criteria with suggested methods for primary research, evidence synthesis and assessing quality of evidence. The current WHO EtD framework is intended to be used right from the planning stages of a guideline, to help derive relevant questions and structure the process, but in practice it is usually used at the end of a guideline process to help decide on the recommendations. To determine how the new framework could be used more holistically, we reflected on the literature reviewed in the context of developing the WHO-INTEGRATE framework and sought feedback from a large number of experts (see Acknowledgements). We specifically sought suggestions on how to use the framework during the early stages of the guideline development process and in a context-specific manner.

To make it easier for guideline panels to populate the criteria in the framework with evidence, we identified types of primary research, evidence synthesis methods and methods for assessing evidence quality that could inform each criterion. To accomplish this, two researchers (AP and EAR) reviewed the research questions and methods described or mentioned in the systematic review of health decision frameworks and the overview of systematic reviews (Stratil et al, forthcoming). We also consulted a broad range of experts comprising other authors of papers in this series, selected guideline development organisations (eg, Guidelines International Network, UK National Institute for Health and Care Excellence) and researchers with an interest in evaluating complex health technologies (see Acknowledgements).

Developing the preliminary framework

Using the review of the WHO Constitution,33 chapter 5 of the WHO Handbook for Guideline Development22 and other source documents, we identified six major, partly overlapping concepts. Further sorting of these yielded four sets of principles and concepts (human rights principles, ethical principles, sustainability elements and health system goals and building blocks). Figure 2 illustrates how we derived preliminary criteria from WHO norms and values.

• Human rights principles, for the purposes of this framework, were primarily derived from international human rights law and its interpretation by the Committee on Economic, Social and Cultural Rights’ General Comment on the Right to the Highest Attainable Standard of Health (Art 12).34 These contain the interrelated concepts of availability and accessibility of public health and healthcare facilities, goods and services, which are required to be of appropriate quality and acceptable to users. They also include the general human rights principles of equity and non-discrimination, accountability and participation.

• Given the large number of biomedical and public health ethics frameworks,44 46 56 in consultation with WHO, we structured the ethical principles primarily according to the public health ethics framework of Childress and colleagues. This framework inter alia includes the aspects of producing benefits, avoiding harms, maximising the balance between benefits and harms, as well as distributive justice and autonomy.41 Based on analytical tools by the Nuffield Council of Bioethics,45 we also added the principle of low intrusiveness, which is related to privacy and dignity.

• Acknowledging the importance of the social determinants of health and the SDGs, we derived sustainability elements to capture the wide range of factors that promote conditions in which people can lead a healthy life and allow societies and individuals to develop and flourish; these sustainability elements also reflect the societal impact that interventions can have beyond health outcomes. Importantly, good health is both a precondition for achieving sustainable development and an outcome of sustainable development.65

• To capture the importance of feasibility of implementation as well as the impact of interventions on the health system, we used the WHO health systems framework with its four goals (ie, improved health, responsiveness, social and financial protection, improved efficiency) and six building blocks (ie, leadership and governance, financing, medical products, vaccines and technologies, information, health workforce, service delivery infrastructure).19 38–40

Figure 3 presents the WHO-INTEGRATE framework with its six criteria: balance of health benefits and harms, human rights and sociocultural acceptability, health equity, equality and non-discrimination, societal implications, financial and economic considerations, and feasibility and health system considerations. A seventh criterion, quality of evidence, represents a metacriterion that applies to each of the six substantive criteria. All seven criteria are relevant to health decision-making and the formulation of recommendations as part of the guideline development process. Each criterion may apply to interventions targeting individuals, populations or systems, or any combination of these levels.

While priority of the problem featured in both the health decision frameworks included in our systematic review (eg, Alonso-Coello et al 13) and in the overview of systematic reviews of criteria (eg, Guindo et al 6), we did not include this as a stand-alone substantive criterion for two reasons: First, many of the aspects included, for example, political will or public concern, are used to inform the decision to develop a guideline (or make another health decision) and thus apply before the start of the guideline development process. Second, selected aspects are captured under the other six substantive criteria, for example, burden of disease features under balance of health benefits and harms, and large cost of disease to health system features under financial and economic considerations.

Defining criteria, subcriteria and example questions

Our overview of systematic reviews yielded more than 30 systematic reviews that contained several thousand criteria and subcriteria currently used in decision-making (Stratil et al, forthcoming). Recurrent aspects addressed by the subcriteria focused on the health outcomes and benefits of the intervention, health benefit for individuals and the benefit for society as a whole, the societal importance of the disease, economic considerations, quality or uncertainty of evidence, as well as population priorities, priorities within the health system and stakeholders’ interests and pressures. Feasibility criteria were concerned with the available budget, the capacities within the health system, technological complexity and acceptability of the intervention within society. Some systematic reviews were primarily concerned with interventions that would benefit vulnerable or marginalised populations (eg, children, mothers, people with lower socioeconomic status). In many reviews, normative criteria such as ethics, justice or fairness were mentioned without clear definitions or contextualisation. This comprehensive list did not yield any further criteria beyond the seven presented in figure 2. It did, however, provide many subcriteria as well as elements used in the development of detailed definitions and example questions for each criterion.

Suitability of the framework for decisions about complex health interventions

An earlier paper in this series28 emphasises the importance and added value of reviewing evidence from a complex systems perspective. In developing the new EtD framework, we wanted to ensure that it would be fit for purpose when making decisions about complex interventions implemented in complex systems. We first explored to what extent different features of intervention and system complexity apply to two broad categories of interventions, that is, individual-level versus population-level and system-level interventions (table 1). Notably, even population-level and system-level interventions (eg, regulations and programmes to increase access to improved sanitation) eventually bring about changes in individual behaviour (eg, use and maintenance of toilets or latrines). Some criteria apply to a greater extent with population-level and system-level interventions (eg, societal implications) than individual-level interventions. Some subcriteria may take on a different meaning when applied to individual-level versus population-level and system-level interventions (eg, autonomy). Broadly speaking, most features of complex interventions apply to both individual-level and population-level/system-level interventions but are more salient for the latter. In contrast, many features of complex systems only apply to population-level and system-level interventions.

The last column of table 1 illustrates that distinct features of complexity do not neatly map onto specific criteria. Instead, distinct features of complexity usually affect multiple, sometimes all, criteria in the WHO-INTEGRATE framework. For example, the worked example of childhood obesity, introduced in an earlier paper in this series,28 discussed adaptivity of the system in response to raised taxes on soft drinks (eg, creation of lower-sugar alternatives by the soft drinks industry). This adaptivity can thus influence the balance of health benefits and harms (eg, consumption patterns of soft drinks change but in less pronounced ways, thereby dampening the expected effect on childhood obesity), and it may even have unwanted social consequences by stigmatising those unable to afford soft drinks (social impact). Raising taxes on only one sugar-sweetened product may lead to increasing the sugar content of other sugar-sweetened products (impact on economy, broader positive or negative health-related impacts) or have implications on agricultural production patterns nationally and internationally (impact on economy and environmental impact), illustrating the complexity of downstream implications of a ‘simple’ intervention. Drawing on the same worked example, box 1 illustrates how a simple linear perspective on the effect of an intervention will place the emphasis on one or a few criteria for decision-making, whereas a complexity perspective may take all criteria into account when making a recommendation.

Usefulness of the framework from the perspective of WHO guideline developers

The key informants we interviewed had been involved in developing three very different guidelines—the WHO recommendations on antenatal care,29 the WHO consolidated guideline on sexual and reproductive health and rights of women living with HIV,62 and the WHO guideline on risk communication (online supplementary table S1).63 Each of these guidelines faced different challenges in terms of scope, availability of evidence and ability to incorporate multiple perspectives. All three had used the current WHO EtD framework with varying success. The diverse experiences and viewpoints of the key informants on the practical application of these criteria in guideline development were helpful in refining the framework. Further detail on and complete findings from the key informant interviews will be reported separately (Stratil et al, forthcoming).

Most participants commented positively on the WHO-INTEGRATE framework and highlighted the value of a criterion assessing societal implications, as well as the broader and more detailed specification of the criteria human rights and sociocultural acceptability and health equity, equality and non-discrimination. Two participants questioned the added value of the new EtD framework, since any guideline development process led by an experienced methodologist would automatically address the details covered in the subcriteria. Several participants were concerned about the workload that the use of the WHO-INTEGRATE framework might add to the guideline development process.

Specific remarks were made in regard to (1) missing criteria and subcriteria; (2) the hierarchy and order of criteria and subcriteria; (3) overlap and redundancies between criteria and subcriteria; (4) the precise wording and definitions of criteria; (5) the need for (more) guidance on how to use and interpret criteria and subcriteria; (6) the challenges of identifying and synthesising the required evidence; (7) resource, time and skill implications for the guideline development process; as well as (8) procedural aspects for using the framework in the guideline development process.

In response to these concerns and suggestions, we made several modifications, including changing the name and definition of several criteria and subcriteria to improve clarity and reduce overlap. We also expanded the example questions for the subcriteria to improve understandability and facilitate the development of specific questions for a given guideline. Moreover, we added suggestions on how to prioritise among criteria and subcriteria in a problem-specific and context-specific manner. Finally, we emphasised the importance of incorporating the voices of those directly affected by the recommendations into the guideline development process.

Table 2 presents the WHO-INTEGRATE framework version 1.0 criteria with abbreviated definitions and lists subcriteria. Online supplementary table S2 provides detailed definitions of the criteria as well as example questions for each of the sub criteria.

Facilitating uptake: using the framework holistically and populating the criteria with evidence

The WHO-INTEGRATE framework is intended to improve transparency in health decision-making by supporting a structured process of reflection and discussion in a problem-specific and context-specific manner. To be most effective, this process must begin at the start of a guideline or other health decision-making process and must take evidence into account. The WHO-INTEGRATE framework is not intended as a ‘tick-box exercise’; there must be prioritisation of the most relevant criteria and subcriteria depending on the questions addressed by a given guideline, and the time and resources at disposition. It would be impossible and probably unnecessary for every guideline development or health decision-making process to examine all subcriteria. This flexibility can, however, lead to misuse, as stakeholders may disproportionately (eg, academics from high-income countries) or unduly (eg, participants with substantial declared or undeclared financial or other conflicts of interest) influence the decision-making process. Safeguards can be put in place through explicit procedures, in particular in relation to the composition of guideline panels or other decision-making groups. The WHO-INTEGRATE framework is also not an algorithm for integrating evidence across different criteria: making decisions under uncertainty and agreeing on trade-offs across criteria and subcriteria and among (and within) diverse stakeholder groups remain a core task for a guideline panel.

All criteria are important and should be reflected on, but their relevance varies depending on the type of health decision and the decision-making context. In contrast, not all subcriteria are always relevant. At the start of a guideline or other decision-making process, an appropriately composed guideline panel or other decision-making group needs to discuss which of the subcriteria are applicable and useful in relation to the nature and specific characteristics of the intervention (see table 1); this group will also need to consider the specific information needed to populate criteria or subcriteria (see table 3). Complexity in the intervention and complexity in the system into which this intervention is implemented can usually be detected; the critical question is whether it is of value to examine this complexity in depth (see box 1 in this paper and box 2 in an earlier paper in this series28). This prioritisation process should take the views of relevant stakeholder groups into account; which stakeholder groups are relevant depends on the nature of the problem and the institutional as well as broader physical and social context. In principle, these should include those directly affected by the intervention (eg, patients, beneficiaries), those financing (eg, health insurance providers, ministries of health, other ministries) or implementing the intervention (eg, healthcare providers, public health professionals, professionals outside of the health sector), as well as the general public.

A systematic weakness in many guideline development and other health decision-making processes is that consumer participation is obviated and guideline panels often substitute their own values and views for those of patients/beneficiaries. The voices of patients/beneficiaries and other relevant stakeholder groups can be incorporated through direct participation or representative surveys66 as well as qualitative research (see table 3).

The guideline panel will also need to decide how best to populate the criteria with evidence and whether a formal evidence synthesis or a more pragmatic approach is warranted for each. This decision will be influenced by the relevance of criteria and subcriteria in relation to a specific intervention or decision, and by the likely types and quantity of evidence available, as well as time and resource constraints. At the end of the process, the guideline panel will need to reassess the criteria and relevant subcriteria in light of the assembled evidence and make a judgement regarding each criterion.

Table 3 suggests relevant types of primary research, evidence synthesis or mapping methods, streamlined or pragmatic approaches, as well as methods to assess the quality of evidence for each of the six substantive criteria. We provide a collection of suitable primary research and synthesis approaches, but make no firm distinction between more or less suitable methods. We note that the approach to gathering evidence may depend on the criterion: for some criteria a systematic review will be most appropriate, while for others a representative survey or other single primary study may be more suitable. Surprisingly, the majority of the health decision frameworks included in our systematic review (Stratil et al, forthcoming) did not offer insights for operationalising frameworks, for example by specifying research questions or suggesting methods for primary research or evidence synthesis. The GRADE EtD framework13 67 and the EUnetHTA (EUropean network for Health Technology Assessment) core model68 provided some methods. We also identified relevant information in the following sources: the EVIDEM (Evidence and Value: Impact on DEcisionMaking) framework,14 Marckmann and colleagues,16 the health systems framework19 and publications included in our overview of systematic reviews of criteria.69 Expert consultation played a critical role in identifying methods for inclusion in table 3.

Added value of the WHO-INTEGRATE framework

The WHO-INTEGRATE framework represents a new comprehensive EtD framework that is rooted in WHO norms and values. It offers an explicit conceptualisation of each criterion and a rationale for including relevant concepts as criteria or subcriteria. The WHO norms and values apply across all WHO Member States and settings, and the new framework should, in principle, be relevant for health decision-making at global, national and subnational levels. It reflects a broad understanding of health and its determinants and takes account of complex interventions and complex systems perspectives. It emphasises sustainability and the interconnectedness between health and other sectors, inherent in the SDGs. While the framework is conceived for individual-level, population-level and system-level interventions, it is likely to be particularly well suited for public health and health system interventions characterised by complexity and/or approached from a complexity perspective. The WHO-INTEGRATE framework is intended as a tool to facilitate structured reflection and discussions from the beginning of a guideline development or other health decision-making process. This has ramifications in terms of the need to prioritise among criteria and subcriteria and the need to collect evidence for each. The framework supports this process by offering structured definitions for each criterion and example questions for each subcriterion, and by suggesting methods for primary research, evidence synthesis and assessing the quality of the evidence.

There are many similarities between the WHO-INTEGRATE framework and the widely used GRADE EtD framework. As stated in our methods, we deliberately attempted to stay as close as possible to the GRADE EtD framework, thus building on established terms and concepts (eg, balance of health benefits and harms). In contrast, criteria with a strong normative foundation (eg, health equity, equality and non-discrimination) were much less developed in the GRADE EtD framework; notably, the criterion societal implications, which has its roots in the recognition of the multisectoral determinants of health, is absent from the GRADE EtD framework. There are also more fundamental differences. While the GRADE EtD framework emphasises the efficacy/effectiveness of interventions and their potential harmful impacts, there is no inherent weighting of criteria in the WHO-INTEGRATE framework: guideline panels must decide in a context-specific and problem-specific manner which criteria and subcriteria are most relevant. Moreover, in contrast to the narrower certainty of evidence concept in the GRADE EtD framework, the WHO-INTEGRATE framework has deliberately adopted a broad quality of evidence concept that applies across all criteria and is not linked to a prespecified grading system. For several criteria (and/or subcriteria) GRADE70 71 and GRADE CERQual (Confidence in the Evidence from Reviews of Qualitative Research72) are the most appropriate approaches to examining quality of evidence, and we would encourage users of the WHO-INTEGRATE framework to adopt these. In fact, another paper in this series explores how complexity can be considered when assessing the certainty of evidence on intervention effectiveness.73 For other criteria (and/or subcriteria), these existing tools are not well suited, and we hope that more appropriate approaches will become available—whether through further developments within the GRADE Working Group or independent efforts.

The GRADE EtD framework allows for tailoring of criteria, for example by considering a detailed judgement as a stand-alone criterion or by removing a criterion from the GRADE EtD framework and considering it prior to the start of the decision-making process13; in fact, refinement of the GRADE EtD framework continues and has already resulted in suggestions towards more detailed specifications of selected criteria.74 Similarly, we expect various developments towards a version 2.0 of the WHO-INTEGRATE framework (see below). We thus envisage specific innovations to be adopted across these evolving frameworks and, potentially, convergence over time.

Strengths and limitations of the development process

In developing the WHO-INTEGRATE framework, we combined a principles-based approach with an overview of systematic reviews of decision criteria and thus ensured a solid, comprehensive normative foundation. We were explicit and transparent as to how criteria (see figure 1) and subcriteria (Stratil et al, forthcoming) were derived. While there is some conceptual overlap at the level of the criteria (eg, societal implications and financial and economic considerations), there are no significant redundancies among the subcriteria (Stratil et al, forthcoming). Cross-linkages among the criteria are emphasised in the definitions and example questions.

Solely adapting the substantive criteria may be insufficient to overcome limitations in guideline development or other decision-making processes.22 The WHO-INTEGRATE framework is concerned with substantive criteria; it does not comprise procedural criteria but is intended to be embedded in a clearly specified health decision-making process as described, for example, in the WHO Handbook for Guideline Development.22 We recognise that transparent and inclusive procedures are essential to achieve legitimate health decisions and to resolve reasonable disagreement based on competing criteria and the various individual, social, cultural and political values affecting their interpretation and the explicit or implicit weight assigned to them. In this context legitimacy refers to the reasonableness, or acceptability, of decisions as perceived by the population.75 76 Compromised legitimacy may hinder the effective implementation of guidelines or other health-relevant decisions. Transparent and inclusive procedures require, among other considerations, the involvement of relevant stakeholders in the decision-making process, the public announcement of forthcoming decisions including their underlying argumentation, and the instalment of mechanisms for appeal.75 76 This is relevant for the development of WHO guidelines at the global level, as well as their adaptation at the national or subnational levels, where a wide array of stakeholders with diverse sets of values should be involved.77 78 In our overview of systematic reviews, we distilled procedural criteria (Stratil et al, forthcoming) and suggest that these be reviewed separately to inform guideline development and other health decision processes.18 41 42 45 75 76 79 We also refer to evidence-informed deliberative processes, which explicitly integrate the use of substantive criteria with procedural criteria to set priorities at national and subnational levels.80–82

The WHO-INTEGRATE framework is a highly interdisciplinary framework: each criterion, especially those criteria that are less developed in current EtD frameworks (eg, human rights and sociocultural acceptability) or absent from the literature (eg, societal implications), merits research to unpack them and, where applicable, provide a more detailed normative justification. We anticipate constructive input from and exchange with relevant disciplines, in particular public health ethics but also sociology, environmental sciences, economics and many others. Future collaborative research is expected to lead to a WHO-INTEGRATE framework version 2.0. This may advance the criteria and subcriteria and their normative foundations, as well as methodological approaches to populate these criteria with evidence.

To examine the value of the WHO-INTEGRATE framework to potential users, we conducted empirical qualitative research. Insights from interviews with key informants in relation to their recent experiences with developing WHO guidelines led to several refinements in the wording of the criteria and subcriteria and highlighted the importance of providing example questions as well as suggested methods. We expect that the second empirical qualitative research component, focus group discussions in Nepal, Uganda, Germany and Brazil, will yield additional insights from different perspectives and possibly further modifications to the framework. An integrated analysis of the views of WHO guideline developers and users will be published separately (Stratil et al, forthcoming).

Several of our key informants expressed concern about the potential workload resulting from collecting evidence for each of the criteria and, in particular, for the many subcriteria in the WHO-INTEGRATE framework. Both the process of prioritisation and the process of collecting evidence—through high-quality evidence synthesis or more pragmatic approaches—need to be tested in practice. We anticipate sharing worked examples and developing additional guidance on how to implement the framework in practice.