You are here

  • Home
  • Archive
  • Volume 11, Issue 12
  • Heterogeneity in outcome assessment for inflammatory bowel disease in routine clinical practice: a mixed-methods study in a sample of English hospitals

Article Text

Article menu
Gastroenterology and hepatology
Original research
Heterogeneity in outcome assessment for inflammatory bowel disease in routine clinical practice: a mixed-methods study in a sample of English hospitals
  1. Violeta Razanskaite1,
  2. Constantinos Kallis1,
  3. Bridget Young2,
  4. Paula R Williamson1,
  5. Keith Bodger1,3
  1. 1Department of Health Data Science, Institute of Population Health, University of Liverpool, Liverpool, UK
  2. 2Department of Public Health, Policy and Systems, Institute of Population Health, University of Liverpool, Liverpool, UK
  3. 3Digestive Diseases Unit, Aintree University Hospital, Liverpool, UK
  1. Correspondence to Dr Keith Bodger; kbodger{at}liverpool.ac.uk

Abstract

Objectives Knowledge of the extent of variation in outcome assessment for inflammatory bowel disease (IBD) in routine practice is limited. We aimed to describe and quantify variation in outcome coverage and to explore patient, clinician and practitioner factors associated with it.

Design Prospective exploratory mixed-methods study.

Setting IBD clinics at six hospitals in North West England with differing electronic health record (EHR) systems.

Methods Mixed-methods study comprising: (a) structured observations of outcomes elicited during consultations (102 patients consulting 24 clinicians); (b) retrospective analysis of outcomes recorded in the EHR (909 consultations; 127 clinicians) and (c) semistructured interviews with the 24 observed clinicians. We determined whether specific outcome ‘sets’ were elicited or recorded, including: (1) a minimum set of symptom pairs (‘PRO-2’); (2) symptom sets from disease activity indices and (3) a reference list of 37 symptoms, signs and impacts. Factors associated with variation were explored in univariate and multivariate binary logistic regression analyses and from clinician interviews.

Results PRO-2 coverage was not invariable (elicited during 81% of observed consultations; recorded in 56% of EHR) and infrequent for complete activity indices (all domains from Harvey-Bradshaw Index: elicited, 18%; recorded, 5%). The median number of outcomes from the reference list elicited per consultation was 12 (13-fold variation) and recorded in EHR was 7 (>20-fold variation). Symptom quantification (PRO-2) seldom adhered closely to standardised descriptors and an explicit timeframe was defined rarely. PRO-2 recording in EHR was associated with a diagnosis of ulcerative colitis (OR: 2.09 (95% CI 1.15 to 3.80)) and nurse-led consultations (OR: 6.98 (95% CI 3.28 to 14.83)) and a three-way model suggested 26% of total variability lay between clinicians, 17% between patients but the remainder was unexplained. Most clinicians expressed preference for individualised health status evaluations versus standardised outcome assessments.

Conclusions There was little evidence for standardised assessment and recording of IBD outcomes and substantial intra-clinician and inter-clinician variation from one consultation to another. Nurses demonstrated a greater tendency to standardised practice.

  • inflammatory bowel disease
  • quality in healthcare
  • gastroenterology

Data availability statement

Data are available upon reasonable request. The data that support the findings of this study are available from the corresponding author upon reasonable request.

https://creativecommons.org/licenses/by/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/.

https://doi.org/10.1136/bmjopen-2021-056413

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Strengths and limitations of this study

    • Our work provides a unique insight into variation in day-to-day assessment and recording of clinical outcomes during inflammatory bowel disease care delivery in England.

    • In view of the magnitude of variation and lack of standardisation demonstrated by our study, there is an unmet need for evidence-based guidance, policies, education and quality standards to define and address unwarranted heterogeneity in outcome assessment.

    • There are limitations to using quantitative methods to define the complex range of patient, practitioner and hospital factors that may be associated with variation in outcome coverage, thus parallel interviews were undertaken to explore practitioners’ views on outcome assessment.

    • Observations were conducted at six sites within one region of England and findings cannot be generalised to the whole country.

    Introduction

    Disease activity in inflammatory bowel disease (IBD) may be assessed from symptoms, physical signs and laboratory, endoscopic or radiological measures of inflammation.1 Subjective symptoms overlap with other conditions and are neither sensitive nor specific for active bowel inflammation.2 However, more objective measures of intestinal inflammation are invasive, costly and may not be available to clinicians when making treatment decisions.

    Clinical trials have traditionally used physician-reported composite outcome measures. These ‘activity indices’ combine symptoms, signs and some objective parameters, such as the Crohn’s Disease Activity Index (CDAI)3 or Mayo Score.4 However, the precise choice of index has varied across trials.5–7 Furthermore, it is recognised that the requirement for objective test results makes such instruments impractical for routine use at every clinical encounter.

    Several international efforts are seeking better consensus on standardisation of outcome assessment for comparative effectiveness research, including development of core outcome sets (COSs).8 9 In trials, the use of symptom-based end-points as co-primary outcomes alongside more objective measures of inflammation has been proposed.10–12 For the symptom component, there has been the suggestion of adopting simple two-item instruments that aim to capture the dominant symptom ‘pair’ for each condition, along with a shift towards collecting such outcome data directly from patients (‘PRO-2’). The PRO-2 symptom pairs focus on stool frequency and blood in stool for ulcerative colitis (UC)13 and abdominal pain and stool frequency in Crohn’s disease (CD).14 In parallel with work focused on trials, a recent initiative has sought to promote standardisation of outcome assessment for clinical practice to support international benchmarking.15 16

    Despite a renewed focus on standardisation, there are currently no explicit minimum data standards for IBD outcome assessment in routine settings, nor any regulatory requirement to record certain outcomes or to use specific tools. Units providing IBD care in the UK are not formally accredited at present. However, key performance indicators in the national audit of biological therapies have long encouraged the collection of traditional disease activity indices (eg, Harvey-Bradshaw Index (HBI) for CD17 or Simple Clinical Colitis Activity Index (SCCAI) for UC),18 although participation is voluntary.19 UK hospitals still vary widely in digital maturity and choice of electronic health record (EHR) systems which may serve as a barrier to implementing standardised outcome sets for specific conditions.

    Remarkably little is known about how clinicians currently elicit and record clinical outcomes in daily IBD practice, including whether minimum ‘sets’ of core symptoms are captured systematically at each consultation. We propose that it is not unreasonable to expect that all clinical encounters might elicit and record the presence or absence of the ‘PRO-2’ symptoms and yet it is currently unknown whether such minimum symptom sets are covered invariably in routine settings. Nor do we know the extent to which practitioners vary in their approach to quantifying individual outcomes or in their coverage of broader sets of IBD outcomes such as those from traditional disease activity indices or quality of life instruments.

    We hypothesised that marked variation and inconsistency will exist in the eliciting and recording of IBD outcomes. Undue variation in practice may lead to inequalities in care, including differences in thresholds for starting new treatments or in judging their effectiveness. We aimed to describe and quantify this variation in outcome coverage during routine care delivery and examine factors associated with it.

    Methods

    Study overview

    We undertook an exploratory mixed-methods project involving quantitative and qualitative methods at a sample of English hospitals. This included:

    1. Structured observation and audio recording of ‘live’ consultations to investigate outcomes ‘elicited’ during face-to-face interactions.

    2. Retrospective review of EHRs to examine how those outcomes are ‘recorded’ by healthcare professionals.

    3. Qualitative observations and interviews with clinicians to explore their approach to outcome assessment and their views on outcome standardisation.

    Selection of hospital sites, clinicians and patients

    Hospitals

    Six acute hospitals in North West England were selected purposively, reflecting a range of service sizes, digital maturity and different EHR systems, in order to explore the capture of clinical outcomes across varied Information Technology (IT) infrastructures.20 At each site, we determined whether the system contained pre-defined fields for capturing symptom checklists, indices, scores or other outcomes (as opposed to recording such information as unstructured text).

    Clinicians

    Participants were recruited from those delivering care to adult patients with IBD and focused on medically trained physicians and IBD specialist nurses. Clinicians were selected purposively by a local study collaborator at each site, aiming for a minimum of three clinicians per service. Of 25 clinicians invited to participate, 24 agreed. Written, informed consent was obtained. Participants comprised 10 consultant gastroenterologists, 4 gastroenterology specialist trainees and 10 specialist nurses (3–6 clinicians per hospital). The clinicians understood that the research was focused on information elicited and recorded but were not aware of the specific focus on standardised outcome assessment. Sample size calculations could not be undertaken in advance as our project was an exploratory mixed-methods study, aiming to achieve theoretical saturation.21

    Patients

    Patients were recruited from among those attending face-to-face consultations with the 24 participating clinicians. We aimed for a minimum of three patients per clinician, conducting observations over the course of one or more clinics. Inclusion criteria for patients were age ≥18 years, ability to communicate in English (ie, not requiring an interpreter) and a clinical diagnosis of either CD, UC or unclassified IBD (IBD-U). Patients were selected from clinic lists, sent study information before their consultation and approached by the researcher before their appointment. Of 112 patients invited to participate, 10 (9%) declined. Patients provided written consent and none withdrew after the consultation. Electronic medical records of each participating patient were subsequently reviewed to capture demographics, clinical characteristics and outcome data.

    Structured observation and audiorecording of consultations

    One observer (VR, a medical gastroenterologist in training) performed structured observations of 102 clinician–patient consultations between May 2018 and June 2019. Consultations were audiorecorded, transcribed and anonymised. A structured data template was used to guide observations, focusing on outcomes elicited and recorded from pre-specified lists (outcomes sets).

    Definition of outcome sets

    In the absence of a singular, internationally agreed minimum outcome set for IBD, we defined two alternative ‘sets’ of outcomes that might be regarded as candidates for routine coverage during every clinical review. The first sets were based on symptoms pairs from the relevant PRO-2 (stool frequency and abdominal pain for CD; stool frequency and rectal bleeding for UC (or IBD-U)). The second sets were focused on the items from traditional disease activity indices used in clinical trials,5–8 observational studies,22 registries23 and clinical audits.19 For CD, we selected the CDAI and HBI, and for UC we identified the Mayo Score and SCCAI. The outcomes were categorised as either a global assessment of health status, an individual symptom or a physical sign. Our study focused on information that is elicited by clinicians from patients during consultations and so we excluded results of investigations (eg, blood tests). Within the outcome sets from each disease activity index, we also defined subsets of ‘main symptoms’ from within the complete list of all domains.

    To quantify the full breadth of outcome coverage during routine practice, we generated a longer list of potential symptoms, signs and life impacts of IBD that might be actively ruled in or out during a consultation. This reference list enabled us to count the number of ‘relevant’ IBD outcomes covered in each consultation. In addition to symptom pairs and the items from activity indices, we added discrete outcomes from other clinician-reported outcomes (CLIN-ROs) or patient-reported outcomes (PROs) based on existing systematic reviews of outcomes used in clinical trials5–7 and from guidelines for clinical practice.15 24 25 Life impact outcomes were classified using a standardised outcome taxonomy proposed by the Core Outcome Measures in Effectiveness Trials collaboration.26 This resulted in a reference list of 37 outcomes (online supplemental information S1).

    Outcomes elicited during observed consultations

    VR reviewed field notes and audiorecordings for each observed consultation to determine whether each item from our reference list was covered and whether selected symptoms were quantified using standardised descriptors. ‘Eliciting’ a symptom was defined as any explicit mention regardless of formal quantification and included whenever an item was verified as being absent (ie, was actively ‘ruled out’, such as the absence of abdominal pain).

    Quantification of the PRO-2 symptom sets was assessed with reference to standardised descriptors from disease activity indices. Each item was categorised according to whether the clinician used standardised descriptors in accordance with the activity indices from which the PRO-2 was derived),13 14 informal descriptors (non-standardised) or none at all. We also noted whether outcomes were assessed over the relevant timeframe stipulated in the indices (eg, symptoms over the last 24 hours for HBI, or the last 3 days for SCCAI). It was also noted whether a disease activity score was calculated and recorded.

    Outcomes recorded in the EHR

    For each observed consultation, VR subsequently reviewed the local EHR to locate any record of the items from the outcome sets. This included clinical letters, scanned hand-written case notes, electronic data entries and any other clinician-generated information created for the observed encounter. Documentation of coverage and quantification was noted for each relevant item. This process was repeated for up to 10 consecutive previous consultations for the same patient (depending on the number available in the EHR). The details of which clinician recorded each encounter was noted (n=127 practitioners in total).

    Measures of outcomes elicited and recorded

    Table 1 summarises the measures we derived for the outcomes elicited during the 102 directly observed consultations and those recorded in 909 consultation records. In addition to coverage and quantification, we determined the extent of ‘information loss’ for the observed consultations by calculating the number of items from the reference list that were elicited during the consultation but not recorded in the EHR. We also analysed the order in which outcomes were discussed to establish whether a standardised ‘checklist’ approach was used. We defined a checklist as any list of three or more outcomes that were elicited by direct questioning in a fixed order by an individual clinician during ≥2 observed consultations for patients having the same diagnosis.

    View this table:
    Table 1

    List of measures derived from observed consultations (n=102) and from review of consultations recorded in the electronic health record (n=909)

    Determining factors associated with outcome coverage

    In order to explore factors associated with coverage of minimum sets of outcomes, we defined a series of patient, practitioner and site level variables. Variables were defined a priori based on clinical significance and to control for potential confounders, further informed by clinician interviews. Patients with a stoma were excluded where appropriate.

    Statistics

    Descriptive data for the main metrics of outcome coverage are presented as percentages of consultations and variation expressed as the range of values (low to high). We used random effects multivariable logistic regression to explore factors associated with outcome set coverage. For observed consultations, a two-level model was applied (to control for repeated consultations per clinician but with only one observation per patient). A three-level model was used for clinical records (to control for repeated records per patient and clinician). We used the random effect models to estimate the percentage of variation attributed to patient or clinician levels.

    One-to-one interviews with IBD clinicians

    A single interviewer (VR) conducted face-to-face qualitative interviews with each of the 24 IBD clinicians to explore influences on their approaches to outcome assessment. Interviews were semistructured and informed by an interview guide (see online supplemental information S2). Interviews lasted 25 min on average, ranging from 17 to 35 min. They were digitally audiorecorded, transcribed and anonymised. Qualitative data analysis was iterative and ongoing throughout the study drawing on thematic approaches, aided by Nvivo software package. Our work was informed by the literature on quality and rigour in qualitative research27 although we recognise that procedures alone do not guarantee quality.28

    Integration of quantitative and qualitative data

    Structured observations and interviews with clinicians were conducted in parallel, allowing for simultaneous analysis of observed practices and self-reported narratives. Interim quantitative analysis of outcomes collected during observed consultations was performed after 50 observations, informing adjustments to the interview guide for the remainder of data collection (online supplemental information S3). Retrospective review of health records was performed after completion of observations and interviews, and informed further qualitative analysis to triangulate and interpret quantitative findings.

    Patient and public involvement

    Our study design was discussed with, and approved by, the IBD patient panel at the hospital sponsoring the study.

    Results

    Characteristics of participating hospitals, clinicians and patients

    Hospitals

    Centres varied widely in population size served, compliment of medical and nursing staff delivering IBD services and digital maturity of IT systems (table 2). Three sites were global digital exemplars according to the Digital Maturity Assessment for English Trusts.20 However, at all sites, the main consultation record was a letter to the patient’s general practitioner. Letters were semistructured but variable and none contained a pre-defined symptom checklist, nor a designated field for a disease activity index score. Documentation of individual outcomes was generally within unstructured free text narratives. At two sites, outcomes or disease activity indices could be recorded electronically in a designated part of the EHR, whereas at other centres such information might be captured in hand-written entries (scanned notes). At one site, there was an electronic system for patients to directly record outcomes via a web-based application (portal), although none of the patients studied used this system.

    View this table:
    Table 2

    Characteristics of participating hospital sites and their IBD services, number of directly observed consultations and number of electronic health record reviews

    Clinicians

    At least one doctor and one specialist nurse were directly observed at each site with a minimum of three consultations per practitioner (table 2). Retrospective review of the EHR for the participating patients included records created by 127 clinicians in total (at least 15 per site), covering a wide range of practitioners.

    Patients

    The sample contained equal representation with respect to sex (48% men) and diagnosis (49% CD) and a broad range of ages (18–84 years), disease classification and treatment history (table 3). Reflecting the location of the hospitals and demographics of the local population, the ethnicity of the patients was almost exclusively white (97%) and mostly living in urban areas. The mean duration of disease was 13 years and approximately one-third were on immunomodulators or biologic therapies.

    View this table:
    Table 3

    Demographic and clinical characteristics of participating patients (n=102) at the time of the observed consultations (‘Observed’) and for consultations reviewed in their electronic health record (‘Records’)

    Coverage of PRO-2 symptom pairs

    Analysis of the relative frequency of coverage of the 37 outcomes on our reference list confirmed that the PRO-2 symptom pairs were the ‘top two’ (most frequently) elicited and recorded gastrointestinal symptoms (figure 1). Although coverage of such symptom pairs was very common, it was not invariable in routine practice, being elicited in 81% of observed consultations but recorded in only 71% of records for those visits. In the larger scale review of records made by 127 practitioners, the presence or absence of a PRO-2 was noted in just 56% of IBD consultations (table 4).

    Figure 1
    Figure 1

    Frequency (%) of individual inflammatory bowel disease (IBD) outcomes that were: (A, left panel) elicited during observed consultations with 102 patients; or (B, right panel) recorded in the electronic health records of 909 consultations with the same patients. *Significant difference in frequency between Crohn’s disease and ulcerative colitis/IBD-U (p<0.05); 1Patients with a stoma excluded. IBD-U, inflammatory bowel disease-unclassified.

    View this table:
    Table 4

    Frequency of eliciting and recording relevant PRO-2 symptom pairs for Crohn’s disease and ulcerative colitis/IBD-U during routine clinical practice at six IBD centres

    At the level of individual practitioners, the percentage of PRO-2 elicited during observed encounters ranged from 20% to 100% of consultations. Aggregated rates were 75% for doctors versus 94% for nurses. Eight out of 10 nurses (80%) but only 7 out of 14 doctors (50%) elicited PRO-2 in every observed consultation. Within the EHRs created for the 102 observed consultations, PRO-2 coverage ranged from 0% to 100% at practitioner level. Again, the aggregated figures suggested a systematic difference between doctors and nurses (58% vs 91%). One in five nurses (20%) recorded symptom pairs for every observed consultation, compared with one in seven doctors (14%). None of the factors studied in two-level modelling of the 102 observed consultations were significantly associated with eliciting PRO-2 but this exploratory analysis was based on low sample size (data not shown).

    However, with the benefit of a large sample size of clinicians (n=127) and consultations (n=909), we were better able to explore factors associated with PRO-2 recording using a three-level model and derive estimates for the relative contributions of clinician as opposed to patient-related factors (table 5). The likelihood of finding the appropriate pair of symptoms recorded in the EHR was independently associated with a diagnosis of UC (OR: 2.09 (95% CI 1.15 to 3.80)) and nurse-led consultations (OR: 6.98 (95% CI 3.28 to 14.83)). In this three-level model (containing multiple records per patient), 26% of total variability lay between clinicians, 17% between patients and 57% of variation remained unexplained. This confirmed that clinicians from a nursing background were significantly more likely to make a formal record of a relevant symptom pair than doctors.

    View this table:
    Table 5

    Factors associated with recording the relevant PRO-2 symptom pair in the electronic health record (EHR) following consultations for inflammatory bowel disease

    Quantification of PRO-2 symptoms

    To study clinical practice with respect to quantifying symptoms using standardised descriptors, we focused on the PRO-2 items, namely stool frequency (both CD and UC), severity of abdominal pain (CD) and severity of rectal bleeding (UC).

    Stool frequency

    Overall, some form of stool count (per day) was elicited in 76 of 86 eligible consultations (88%). However, evaluation over a clearly defined time interval was rare. For patients with CD, eliciting stool frequency for the previous 24 hours (HBI) occurred in just 3 of 32 relevant consultations (9%) and assessment over 7 days (CDAI) in just one case (3%). For UC/IBD-U, an assessment over 3 days (Mayo Score) was never observed in 44 relevant encounters, whereas a timeframe of 7 days was seen in one case (2%). Within the EHR, rates of recording stool frequency during a standardised timeframe were very low for both forms of IBD. For CD, the rates of recording a 24-hour or 7-day timeframe were, respectively, 4% and 2% (n=253 records). For UC/IBD-U, corresponding rates for 3-day or 7-day assessment periods were 1% in both cases (n=281 records). Interestingly, lack of specification of a precise time period was observed even in consultations where a disease activity score was generated, suggesting such scores were not strictly valid. Of note, stool frequency was specified relative to normal, as defined in the Mayo Score, in just half of consultations where this outcome was elicited or recorded (56% consultations and 49% records).

    Abdominal pain severity

    Pain was verified as being an active symptom in 27 observed consultations of patients with a diagnosis of CD, with severity quantified using standardised descriptors (mild, moderate or severe) in only 15 (55.5%). Non-standard descriptors of severity were used in a further seven (26%), leaving five with no observed discussion of severity (18.5%). Assessment over a standardised timeframe was rare (over last 7 days, as per CDAI, in one case (4%); last 24 hours, as per HBI, in two cases (7%)). There were 160 EHRs of consultations for CD where abdominal pain was noted to be an active symptom, with only 41 (26%) using the standardised descriptors, 71 (44%) using non-standard descriptors and 48 (30%) having no record of severity. None of these records indicating that pain was assessed over a specific time period.

    Blood in stool

    This was an active symptom covered in 20 observed consultations for UC/IBD-U but bleeding severity was quantified using standardised descriptors from the Mayo Score during only six of those encounters (30%) and those from the SCCAI in just seven cases (35%). Similarly in the EHR, recording of quantification of bleeding severity within 134 relevant records was identified, respectively, in 24 for Mayo (18%) and 29 for SCCAI (22%).

    Coverage of outcome sets from the disease activity indices

    Complete coverage of all symptom and sign domains required for disease activity indices was rare, although coverage of just the symptom items was more common (table 6). Rates were highest for the symptoms from the HBI for CD (76% of observed consultations, 31% of records) and for the simple partial Mayo Score for UC (46% and 31%, respectively).

    View this table:
    Table 6

    Frequency of eliciting sets of symptoms and signs from relevant disease activity indices for Crohn’s disease and ulcerative colitis during observed consultations and of recording them in the electronic health record

    Only 7 of the 24 clinicians (three nurses and four doctors) ever collected or recorded a complete disease activity index (HBI or SCCAI). Interestingly, only 4 of 38 (10.5%) clinical records of consultations where biologic therapies were initiated had a disease activity index recorded in the EHR, despite a formal assessment of disease activity being a key performance indicator for the UK Biologics Audit.19

    As with PRO-2, exploratory models identified no significant patient, practitioner or site level factors associated with eliciting relevant symptom items of selected indices (HBI and SCCAI) within the 102 observed consultations (data not shown). However, in the three-way models of EHRs, the likelihood of finding the main set of symptoms from HBI captured in CD records was independently associated with hospitals with a high degree of digital maturity (OR: 3.09 (95% CI 1.14 to 8.37), online supplemental information S4) and was almost significant for nurse-led consultations (OR: 2.34 (95% CI 0.98 to 5.60)). With regards to records for UC/IBD-U, there was a significant association for nurses recording the main symptoms from SCCAI (OR 20.15 (95% CI 3.82 to 106.33), online supplemental information S5).

    Coverage of outcomes from the complete reference list of symptoms, signs and impacts

    Next, we measured the extent to which the presence or absence of our reference list of 37 items were covered systematically during consultations and explored variation in this ‘breadth’ of coverage from one consultation to another.

    All the items were elicited during at least one observed encounter, confirming their relevance to IBD practice. Figure 1 shows the relative frequencies of eliciting individual outcomes during observed consultations (A) or recording them within the EHR (B), stratified by diagnosis. ‘General well-being’ was almost invariably mentioned during observed consultations. Most elicited outcomes were symptoms (75%), followed by life impact outcomes (19%) and physical signs (6%). As expected, figure 1 shows that the frequency of eliciting and recording certain individual symptoms was statistically different between the two main forms of IBD, most notably the coverage of blood in stool, mucus in stool and nocturnal stools more frequently in UC. Nevertheless, even these items were not covered at all encounters with UC cases. With respect to CD phenotype, abdominal pain was covered more frequently in consultation records for patients with stricturing as opposed to non-stricturing disease (65% vs 55%; p=0.001), although this still left a third of visits with no explicit record of the presence or absence of abdominal pain.

    A median of 12 outcomes (IQR 8–14) were elicited during the 102 observed consultations, with no statistical difference between CD and UC/IBD-U (12 (7–14) vs 12 (9–14)). This represents only one-third of potential outcomes covered in a typical consultation. Remarkably, there was 13-fold variation in the number of outcomes discussed, ranging from 2 to 26.

    A median of 8 (IQR 4–11) outcomes were recorded in the EHR following an observed consultation, comprising just under a fifth of the list and there was 21-fold variation between the records (range: 1–21; with a small minority recording none of outcomes on the list, n=3). The mean ‘information loss’ between outcomes discussed and those subsequently captured in the EHR for the same consultation was 3.45 outcomes, with mean 2.37 symptoms not captured in EHR per consultation.

    In our large-scale review of 909 EHRs generated by 127 clinicians, there was a median of 7 (IQR: 4–10) outcomes per visit recorded overall, with 7 (4–10) for CD and 6 (4–9) for UC/IBD-U. This constitutes less than a fifth of the pre-specified list. Again, there was substantial variation between consultations (range: 0–23). Furthermore, when we calculated the cumulative count of outcomes recorded for each patient over the preceding year from the observed clinical encounter, this was also relatively low, comprising only a third of pre-specified outcomes (median 13 (9.75–17) over median 3 consecutive clinic visits (2–7)), with over 13-fold variation (range 2–27).

    Most clinicians showed a high degree of variability in outcome coverage from case to case. Using the large sample of EHRs, we aggregated data for multiple consultations between the same patient and practitioner. While showing variation within and between such patient–practitioner pairs (figure 2), this also revealed that certain practitioners tended to record a consistently high (Nurse 1), medium (Doctor 8) or low (Doctor 14) number of outcomes from case to case, pointing to potential systematic differences in their practice.

    Figure 2
    Figure 2

    Number of outcomes recorded in the electronic health record for consecutive consultations by the same practitioner with an individual patient. The number of consultation records ranges from 5 to 9 per practitioner–patient pair. Boxplots show median, IQR, minimum–maximum and outlier values of outcomes covered from the pre-specified list. Doctors (D), nurses (N), patients with ulcerative colitis (UC) and those with Crohn’s disease (CD) are numbered. Hence N1-CD8 indicates consecutive consultations of nurse number 1 with Crohn’s disease patient number 8. Boxplots for selected practitioners are coloured. Nurse 1 (light grey) recorded a consistently high median number of outcomes, Doctor 8 (dark grey) showed more variability within patients and Doctor 14 (black) recorded a consistently low median number.

    Coverage of three or more symptoms in a standardised sequence of questions

    To further explore whether each observed clinician asked questions in a standardised sequence, we identified whether there was evidence for eliciting any group of three or more symptoms in a fixed order in more than one observed consultation. The use of an ordered checklist was observed in only 7 (14%) consultations for CD and 16 (31%) for UC, all of which were conducted by nurses. The standardised sequences consisted of a mean of 6 (range: 3–11) questions. Median number of outcomes elicited in consultations where a standardised sequence of questions was used was higher compared with consultation without a checklist approach (14 vs 11, p<0.001).

    Practitioners’ views on the variation in outcome coverage in routine practice

    To further interpret variability in outcome coverage demonstrated by descriptive analyses, we used qualitative approaches to explore characteristics of personal practices that were not amenable to quantitative enquiry. During observations of IBD consultations, we witnessed wide variation in time, structure and content of encounters, and observed a range of individualised assessments of health status conducted by practitioners. In interviews, clinicians commonly described the need to tailor health status assessment to an individual patient, as opposed to adopting a standardised approach. However, a minority of participants advocated the use of a personal ‘checklist’ when eliciting symptoms to ensure all relevant items were covered.

    Clinician interviewees identified a range of patient, clinician and hospital factors contributing to variation in outcome coverage which are summarised in online supplemental information S6. Interviewees attributed variation to their initial impressions of overall patient’s health and disease activity (Quotes 1–2), treatment compliance (Q3), availability of objective tests to indicate active or inactive disease (Q4) or perceptions of patient’s personality (Q5–7). Practitioners’ training background, skills and experience were also reported as important sources of variation (Q8–12). Many practitioners described that specialist nurses were more thorough (or less selective) and systematic in their assessments, and therefore actively covered more outcomes than doctors (Q11–12). Moreover, clinicians described specialist IBD clinics and local hospital EHR as helping to support the collection of consistent, structured outcomes as these served as a prompt and facilitated capture (Q16–19).

    Discussion

    Our study demonstrated that coverage of a basic minimum symptom set (PRO-2 pairs) was a common but not an invariable part of routine IBD practice, that capture of complete disease activity indices was relatively rare and that the quantification of individual outcomes did not adhere closely to standardised descriptors. There was substantial inter-individual and intra-individual variation among practitioners in eliciting IBD outcomes and a significant deficit in the elicited information being then recorded in the EHR. Clinicians tailored health status evaluations to individual patients, expressed preference for personalised assessments and defined a range of patient, clinician and hospital factors contributing to variation in outcome coverage.

    While some degree of variation was expected due to heterogeneity of clinical characteristics from one case to the next, the magnitude of inter-practitioner variation is very large and appears unexplained by legitimate clinical factors. Our findings are consistent with previously described heterogeneity in the clinical outcomes and quality measures found in routine EHR,29 and the low rates of standardised disease activity information captured by the National Audit of Biological Therapies in the UK.19 Poor standardisation in practice may partly reflect a current lack of consensus on the most suitable CLIN-ROs and PROs for clinical trials.5–7 It is noteworthy in the case of IBD that explicit recommendations for ‘minimum’ clinical outcome sets are largely lacking from specialist guidelines of the American College of Gastroenterology,30 31 European Crohn’s and Colitis Organisation32 and British Society of Gastroenterology.25 Standards for outcome assessment are not covered by current UK and US quality standards.33 34 We believe our findings are in keeping with lack of standardisation of outcome assessment in other disease areas, as little research exists on variation in capturing outcomes in other conditions, despite the potential major implications for inequality in decision-making and health outcomes.

    However, it must be noted that the benefits of greater standardisation and impact on individual health outcomes remain unknown. There is no current evidence to suggest that adoption of a uniform, standardised approach to outcome assessment delivers better results for patients with IBD than more flexible, unstructured practice. Future research is needed to study inequalities associated with variation in clinical practice, including impact on therapeutic decision-making, treatment access and health outcomes. If standardisation of outcome assessment is to be embedded in routine practice, education, training and regulatory oversight will be needed to transform clinicians’ workflows, as well as additional resource and IT capabilities to enable outcome capture.

    Our study showed that IBD specialist nurses recorded outcome sets more frequently than doctors, and demonstrated greater efforts to standardise routine assessments of health status. This highlights an important role of IBD nurses in capturing standardised outcomes as part of the care delivery process, and suggests that their contribution to routine data collection could be further used.

    Our observations suggest that the use of traditional indices and scoring systems are seldom part of the process of making individual patient decisions in routine care. This highlights the divergence between clinical trials and practice. It is recognised that Randomised Controlled Trials do not reflect the selection of patients, nor the approach to judging treatment effectiveness in day-to-day practice.35 This highlights the need for a debate between researchers, clinical experts and patients on optimal ways to develop and implement COS both for clinical trials and practice. Future pragmatic studies are needed to address the evidence gap between traditional clinical trials and the process of routine decision-making. In the interim, there is an unmet need for an explicit set of minimum standards for IBD outcome assessment in routine settings in the UK and worldwide, and audit and quality improvement initiatives to evaluate any benefits of associated changes to clinical practice. Future research is needed to study inequalities associated with variation in clinical practice, including impact on therapeutic decision-making, treatment access and health outcomes.

    Our findings also reveal major challenges for utilising routinely recorded outcome data for clinical trials and observational research. Lack of routine collection of standardised clinician-reported end-points as part of the care delivery process is a barrier to undertaking large scale analysis of real-world patient outcomes in the UK, including initiatives such as UK IBD Registry,23 Health Data Research UK IBD Hub36 and IBD BioResource.37 Leveraging routinely captured outcome data at scale from real-world settings may require computational methods to interrogate varied terminologies and unstructured data. Our present work provides a starting point for building a lexicon of terms and phrases to support Natural Language Processing approaches.38

    PRO measures are increasingly seen as a practical alternative to reliance on composite activity measures generated by clinicians. This is particularly relevant in the face of the current COVID-19 pandemic which has disrupted traditional face-to-face care and enforced the move to remote consultations, with reduced access and/or willingness to undergo objective testing. Hence clinical assessments have become even more valuable, and adoption of virtual consulting and new models of care have been accelerated.39 The COVID-19 pandemic has also highlighted the need for better solutions for remote disease monitoring, and investment in programmes, technologies and infrastructure to allow electronic capture of PRO measures and integration of patient-reported data into operational records.

    Our study has some inevitable limitations in terms of generalisability. The scope was limited to six centres and one region of England and we observed only 102 consultations directly. Undertaking structured observations and analysis of ‘live’ consultations is a resource-intensive exercise. However, we included a range of sites (from smaller district general hospitals to tertiary centres) and a wide spectrum of practitioners. Given the high degree of practice variation demonstrated, we believe it is unlikely that such findings are confined to the centres studied. On the contrary, our results are even more noteworthy given that clinicians volunteered to be observed by a researcher, and may have adopted a more structured approach than their usual practice due to the Hawthorne Effect from direct observation.

    In conclusion, we have shown substantial variability in the breadth, depth and quantification of IBD clinical outcomes during routine clinical assessments. Evidence-based policies, education, quality standards and audit are needed to define and address unwarranted variations in outcome assessment. Future efforts are required to converge clinical IBD activity assessments between clinical trials and practice. Direct capture of PROs using validated instruments is likely to provide a more feasible approach to capturing standardised outcomes as part of the care delivery process.

    Data availability statement

    Data are available upon reasonable request. The data that support the findings of this study are available from the corresponding author upon reasonable request.

    Ethics statements

    Patient consent for publication

    Ethics approval

    This study involves human participants and was approved by The study was granted by the NHS Research Ethics Committee (REC) approval (18/SC/0232). Participants gave informed consent to participate in the study before taking part.

    Acknowledgments

    We would like to record our appreciation to the local study collaborators for their support and facilitation of this study: Dr Ash Bassi, Dr Ian London, Dr Paul Flanagan, Dr Sreedhar Subramanian and Professor John McLaughlin.

    References

    1. ↵
      1. Peyrin-Biroulet L,
      2. Panés J,
      3. Sandborn WJ, et al
      . Defining disease severity in inflammatory bowel diseases: current and future directions. Clin Gastroenterol Hepatol 2016;14:e17:348–54. doi:10.1016/j.cgh.2015.06.001pmid:http://www.ncbi.nlm.nih.gov/pubmed/26071941
      OpenUrlPubMed
    2. ↵
      1. Gracie DJ,
      2. Williams CJM,
      3. Sood R, et al
      . Poor correlation between clinical disease activity and mucosal inflammation, and the role of psychological comorbidity, in inflammatory bowel disease. Am J Gastroenterol 2016;111:541–51.doi:10.1038/ajg.2016.59pmid:http://www.ncbi.nlm.nih.gov/pubmed/27002800
      OpenUrlCrossRefPubMed
    3. ↵
      1. Best WR,
      2. Becktel JM,
      3. Singleton JW, et al
      . Development of a Crohn's disease activity index. National cooperative Crohn's disease study. Gastroenterology 1976;70:439–44.pmid:http://www.ncbi.nlm.nih.gov/pubmed/1248701
      OpenUrlPubMedWeb of Science
    4. ↵
      1. Schroeder KW,
      2. Tremaine WJ,
      3. Ilstrup DM
      . Coated oral 5-aminosalicylic acid therapy for mildly to moderately active ulcerative colitis. A randomized study. N Engl J Med 1987;317:1625–9.doi:10.1056/NEJM198712243172603pmid:http://www.ncbi.nlm.nih.gov/pubmed/3317057
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Ma C,
      2. Panaccione R,
      3. Fedorak RN, et al
      . Heterogeneity in definitions of endpoints for clinical trials of ulcerative colitis: a systematic review for development of a core outcome set. Clin Gastroenterol Hepatol 2018;16:637-647.e13.doi:10.1016/j.cgh.2017.08.025pmid:http://www.ncbi.nlm.nih.gov/pubmed/28843356
      OpenUrlPubMed
    6. ↵
      1. Ma C,
      2. Hussein IM,
      3. Al-Abbar YJ, et al
      . Heterogeneity in Definitions of Efficacy and Safety Endpoints for Clinical Trials of Crohn's Disease: A Systematic Review. Clin Gastroenterol Hepatol 2018;16:e22:1407–19. doi:10.1016/j.cgh.2018.02.051pmid:http://www.ncbi.nlm.nih.gov/pubmed/29596987
      OpenUrlPubMed
    7. ↵
      1. Catt H,
      2. Hughes D,
      3. Kirkham JJ, et al
      . Systematic review: outcomes and adverse events from randomised trials in Crohn's disease. Aliment Pharmacol Ther 2019;49:978–96.doi:10.1111/apt.15174pmid:http://www.ncbi.nlm.nih.gov/pubmed/30828852
      OpenUrlPubMed
    8. ↵
      1. Sahnan K,
      2. Tozer PJ,
      3. Adegbola SO, et al
      . Developing a core outcome set for fistulising perianal Crohn's disease. Gut 2019;68:226–38.doi:10.1136/gutjnl-2017-315503pmid:http://www.ncbi.nlm.nih.gov/pubmed/29437911
      OpenUrlAbstract/FREE Full Text
    9. ↵
      1. Ma C,
      2. Panaccione R,
      3. Fedorak RN, et al
      . Development of a core outcome set for clinical trials in inflammatory bowel disease: study protocol for a systematic review of the literature and identification of a core outcome set using a Delphi survey. BMJ Open 2017;7:e016146. doi:10.1136/bmjopen-2017-016146pmid:http://www.ncbi.nlm.nih.gov/pubmed/28601837
      OpenUrlPubMed
    10. ↵
      1. Dulai PS,
      2. Jairath V,
      3. Khanna R, et al
      . Development of the symptoms and impacts questionnaire for Crohn's disease and ulcerative colitis. Aliment Pharmacol Ther 2020;51:1047–66.doi:10.1111/apt.15726pmid:http://www.ncbi.nlm.nih.gov/pubmed/32319120
      OpenUrlPubMed
    11. ↵
      1. Higgins PDR,
      2. Harding G,
      3. Leidy NK, et al
      . Development and validation of the Crohn’s disease patient-reported outcomes signs and symptoms (CD-PRO/SS) diary. J Patient Rep Outcomes 2018;2:24.doi:10.1186/s41687-018-0044-7
      OpenUrl
    12. ↵
      1. Higgins PDR,
      2. Harding G,
      3. Revicki DA, et al
      . Development and validation of the ulcerative colitis patient-reported outcomes signs and symptoms (UC-pro/SS) diary. J Patient Rep Outcomes 2017;2:26.doi:10.1186/s41687-018-0049-2pmid:http://www.ncbi.nlm.nih.gov/pubmed/29888745
      OpenUrlPubMed
    13. ↵
      1. Jairath V,
      2. Khanna R,
      3. Zou GY, et al
      . Development of interim patient-reported outcome measures for the assessment of ulcerative colitis disease activity in clinical trials. Aliment Pharmacol Ther 2015;42:1200–10.doi:10.1111/apt.13408pmid:http://www.ncbi.nlm.nih.gov/pubmed/26388424
      OpenUrlCrossRefPubMed
    14. ↵
      1. Khanna R,
      2. Zou G,
      3. D'Haens G, et al
      . A retrospective analysis: the development of patient reported outcome measures for the assessment of Crohn's disease activity. Aliment Pharmacol Ther 2015;41:77–86.doi:10.1111/apt.13001pmid:http://www.ncbi.nlm.nih.gov/pubmed/25348809
      OpenUrlCrossRefPubMed
    15. ↵
      1. Kim AH,
      2. Roberts C,
      3. Feagan BG, et al
      . Developing a standard set of patient-centred outcomes for inflammatory bowel Disease—an international, Cross-disciplinary consensus. Journal of Crohn's and Colitis 2018;12:408–18.doi:10.1093/ecco-jcc/jjx161
      OpenUrl
    16. ↵
      1. Bodger K,
      2. Ormerod C,
      3. Shackcloth D, et al
      . Development and validation of a rapid, generic measure of disease control from the patient's perspective: the IBD-control questionnaire. Gut 2014;63:1092–102.doi:10.1136/gutjnl-2013-305600pmid:http://www.ncbi.nlm.nih.gov/pubmed/24107590
      OpenUrlAbstract/FREE Full Text
    17. ↵
      1. Harvey RF,
      2. Bradshaw JM
      . A simple index of Crohn's-disease activity. Lancet 1980;1:514. doi:10.1016/s0140-6736(80)92767-1pmid:http://www.ncbi.nlm.nih.gov/pubmed/6102236
      OpenUrlPubMedWeb of Science
    18. ↵
      1. Walmsley RS,
      2. Ayres RC,
      3. Pounder RE, et al
      . A simple clinical colitis activity index. Gut 1998;43:29–32.doi:10.1136/gut.43.1.29pmid:http://www.ncbi.nlm.nih.gov/pubmed/9771402
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. Royal College of Physicians IBD Programme
      . National clinical audit of biological therapies. Annual report, 2016. Available: https://www.rcplondon.ac.uk/projects/outputs/national-clinical-audit-biological-therapies-annual-report-2016 [Accessed 21 Sep 2020].
    20. ↵
      1. NHS England
      . Digital maturity assessment, 2016. Available: https://www.england.nhs.uk/digitaltechnology/info-revolution/maturity-index [Accessed 21 Sep 2020].
    21. ↵
      1. Hennink MM,
      2. Kaiser BN,
      3. Marconi VC
      . Code saturation versus meaning saturation: how many interviews are enough? Qual Health Res 2017;27:591–608.doi:10.1177/1049732316665344pmid:http://www.ncbi.nlm.nih.gov/pubmed/27670770
      OpenUrlCrossRefPubMed
    22. ↵
      1. Christensen B,
      2. Colman RJ,
      3. Micic D, et al
      . Vedolizumab as induction and maintenance for inflammatory bowel disease: 12-month effectiveness and safety. Inflamm Bowel Dis 2018;24:849–60.doi:10.1093/ibd/izx067pmid:http://www.ncbi.nlm.nih.gov/pubmed/29562271
      OpenUrlPubMed
    23. ↵
      1. Royal College of Physicians, British Society of Gastroenterology, and Crohn’s & Colitis UK
      . Uk IBD registry. Available: https://ibdregistry.org.uk/ [Accessed 21 September 2020].
    24. ↵
      1. Walsh AJ,
      2. Bryant RV,
      3. Travis SPL
      . Current best practice for disease activity assessment in IBD. Nat Rev Gastroenterol Hepatol 2016;13:567–79.doi:10.1038/nrgastro.2016.128pmid:http://www.ncbi.nlm.nih.gov/pubmed/27580684
      OpenUrlCrossRefPubMed
    25. ↵
      1. Lamb CA,
      2. Kennedy NA,
      3. Raine T, et al
      . British Society of gastroenterology consensus guidelines on the management of inflammatory bowel disease in adults. Gut 2019;68:s1–106.doi:10.1136/gutjnl-2019-318484pmid:http://www.ncbi.nlm.nih.gov/pubmed/31562236
      OpenUrlAbstract/FREE Full Text
    26. ↵
      1. Dodd S,
      2. Clarke M,
      3. Becker L, et al
      . A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. J Clin Epidemiol 2018;96:84–92.doi:10.1016/j.jclinepi.2017.12.020pmid:http://www.ncbi.nlm.nih.gov/pubmed/29288712
      OpenUrlCrossRefPubMed
    27. ↵
      1. Levitt HM,
      2. Motulsky SL,
      3. Wertz FJ, et al
      . Recommendations for designing and reviewing qualitative research in psychology: promoting methodological integrity. Qualitative Psychology 2017;4:2–22.doi:10.1037/qup0000082
      OpenUrl
    28. ↵
      1. Barbour RS
      . Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 2001;322:1115–7.doi:10.1136/bmj.322.7294.1115pmid:http://www.ncbi.nlm.nih.gov/pubmed/11337448
      OpenUrlFREE Full Text
    29. ↵
      1. Afzali A,
      2. Ciorba MA,
      3. Schwartz DA, et al
      . Challenges in using real-world clinical practice records for validation of clinical trial data in inflammatory bowel disease: lessons learned. Inflamm Bowel Dis 2017;24:2–4.doi:10.1093/ibd/izx015pmid:http://www.ncbi.nlm.nih.gov/pubmed/29272481
      OpenUrlPubMed
    30. ↵
      1. Lichtenstein GR,
      2. Loftus EV,
      3. Isaacs KL, et al
      . ACG Clinical Guideline: Management of Crohn’s Disease in Adults. Am J Gastroenterol 2018;113:481–517.doi:10.1038/ajg.2018.27
      OpenUrlCrossRefPubMed
    31. ↵
      1. Rubin DT,
      2. Ananthakrishnan AN,
      3. Siegel CA, et al
      . Acg clinical guideline: ulcerative colitis in adults. American Journal of Gastroenterology 2019;114:384–413.doi:10.14309/ajg.0000000000000152
      OpenUrlCrossRefPubMed
    32. ↵
      1. Sturm A,
      2. Maaser C,
      3. Calabrese E, et al
      . ECCO-ESGAR guideline for diagnostic assessment in IBD Part 2: IBD scores and general principles and technical aspects. Journal of Crohn's and Colitis 2019;13:273–84.doi:10.1093/ecco-jcc/jjy114
      OpenUrl
    33. ↵
      1. Kapasi R,
      2. Glatter J,
      3. Lamb CA, et al
      . Consensus standards of healthcare for adults and children with inflammatory bowel disease in the UK. Frontline Gastroenterol 2020;11:178–87.doi:10.1136/flgastro-2019-101260pmid:http://www.ncbi.nlm.nih.gov/pubmed/32419908
      OpenUrlAbstract/FREE Full Text
    34. ↵
      1. Melmed GY,
      2. Siegel CA,
      3. Spiegel BM, et al
      . Quality indicators for inflammatory bowel disease: development of process and outcome measures. Inflamm Bowel Dis 2013;19:662–8.doi:10.1097/mib.0b013e31828278a2pmid:http://www.ncbi.nlm.nih.gov/pubmed/23388547
      OpenUrlCrossRefPubMedWeb of Science
    35. ↵
      1. Levesque BG,
      2. Sandborn WJ,
      3. Ruel J, et al
      . Converging goals of treatment of inflammatory bowel disease from clinical trials and practice. Gastroenterology 2015;148:37–51.doi:10.1053/j.gastro.2014.08.003pmid:http://www.ncbi.nlm.nih.gov/pubmed/25127678
      OpenUrlCrossRefPubMed
    36. ↵
      1. UK Research and Innovation
      . Health data research hubs, 2019. Available: https://www.hdruk.ac.uk/infrastructure/the-hubs/ [Accessed 21 September 2020].
    37. ↵
      1. Parkes M, IBD BioResource Investigators
      . IBD BioResource: an open-access platform of 25 000 patients to accelerate research in Crohn's and Colitis. Gut 2019;68:1537–40.doi:10.1136/gutjnl-2019-318835pmid:http://www.ncbi.nlm.nih.gov/pubmed/31270165
      OpenUrlFREE Full Text
    38. ↵
      1. Nehme F,
      2. Feldman K
      . Evolving role and future directions of natural language processing in gastroenterology. Dig Dis Sci 2021;66:29-40. doi:10.1007/s10620-020-06156-ypmid:http://www.ncbi.nlm.nih.gov/pubmed/32107677
      OpenUrlPubMed
    39. ↵
      1. Dobrusin A,
      2. Hawa F,
      3. Gladshteyn M, et al
      . Gastroenterologists and patients report high satisfaction rates with telehealth services during the novel coronavirus 2019 pandemic. Clin Gastroenterol Hepatol 2020;18:2393–7.doi:10.1016/j.cgh.2020.07.014pmid:http://www.ncbi.nlm.nih.gov/pubmed/32663521
      OpenUrlPubMed

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

    Footnotes

    • Twitter @Violeta_Raz

    • Contributors VR and KB wrote the article; VR, KB, PRW and BY designed the research; VR performed the research; VR and CK analysed the data. All authors approved the final version of the article, including the authorship list. KB is guarantor for the work.

    • Funding This work was supported by the Medical Research Council (MRC) Hubs for Trials Methodology Research (HTMR) Network (grant number MR/L004933/2).

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.

    • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.