The medical knowledge system

Participants all drew from scientific knowledge pertaining to medical facts about the patient’s condition and specific comorbidities. For example, the decision about extubation is affected by the presence of T18, which involves central nervous system disturbances (including disorganised breathing). Participants referred to knowledge regarding how the relevant features of Siri’s case could potentially have interaction effects (e.g., comorbidities) and how these interactions may influence their risk assessment. Several participants referred to looking into the academic literature (e.g., clinical guidelines, evidence from clinical trials, reviews) to inform their judgement as well as their own experiential knowledge working with similar cases. Participants also noted the limitations of knowledge, such as the uncertainty about T18’s relative contribution to extubation risk, the recognition that complexity and uncertainty are related and how sometimes patients surprise clinicians (e.g., having an unexpected positive or negative clinical response).

Context

The bedside examination was essential for all participants, citing the observation of a previous extubation attempt, observing interactions on the ventilator and frequent patient checks as valued information contributing to their decision. Participants articulated that the bedside examination was the most reliable form of knowledge—as measurements and metrics become less proximal and more distal to the patient, they were perceived as less reliable.

The goals of care for the patient were the guideposts for medical judgement, as decided upon by Siri’s parents (We refer to family factors here in recognition of Siri’s age—as an infant, Siri would not have decisional capacity. This category of considerations, however, can readily be abstracted to include the goals of care as decided on by a capable patient themselves, or by a surrogate decision-maker in conjunction with a non-capable patient. Notably, we describe a shared decision-making process, where the decision itself is reached through a reciprocal process between clinicians and families, recognising that a legal surrogate is the actual decision-maker). Participants noted that families in the same situation might legitimately make different choices (eg, non-interventional plan of care or curative). Participants in paediatrics remarked that such decisions are subject to the best interests standard (the legal and ethical obligation of both substitute decision-makers, typically parents, and physicians), requiring an overall balance of more benefits than harms. These goals of care dictated how they would approach the decision about extubation.

Many participants cited how clinician-level factors such as personal risk assessment and tolerance, familiarity with difficult airway management and knowledge of complex patients modified the decision. For example, some stated they might prefer a more senior clinician perform the extubation given Siri’s particular risk level. Local context also contributed, particularly regarding the timing of the decision. For example, several participants expressed the desire for the ear, nose and throat team (specialists who support airway management issues) to be on hand. Some also cited Child Life Specialists as a resource to manage patient stress which would affect their timing. These factors highlight how a decision’s ‘goodness’ can be modified by contextual factors and that clinicians should overall aim to maximise all potentially contributing factors for patient benefit.

All participants spoke about communication with families and the need to prepare them for the intervention, modifying how they communicate based on that particular family’s knowledge, preference and context as well as considering how the family can support optimisation of the extubation (e.g., bedside presence). Several participants remarked that a good decision can be rendered less ‘good’ by poor communication, harming the relationship with the family and compromising trust.

The machine learning tool

All participants drew from visualisation 1, which provides an overview of the model’s training, validation, prospective clinical evaluation and regulatory approvals (online supplemental Appendix B, Table B.1).8 Nonetheless, participants felt this regulatory and testing information did not directly tell them how to interpret the output for Siri. Most participants pointed to the model having received regulatory approval as a bit of comfort, to varying degrees. Some were reassured but noted that in paediatrics other interventions have been approved without having been tested on paediatric patients which gives them pause. Others felt regulatory approval was an unacceptably low bar in and of itself.

The phrase ‘my patients’ came up repeatedly, referring to whether the model had been tested prospectively for its performance on the patients in their specific unit. Participants were most reassured by the prospective testing and good performance of the model among the patients at the hospital and in the unit in which it was now being used.

Participants reported appreciation about the model use, notably the specific task it performed (risk prediction for extubation success, defined as no need for reintubation within 48 hours). The link between the prediction and the clinical decision, however, must be clearly specified, as one participant remarked on inadvertent phrasing by one of the interviewers: “Is the model actually telling me a recommendation or is it just presenting me the data?”

All participants wanted to know the inputs (i.e., the specific data sources the model was pulling). However, participants also raised that knowing the inputs did not necessarily convey sufficient information about how those inputs were transformed into computable data. For example, the genetic diagnosis and the chart notes were raised frequently as sources of uncertainty as to how the variables were precisely computed, which had bearing on how the participant would interpret the prediction.

Several participants remarked that the performance metrics were less convincing to them compared with their own experience with the model. Several remarked that the outcomes would ‘speak for themselves’, indicating that ultimately performance (eg, compared with explanations) would calibrate the scope of their reliance on the model. Performance did not have to be perfect, but participants noted the importance of learning through experience where the model tended to ‘get it right’ and develop a schema of the types of errors that the model was prone to. Most asked about representation—whether patients ‘like Siri’ were part of the training data.

As explainability was directly tested in the interview, we observed a range of perspectives. Nearly all stated that they would like to know how both the predictions and the explanations were computed for Siri, but for many it appeared a curiosity rather than something critical to their correctly interpreting the model output. For example, one experienced physician remarked “I’m not hugely convinced that explanations are going to be necessary for the most efficacious models. And you know what I mean by that is that we have a long track record in medicine of using things that work even if we don't understand them” [P004, male]. Some, however, felt strongly that they would not accept a ‘black box’ model.

All participants did, however, want to get a general sense of the method by which the algorithm computes the task. Some referred to the formula, others the weights, and others a ‘skeleton’ of the model’s computational processes. When queried, the primary purpose of requesting the above information seemed to be to compare the model’s general computational process to their own clinical schema.

Knowledge in context

Participants all spoke about applying knowledge towards goals of care, where they considered how knowledge about medical evidence and context are interpreted in light of the specific patient at hand to engage in shared decision-making with the family.

Participants reflected on their interpretation of information, noting that individual sources of information (eg, bedside signals) need to be interpreted relative to the patient’s individual situation. For example, knowing the patient’s baseline relative to the general baseline for patients of their age or condition modified how they used bedside information. Participants also wanted to know what had been done previously to optimise the patient, and where one source of information was less reliable, the triangulation of information sources to justify decisions. At a higher-level, this way of personalising medical care drew from established knowledge about patterns in general and applying it to the individual patient.

Several participants raised questions about T18’s social, political and historical context. Many stated that knowledge established from clinical research was prone to bias because of pessimistic prognostications in the context of disability. The knowledge foundation in this case (i.e., with respect to the presence of T18) was perceived as less reliable than for other medical issues.

The model’s ‘knowledge’

All participants wanted to not just know the inputs, but whether they were the right inputs—that is, processing the information deemed clinically relevant to the task. A few participants expressed a belief that more complicated models (meaning more inputs) could take more patient features into account, and assumed improved accuracy on this basis. Others remarked that one could simultaneously increase the uncertainty around the knowledge produced.

Several participants considered data quality, noting that the data sources were more or less reliable as model inputs depending on the context in which they were collected. For example, some remarked that there is variability in clinical note documentation within and between specialities and between individual providers, with trustworthiness of those sources also being variable.

Participants wanted the model to convey elements of the patient’s prediction which were actionable versus non-actionable. Specifically, nearly all participants decided not to extubate Siri despite what they viewed as a generally legitimate prediction because they felt more could be done to optimise Siri for extubation.

Several participants wondered whether the limitations of medical knowledge about T18 had become incorporated into the model predictions. Bias as a knowledge problem could happen when consistent patterns of care were encoded by the model, but were not actually representative of the true situation. Multiple participants cited previous medical research where decisions about withdrawal of life-sustaining therapies were made on the basis of prognostic factors that themselves became determinants of outcomes (the ‘self-fulfilling prophecy’). Participants worried about being overly pessimistic about Siri’s prognosis due to stigma against disability.

Participants all wanted to know how the explanation related to the prediction. Given that the prediction was a risk score, participants were unclear about whether a particular highlighted feature or signal was indicative of the favourable or unfavourable direction of the prediction; for example, did the variables presented in the Feature Importance visualization online supplemental file 1 Appendix B, Visualization 2 in Table B.1 contribute to the 60% likelihood of success or to the 40% possibility of failure? They often supplied their own interpretation of the explanation in line with their clinical knowledge.

Model’s ability to reflect the context

There was a consistent notion that the model outputs needed to be contextualised with direct knowledge of the patient, positioning the patient as anchor for the ultimate verification of a prediction’s reasonableness. A common example from ICU-trained participants was that physiological signals can look quite nefarious just by observation, yet on setting eyes on the patient something benign had happened. Participants who were aware of these artefacts wanted to know if the model had been designed to extract these events from the overall prediction task (artefact recognition) to help them appraise the relevance of the prediction.

Nearly all participants remarked that the model prediction as well as the task the model was supporting were but part of a larger judgement they needed to make that centred on the big picture. The big picture considered the goals of care for Siri and how the task itself related to their ability to achieve those goals. The risk was relative to those goals, such that 60% for Siri was not the same as a 60% risk for another child with lesser medical complexity; and 60% for a child who was not to be re-intubated was not the same as 60% for a child on an interventional care pathway.

Reasoning on discrepancies

To challenge their thinking, we constructed two visualisations with directly contrasting information (see online supplemental appendix B, table B.1, visualisations 5 and 7). The contrast related to the extent to which T18 was influencing Siri’s risk for extubation failure: (1) if Siri’s T18 phenotype is relatively mild and thus does not substantially modify their risk—visualisation 7 is wrong or (2) T18 is a significant predictor despite Siri’s apparent robustness—visualisation 5 is wrong. In the clinical presentation, participants were told that Siri had a history of difficult intubations online supplemental file 1 Appendix A: Extended Methods, and Box A: Simulated Case Scenario, which was a clue that T18 was in fact a significant contributing factor.

Participants responded in two ways. Some assumed the model had knowledge they did not have, with many participants stating there must be ‘a reason’ for this explanation. Others stated that visualisation 5 and the mere presence of a discrepancy made them question the model’s reliability.

Those who detected the discrepancy drew from their knowledge about T18 generally (“medical knowledge” category) and considered Siri’s ‘robustness’ based on their bedside observations (“context” and “integrating knowing in context” categories). Several also appropriately recognised that T18 is a condition for which data patterns may be biased by virtue of under-representation in the training set (”machine learning model” category) and incomplete knowledge (“model’s ‘knowledge’” category) about the prognostic implications. There were no clear patterns in terms of level of experience, role or other participant-level factors, other than that nearly all participants from cardiac critical care observed that visualisation 5 struck them as unusual. While the sample size is small, it is noteworthy that multiple participants without ML knowledge or cardiac intensive care experience picked up on the discrepancy while others with domain expertise did not.

Decision space

Participants generally recognised that the model’s output represented a subset of the considerations they needed to reflect on, taking the information provided by the model as informing their calculus of the overall risks and benefits and their respective likelihoods to this individual patient, mediated by contextual factors. We asked how they would explain their thinking to a querying parent:

I would explain all of the different factors that I was clinically taking into account and say that we also have a general model that can give us a score and taking that score into consideration with what I know about Siri and [their] medical history, this is how we’ve come to our clinical decision. I’m […] giving a higher level of all the different aspects that are contributing to my decision and to let me come to this conclusion. [P015, female, RN]

Participants reported that the extubation prediction formed a larger part of a care plan, and that ultimately the accuracy of the prediction would come secondary to many contextual factors such as the goals of care and the status of the unit. Overall, decisions were guided primarily by acting towards the best interests of patients (“I want to make the good decision and not just the ‘algorithm-correct’ decision” [P015, female, RN]).