Protocol•

Evaluating the implementation of a digital coordination centre in an Australian hospital setting: a mixed method study protocol

Abstract

Introduction This protocol outlines a mixed methods study evaluating a new Digital Coordination Centre (DCC) at the Royal Melbourne Hospital (RMH), Melbourne, Australia. While coordination centres show potential for impact, evidence on effective implementation in the Australian context remains scarce. This study aims to address this gap.

Methods and analysis The evaluation involves a two-stage approach: a process evaluation to clarify DCC design and identify implementation factors, and an initial outcome evaluation to assess short and medium term outcomes. A developmental approach will support continuous improvement, and implementation science theories applied to unpack change processes. Data sources will include interviews, project documentation and observations, with qualitative and quantitative analyses targeting metrics like emergency department boarding and length of stay.

Ethics and dissemination This study has been approved by the RMH Human Research Ethics Committee (QA2023089). Findings will be shared through peer-reviewed publications and conference presentations.

What is already known on this topic

  • There is currently limited evidence on the process of implementing digital coordination centres and their impact on patient flow in healthcare, particularly within Australia.

  • Common outcome measures for coordination centres typically relate to patient transfer volume and emergency department boarding. While performance improvements are commonly reported, variability in design and the quality of evidence collected limits the robustness and generalisability of findings.

What this study adds

  • This study protocol provides a detailed description of a mixed method design for comprehensively evaluating a hospital digital coordination centre.

  • The evaluation design includes a developmental approach to support continuous real-time improvement, a logic model to clarify how the digital coordination centre works to achieve outcomes, and application of implementation science and normalisation process theories to unpack change processes.

How this study might affect research, practice or policy

  • The study protocol will be useful for others designing studies/evaluations of digital coordination centres in Australia and internationally.

  • The findings will inform hospital policy decisions and practice changes though identification of critical success factors for implementing an effective digital coordination centre.

Introduction

Healthcare providers recognise the need to improve patient access and flow, so patients get the right care, at the right time and in the right place.1 2 Increasingly, digital command or coordination centres are being adopted to improve information flow and situational awareness in several different industries.1

In a healthcare setting, command centres have been defined as the colocation of interdisciplinary workgroups, such as bed management and environmental services, that use real-time data to integrate and manage multiple processes related to patient flow.1 However, there is currently limited research evaluating the implementation of digital command centres and their impact on patient flow in healthcare.

In Australia, like in many parts of the world, there is currently high demand for the healthcare system. Contributing factors include unpredictable COVID-19 waves, staffing shortages and difficulties accessing primary care services.3 4 As service demand continues to increase, the Royal Melbourne Hospital (RMH) has committed to implementing a digital coordination centre (DCC) focused on improving patient access and flow to optimise the use of existing resources and provide better patient care.

The RMH DCC will be implemented across three phases. Phase 1, the focus for this evaluation protocol, addresses patient flow and involves colocation of resources and services, including access and bed management, elective surgery scheduling, nursing allocations, the patient flow coordination team, facilities management and transport coordination along with medical and administrative support. Newly developed digital tools and real-time dashboards will support the daily operations of the DCC (see online supplemental file 1 for an explanation of the digital tools and dashboards).

To understand factors associated with successful implementation of Phase 1, the Centre for Digital Transformation of Health (CDTH) at the University of Melbourne in partnership with the RMH is conducting a design, process and initial outcomes evaluation. The objectives and questions are listed in table 1.

Table 1
•
Evaluation objectives and key evaluation questions

Phase 2 will be informed by the results of Phase 1 and concentrate on optimising clinical pathways and reducing preventable complications. Phase 3 extends the hospital’s reach through virtual health and community services. Each phase will be progressively evaluated to support continuous improvement.

Methods and analysis

Study design

Phase 1 evaluation of the DCC will use a developmental approach conducted in two stages: process evaluation and initial outcome evaluation. Developmental evaluation addresses the challenge of evaluating programmes that are still being designed and modified and where inputs, activities and outcomes may not yet be fully known through a process that includes ‘asking evaluative questions, applying evaluation logic and gathering real-time data to inform ongoing decision-making adaptations’.5 The process evaluation incorporates staff experiences, useability, satisfaction and concerns. The objectives were discussed and confirmed with the stakeholders at RMH, with this approach being consistent with developmentally orientated evaluations.

To collect data and understand the operating context of the DCC, two researchers from the CDTH will be embedded within the RMH DCC and form a part of the programme delivery and evaluation team, as is common in developmental evaluations.5 This supports improving the capacity for evaluation and facilitating data-based discussions about what is working and what is not, and what this means in the context of the DCCs implementation.

The process evaluation will examine how the DCC is delivered, describing the current operating conditions and identify barriers and enablers to success. Evaluation of initial outcomes will provide a preliminary assessment of changes resulting from implementation and the extent to which these are sustainable. The mixed methods evaluation will use multiple methods of data collection to inform the process and initial outcome evaluation components (figure 1).

Figure 1
Request permission

Two-stage approach. DCC, Digital Coordination Centre.

The evaluation stages are designed to interact and inform each other. This approach aims to deliver an evaluation that provides practical, reliable and actionable information to support the establishment of the DCC and provide recommendations for future improvement.5

Setting

This study will be conducted at RMH, a major metropolitan quaternary referral centre and teaching hospital located in Melbourne, Victoria, Australia. The RMH provides emergency, medical, surgical, subacute, mental health, inpatient and outpatient services across multiple campuses.6

Participants

Interview and survey participants will include those who work in the RMH DCC and clinical and operations staff who are impacted or work closely with the DCC. A purposive sampling technique will be used to select participants for interviews, aiming for maximum variation in roles. Interviews will be conducted until thematic saturation is achieved.7 The survey will be a census sample of all staff involved in the DCC. The target number of interview and survey participants is 40–50. This is to align with the number of staff directly involved in the implementation and initial usage of the DCC.

Data sources

Stage 1 (process evaluation)

Data collection will include internal documents, literature on digital command centres in healthcare contexts, electronic medical records (EMRs) and hospital information systems data. Documents relating to the RMH DCC include promotional materials and events, meeting minutes, staff position descriptions, training materials, policies and procedures and activities and outputs from staff huddles.

In-depth qualitative data will be collected via staff and external stakeholder interviews. Data on information availability, dissemination and actions will be accessed from the hospital’s EMR. DCC actions and resolution data will be collected via the Hospital Status dashboard and Microsoft Teams, where staff actions and resolutions are updated daily.

Stage 2 (outcome evaluation)

Survey data will be collected via an online survey. Outcome metrics on patient flow will be retrieved from a combination of the EMR and data warehouse. Escalation metrics, such as thresholds for beds closed, patients in emergency and patients waiting for an in-patient bed, will also be obtained from the Hospital Status and Progression of Care dashboards (see online supplemental material for definitions).

Study variables

Stage 1 (process evaluation)

Interview study variables were adapted from the Consolidated Framework for Implementation Research (CFIR) interview guide and include the following interview question constructs—intervention source, implementing climate, structural characteristics, culture, compatibility, planning, engaging, patient needs and resources, adaptability and knowledge and beliefs about the intervention.8 9

Observation templates were adapted to the context and based on the CFIR observation template10 and the Theoretical Domains Framework (TDF) Codebook.11 The observation template is designed to record data on events and observations, such as meetings notes and evaluator reflections on the implementation and impact of the DCC on staff routines and practices. Analysis will include frequency of dashboard usage, number of users over time by service and data on DCC dashboard uptime and completeness (including data outages). The evaluation will capture actions determined by the DCC, including the service that is directed to take the action and the resolution.

Stage 2 (outcome evaluation)

Survey variables will consist of a 23-item questionnaire based on normalisation process theory,12 with additional Likert-scale items to capture staff perceptions of outcomes, as identified in the logic model (figure 2), and salient implementation factors adapted from CFIR.13 The survey will also include a question on participants’ role/team at RMH, years working at RMH and open text fields for additional comments and feedback. The planned variables for the outcome metrics include emergency department (ED) boarding time, ED length of stay, overall length of stay and cancelled surgeries.

Figure 2
Request permission

RMH DCC logic model for Phase 1: patient flow. The DCC is a dedicated hub that leverages data analytics to optimise hospital operations and enhance patient care. It will enable a more proactive approach to monitoring, coordinating and managing various aspects of hospital operations, with a focus in Phase 1 on patient flow and patient care logistics. DCC, Digital Coordination Centre; ED, emergency department; RMH, Royal Melbourne Hospital. * Indicates outcomes that will be explored in future phases of the evaluation.

Evaluation data matrix and logic model

The evaluation data matrix (table 2) shows how qualitative and quantitative data sources, measures and baselines align with key evaluation questions and subquestions.

Table 2
•
Evaluation data matrix

A mixed method approach was selected involving triangulation and synthesises of evidence to provide robust conclusions about the quality of DCC implementation and its contribution to the initial outcomes anticipated in the logic model (figure 2).

Analysis

Stage 1 (process evaluation)

For the document review, all documents will be logged using an Endnote library. This will include recording details on the type, author, date, source and summary of each programme document used in the development and operations of the DCC. Thematic analysis will be used to inform the summary and coding of each.

Semistructured interviews of up to 1 hour will be conducted. Interviews will be transcribed verbatim and entered into NVivo 14, a qualitative software package. Transcripts will be analysed using the coding approach developed by Braun and Clarke to identify, analyse and determine recurring and emergent themes and subthemes relating to the CFIR domains and constructs.14

Observation data collected by the embedded evaluators to produce insights into the daily operation of the DCC will also be coded using a thematic coding technique based on the CFIR and TDF constructs. Regular evaluation team meetings will be held to share and discuss learnings from the observations and meeting notes to consolidate emergent findings on factors affecting the DCC implementation. For the qualitative data, thematic analysis will be employed to both describe and explain what implementation looks like in the DCC and hospital teams, what effects are observed and what influences those effects—such as enablers, barriers and contextual factors.

The information availability, dissemination and actions analysis will include summary statistics of the number and duration of dashboard outages or data outages by dashboard over time. It will also include assessing the summary of usage (including times accessed and number of refreshes) by user service and dashboard over time and the summary of actions taken by type of action, escalation level and service over time. Analysis of completeness of action data, including percent of actions with documented responses, will also be analysed.

For escalation metrics, pre- and post-go-live analysis of escalation triggers will include a description and display of time spent in escalation prior to go-live and the distribution of escalation state duration, and principal components analysis to determine the main drivers of the escalation state. The analysis will also include the historic progression of care data relating to escalation, including the analysis of sensitivity, specificity and positive and negative predictive value of predicted discharge, as well as descriptive statistics and visualisation to determine temporal and other patterns in estimated versus actual discharge data.

Stage 2 (outcomes evaluation)

The staff survey will be analysed using descriptive statistics (such as frequencies, means and measures of central tendency and variation). For the outcome metrics, descriptive analysis and visualisations to identify seasonal and other patterns and establish optimal comparison periods for key patient flow outcome metrics will be conducted. Key patient flow outcome metrics for services and the RMH overall will be compared before, during and after implementation of the DCC.

Supplementary PDF Icon
Supplementary PDF

  • Contributors: All authors contributed to the planning, design and development of this study protocol (SF, CM, WB, DC, WWC, TF, JG, MJL, KL, LP, SP and BA). CM, WB and DC led design of the quantitative component and BA and SF led the qualitative component. SF and BA led writing of the manuscript. All authors reviewed drafts and approved the final version. BA is the guarantor.

  • Funding: The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests: TF, JG and SP are employees of the Royal Melbourne Hospital with responsibilities for leading the implementation of the digital coordination centre at this site. The Centre for Digital Transformation of Health at the University of Melbourne is supporting staff time to conduct the evaluation.

  • Provenance and peer review: Not commissioned; internally peer reviewed.

  • Supplemental material: This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Ethics statements

Patient consent for publication:
Ethics approval:

This project has been approved by the Royal Melbourne Hospital Human Research Ethics Committee, as a part of the low-risk quality assurance project (QA2023089).

  1. close Franklin BJ, Mueller SK, Bates DW, et al. Use of Hospital Capacity Command Centers to Improve Patient Flow and Safety: A Scoping Review. J Patient Saf 2022; 18:e912–21.
    doi:10.1097/PTS.0000000000000976•Google Scholar
  2. close Safer Care Victoria. Understanding the ins and outs of patient flow: safer care Victoria. 2022;
    Available: here
    Google Scholar
  3. close Koontalay A, Suksatan W, Prabsangob K, et al. Healthcare Workers’ Burdens During the COVID-19 Pandemic: A Qualitative Systematic Review. J Multidiscip Healthc 2021; 14:3015–25.
    doi:10.2147/JMDH.S330041•Google Scholar
  4. close Australian Medical Association. The general practitioner workforce: why the neglect must end: australian medical association, Australian Medical Association. 2022;
    Available: here
    Google Scholar
  5. close Patton MQ. Developmental evaluation: applying complexity concepts to enhance innovation and use. New York, NY, US, Guilford Press 2011;
    Google Scholar
  6. close Locations: the Royal Melbourne Hospital.
    Available: here
    Google Scholar
  7. close Saunders B, Sim J, Kingstone T, et al. Saturation in qualitative research: exploring its conceptualization and operationalization. Qual Quant 2018; 52:1893–907.
    doi:10.1007/s11135-017-0574-8•Google Scholar
  8. close Consolidated Framework for Implementation Research. CFIR guide: consolidated framework for implementation research.
    Available: here
    Google Scholar
  9. close Damschroder LJ, Reardon CM, Widerquist MAO, et al. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci 2022; 17.
    doi:10.1186/s13012-022-01245-0•Google Scholar
  10. close Observation template: consolidated framework for implementation research.
    Available: here
    Google Scholar
  11. close Cahill LS, Carey LM, Mak-Yuen Y, et al. Factors influencing allied health professionals’ implementation of upper limb sensory rehabilitation for stroke survivors: a qualitative study to inform knowledge translation. BMJ Open 2021; 11.
    doi:10.1136/bmjopen-2020-042879•Google Scholar
  12. close Murray E, Treweek S, Pope C, et al. Normalisation process theory: a framework for developing, evaluating and implementing complex interventions. BMC Med 2010; 8.
    doi:10.1186/1741-7015-8-63•Google Scholar
  13. close Consolidated framework for implementation research: consolidated framework for implementation research. 2022;
    Available: here
    Google Scholar
  14. close Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006; 3:77–101.
    doi:10.1191/1478088706qp063oa•Google Scholar

  • Received: 11 September 2024
  • Accepted: 8 January 2025
  • First published: 6 February 2025

Article metrics
Altmetric data not available for this article.
Dimensionsopen-url