Introduction
Intensive care units (ICU) are increasingly involved with rapid response teams (RRT) and provision of critical care outreach services (CCOS) to ward areas.1,2 The extent and character of these activities, with respect to the interventions undertaken and human resources required, are underreported,3 particularly as the focus until now has been the effect of the intervention on mortality and adverse event rates,4 and tools to allow detailed examination of CCOS outcomes are currently lacking. Though guidelines regarding outcome data to be collected have been available for some time,5 published information regarding practical CCOS design and implementation in the digital era is lacking. In 2018 a database of RRT activity6 was maintained by 200 of 232 hospitals in Australia and New Zealand (86.2%), of which 196 (98.0%) audited RRT events. Detailed data characterising CCOS such as time spent with patients and interventions, frequency of follow-up, and qualitative analysis of participant experience may yield valuable insights into critical care resource allocation. Other examples are planning the commitment of time and expertise, audit the quality of service delivery, assist the design of ICU referral criteria, and designing training programs for staff.
Here we describe the design and implementation of software used as a data entry and archiving instrument (“eTool”) for tracking the activity of an ICSU outreach service, including a comparison of the paper and electronic records with a focus on data completeness and auditability.
Methods
Pre-existing systems and the case for change
The study hospital has a total of 950 beds including a 46-bed ICU. Prior to the implementation of the eTool, data regarding consultations were collected by hand-written entries, which produced variable entry practices and inconsistent legibility. After CCOS consultations, outreach medical staff returned to ICU with patient and referral details in the form of handwritten paper notes. This data were entered into a paper record book. The book of entries were collected at the end of each month by the ICU data manager and entered manually into an Excel® spreadsheet. This task consumed approximately 40 hours of the data manager’s time each month. Most of this time was spent obtaining missing patient details, completing information about each consultation from the medical record, or outreach team member recollections. Qualitative data regarding management and outcome details for each consultation was not entered into the database, which significantly hampered the task of auditing the outreach workload.
The CCOS service consisted of 12-hour shifts staffed by 1 or 2 ICU registrars per shift. Record keeping was paper-based. Data were collected prospectively for six months for all referrals, from 1 August 2021 to the 1 January 2022. During each shift an average of 8.7 patients were referred to the CCOS.
Design and implementation of the eTool
The overall structure of the eTool and its interconnections with software platforms is shown in Table 1. A mobile ‘App’ compatible with major mobile operating systemswas developed for data collection at point of care. CCOS staff were supplied with mobile devices containing the mobile App (Fig. 1). A desktop version was also created to facilitate bedside handover (Fig. 2).
Workflow
Because the Microsoft Teams® platform was used to deploy the data collection App, the desktop App was accessible anywhere in the hospital, and the mobile App could be accessed by the outreach team by installing Microsoft Teams® on the outreach mobile phones provided to them by the department. Over the course of the day, team members were able to access the Apps contemporaneously with their consultations, including while attending the patient. Each entry could be edited or amended as more details about the case became available. At the overlap between shifts, the outreach team members were able to handover to each other using the eTool as a record of ongoing activity (for example filtering the patient encounters by those identified as still needing follow-up). The database backup function ran weekly (backing up to an archive Sharepoint® List) and daily (backing up to an Excel® spreadsheet), with the daily incremental rows emailed to the ICU data manager every morning at 08:00. The ICU data manager was able to access these data and produce immediate summarised reports regarding outreach workload patterns. Theoretically, regular reporting functions (PowerBI®, Microsoft®) could be used to automate this task.
Implementation
The database was implemented with a period of overlap to insure against data loss. During the first two weeks of the implementation (July 18th to July 31st 2021) the outreach team were instructed to double-document, using the App to document patient consults only after making their entry into the paper record. At the end of each week the data entered into the App were compared to the data entered into the paper record by the primary author and data manager. At the same time this period permitted a test of the automated backup systems and other software elements. Feedback regarding the user interface was collected from outreach team members and used to adjust interface elements (for example text size was increased to 12 points for the mobile version of the app). From August 1st 2021, the electronic system became the default mechanism for recording outreach workload, and the paper record system was retired.
Comparison of the eTool with the paper record system
An audit was carried out to assess key performance characteristics of the eTool with respect to data completeness and auditability. Four auditors independently completed a review of a single month from the paper record (January 2021, 450 records) and a single month from the electronic record (January 2022, 517 records). Each auditor was supplied a scanned copy of the paper record and an electronic copy of the database as a Microsoft Excel® document.
Each auditor was instructed to count illegible or missing entries for each of the fields of interest in a simple data collection spreadsheet. For the electronic database sample, counting using simple spreadsheet formulae (eg. COUNTIF using Microsoft Excel) was permitted. Each auditor was also instructed to report the total time taken to perform the audit. The completeness of the paper record was then compared to the electronic record. Averages were calculated from the number of incomplete records found by each auditor, and intraclass correlation coefficients were computed to test the agreement between auditors.
Results
Types of consultations include non-urgent referrals from ward staff and emergency department personnel, “Major Trauma” calls from the emergency trauma team, medical emergency team calls (“Code Blue”), telephone requests for patient transfer from smaller hospitals and follow-up visits for patients previously referred to the ICU.
The results of comparing the paper and electronic records are presented in Table 2. There was excellent agreement among the four auditors (ICC = 0.9977 and 0.9999, respectively). The overall rate of missing data using the electronic record was substantially lower compared to the paper record. The paper database contained a substantial number of omissions or illegible entries, most significantly patient name and medical record number details, the duration of time spent on the consultation, and the “Time accepted” (time at which the patients were accepted to ICU). In contrast, the electronic database contained very few missing or incomplete entries. The auditors took an average of 5 hours and twelve minutes to complete the audit of the paper database, and an average of 32 minutes to complete the audit of the electronic database. Assuming one month is representative, the same audit of a year of paper data would therefore occupy sixty hours of auditor time.
Further data analysis
Figs 3 and 4 show examples of the power of eTool in describing workloads and clinical outcomes. Fig 3 shows the result of an audit of the database to provide a detailed breakdown of ICU activity over 2022 and 2023. The graphs demonstrate weekly activity rates and outcomes of ICU outreach activity. Weekly CCOS activity was 120-140 consultations per week, with the majority being referrals and Code Blue events. Fig. 4 represents further analysis of Code Blue and Major Trauma events. In over 60% of consultations no further CCOS input was required, whereas in the case of direct referrals to the ICU, 44% required ICU admission. This audit raised the suggestion that the criteria used to trigger Code Blue rapid responses were too sensitive and produced too many false alarms.
Discussion
A combined software and database system for recording outreach activity is an innovative method of documenting the expanded role of ICU involvement in non-ICU clinical work via CCOS. To our knowledge, the use of a mobile platform for this purpose, the design of the database fields and the details of the graphical user interface are presented for the first time in the literature.
Technical accounts of medical emergency team review documentation methods typically emphasise the recording of physiological derangements or interventions, often using paper-based forms.7,8 The deficiencies of paper-based systems for recording these data are well described,9 most notably the poor data integrity associated with missed fields or illegible handwriting.
The ability to use the eTool on hand-held devices or desktop computers across the hospital campus guaranteed high accessibility. Time stamped entries are beneficial in many ways. Firstly, contemporaneous data entry can be ensured, enhancing accuracy and validity of the entries, and minimizing forgotten data. Secondly, they allow for determination of activity trends through the working day, assisting decisions on when staffing should be bolstered, coinciding with peaks and troughs of activity. Resources can also be allocated for services covering large distances. As outreach and emergency response services across the hospital are mobile, there are additional medico-legal and governance advantages of contemporaneous time stamped data entries that are unaltered.
To the best of our knowledge, this electronic system is the most comprehensive method of documenting CCOS workload currently reported. The use of an electronic database informs decision-making regarding CCOS staffing and operation. For example, the audit presented in our article sheds light on the workload demands placed upon junior doctors in the support of these services in a tertiary metropolitan hospital. The database and interface can be configured to include data entry fields which are individualised to the needs of any specific service provider group and are not limited to CCOS. Furthermore, integration with electronic patient records could allow the association of workload data with patient-centred outcomes data.
Our technique has several limitations. Though a transactional database design was selected, a relational database structure such as Structured Query language (SQL) would have been preferred for the purpose of connecting patient data with ICU contact data (joined, for example, by unique patient identifier numbers). A transactional database architecture decreases the efficiency of search queries and degrades the performance (speed of record retrieval and interface responsiveness) of the application when handling large data sets. Use of SQL was not possible because of the limitations imposed by the local integrated patient management system (IPMS). From the point of view of collecting ICU outreach workload data for purposes of detailed audit, including measuring outcomes, a relational database structure would be ideal. In practice, this was not a barrier to data analysis, as both the IPMS and the SharePoint List database allowed the data to be exported as an Excel-readable .xlsx file, which could be imported into SQL and manually reconciled. The ability to change data into an SQL format is important for inter-operability issues when combining data with other applications.
Limitations
The limitations of our study and design process represent constraints of the small scale of our implementation. We present the experience of a single centre and this experience may be context-specific to our institution. This is a stand-alone App with a separate database, and ideally all records of clinical data should be unified within the institutional electronic medical record system or have sufficient interoperability. As with any other electronic tool, the quality of data is limited by the quality of user input. Increasing accessibility of the App through remote access and mobile apps can assist with decreasing the time spent for data entry, while improving the heuristics of the user interface. Structured entry fields can improve the quality of the collected data, but these strategies do not offer any definitive way to determine whether any data are missing. Missing data are a problem with any such service and future apps might look to implement ways to prevent this, such as reminders based on geolocation data, or interval reminders for users to input data. Ongoing training and education of users is just as important to refresh users with importance of timely and accurate data entry.
Conclusions
Electronic records are an essential part of a modern clinical workplace. Currently, electronic methods dedicated to the recording of CCOS data are lacking. We propose an application of widely available software to create an eTool for recording storing and retrieving CCOS data. We demonstrate the improved data integrity and auditability associated with the use of electronic outreach database systems and offer solutions to practical questions in the design and implementation of this instrument.
Provenance: Externally reviewed
Statement of financial support: This research was performed without external funding support.
Competing interests: None declared
Ethical approval: Ethics approval for the use of patient information was granted by the Scientific Advisory Committee of the local health district human research ethics committee (approval number 2203-15 QA).
Acknowledgements: None.
Author statement: The research data associated with this manuscript is available from the Corresponding Author. A compressed archive of the eTool application source code is also available on request.
Corresponding Author: Dr Alex Yartsev, Department of Intensive Care, Westmead Hospital, Westmead, New South Wales, Australia 2145. Email: alex.yartsev@health.nsw.gov.au
