Electronic medical record (EMR) systems have the potential to improve quality of health care, streamline workflow and increase efficiency in the health care system.1,2 However, potential problems have also been identified, such as the cost of implementing and maintaining EMR systems, and the skills and training needed for using them.1,2

Despite the proposed benefits and pitfalls of EMR systems, there is limited research on their effectiveness.2–4 Specifically, there is a lack of data regarding their effect on definable clinical end points, such as patient mortality.5 Locally developed systems, designed around local procedures and conditions and implemented from the ground up, have shown benefit.6–8 Specific computer-assisted decision support has also been shown to be of benefit.7–9 A clinical information system must be fit for purpose for any gain to be made.10

FirstNet (Cerner), an EMR system, has been introduced in emergency departments (EDs) around New South Wales since 2007. This system replaced the Emergency Department Information System (EDIS; Healthcare Group, CSC), which was previously used in most NSW EDs and is still in use in some. Despite limited literature indicating that FirstNet has decreased performance in EDs in Australia,10,11 and reports of problems with Cerner programs overseas,12–15 there were no objective quantitative data to substantiate these concerns.

FirstNet was introduced to the Nepean Hospital ED in March 2009. All medical documentation became electronic, including doctors’ examination notes, nursing progress notes, patient observations, and electronic ordering of pathology and radiology investigations. Our study’s aim was to determine whether the implementation of FirstNet was associated with an improvement or decline in key performance indicators (KPIs) of patient flow in the Nepean Hospital ED.

We conducted a retrospective observational study comparing ED performance before and after FirstNet was introduced on 24 March 2009. Nepean Hospital is a 453-bed tertiary referral and teaching hospital serving the western suburbs of Sydney, NSW. The ED sees about 54 000 patients each year, with a 20% paediatric caseload and a 36% admission rate.

The control group consisted of patients presenting during the 6-month period from July to December 2008, when EDIS (version 9) was in use. During this period, only patient triage and tracking was electronic, with all other ED functions relying on paper documentation. The study group consisted of patients presenting during the 6-month period from July to December 2009. This period was considered appropriate because the new FirstNet EMR system had then been operational for more than 3 months. The version of FirstNet in use at Nepean Hospital was the standardised state-based build slated to be introduced across all hospitals in NSW.

Outcome measures were: waiting time for all patients; waiting time, treatment time and total time for discharged patients (see definitions in Box 1); proportion of patients who did not wait to be seen by a doctor (DNW rate); proportion of ambulances with offload waiting times longer than 30 minutes; and mean number of patients seen per non-specialist doctor per shift.

We also collected data on potential confounding variables: number of presentations to the ED; number of presentations by Australasian Triage Scale category; mean daily ED occupancy; and number of shifts worked per week by ED specialists and non-specialists. For this analysis, mean daily ED occupancy (Box 1) was used as a measure of overcrowding. Data relating to access block were not collected.

The numbers of shifts worked by medical staff were retrieved from ED rosters. The number of patients seen by each doctor per shift was extracted from EDIS but could not be extracted from FirstNet. Therefore, a mean number of patients seen per doctor per shift was calculated using the FirstNet data on presentations and staffing levels.

The data were imported into SPSS version 19 (IBM). The distribution of waiting time, treatment time and total time was non-normal, and these results are reported as medians with interquartile ranges and compared using the Mann–Whitney U test. All proportional data were compared using the Pearson χ2 test. The mean number of patients seen per non-specialist doctor per shift and ED occupancy were compared using the independent t test. We used statistical modelling to attempt to control for the effect of potential confounding variables (see Appendix).

Ethics approval for the study was granted by the Sydney West Area Health Service Human Research Ethics Committee, Nepean Campus.

There were 25 620 presentations in the 2008 control period and 26 128 in the 2009 study period. Triage category distribution was similar in each period (Box 2).

There were too few patients in triage category 1 for meaningful analysis of this subgroup. We also found that data on disposition status (did not wait v discharged v admitted) for most patients presenting in July and August 2009 were missing in FirstNet. Therefore, only ambulance offload data could be analysed for the full 6 months. All other comparative measures used data from September to December only of the control and study periods (16 976 and 17 281 presentations, respectively).

There was a statistically significant increase in waiting time for all patients (Box 3) and in waiting time, treatment time and total time for discharged patients (Box 4), both overall and for most triage categories. The waiting time for triage category 2, for all patients and discharged patients, was shorter after the introduction of FirstNet, but treatment time and total time increased for discharged patients in this category. Treatment time for triage category 4 patients discharged from the ED was also shorter after the introduction of FirstNet, but there was no improvement in waiting time or total time for these patients. There was also a reduction in treatment time and total time for category 5 patients discharged from the ED.

After the introduction of FirstNet, there was a statistically significant increase in the DNW rate and the proportion of ambulances with offload waiting times longer than 30 minutes, and a decrease in the mean number of patients seen per non-specialist doctor per shift (Box 5). The daily medical officer roster was unchanged between the control and study periods. However, there was a significant decrease in the mean number of shifts actually worked by both specialists and non-specialists, due to staff vacancies and sick leave. Mean ED occupancy was similar in the two study periods.

After adjusting for the effect of total presentations, specialist and non-specialist staffing, and ED occupancy, the mean total time for discharged patients was 41 minutes longer (95% CI, 29–53; P < 0.001) in 2009 than 2008. Similarly, the DNW rate remained 6.0 percentage points higher (95% CI, 4.8–7.0; P < 0.001) in 2009 than 2008.

Overall, there was a significant increase in the waiting time for all patients, and the waiting time, treatment time and total time for discharged patients after the introduction of Cerner FirstNet in the Nepean Hospital ED compared with the control period.

ED efficiency is affected by many factors both within and outside the ED, including the time taken for inpatient team reviews and “acceptance” of an admission, waiting for a bed on the ward, and completion of clerical and clinical documentation. In a retrospective observational study, it is not possible to directly control for these confounding variables. However, as discharged patients are the group whose care is least affected by these confounding factors, studying this group attempts to isolate actual changes in ED efficiency from the effects of these factors.

For all patients as well as discharged patients, we found that waiting time for those in triage category 2 was significantly shorter in the study period than the control period. Changes to ED workflow in response to poor performance in standard KPIs is a continuous process, and it may be that there was some change in work practice to try to improve the category 2 performance. However, the treatment time and total time for discharged category 2 patients were still significantly longer in the study group. Thus, any benefit from the decreased waiting time in this category was not sustained through to total time in the ED. For triage category 4 patients, treatment time was reduced in the study period, but again this was not borne out in a reduction in total time for these patients.

In contrast, waiting time for all patients in triage category 5 was increased in the study period. However, treatment time and total time for discharged patients in this group were shorter with FirstNet. Category 5 patients are largely seen in the fast-track area of the ED. They do not occupy a bed for the period of their assessment and treatment and thus are generally not affected by ED overcrowding or access block for ward admissions. It could be that there is some benefit from FirstNet for these patients.

The increases in the DNW rate and the proportion of ambulances waiting more than 30 minutes to offload, and the decrease in the mean number of patients seen per non-specialist doctor per shift indicate overall deterioration in the performance of the ED after implementation of FirstNet.17,18 This could potentially increase ED overcrowding because even a small decrease in treatment rate is cumulative, as it causes further increases in the number of patients waiting ahead of new arrivals.17 Worldwide, studies in different centres have found an association between overcrowding and reduced access to care, decreased quality measures and poor outcomes,17 including increasing mortality as overcrowding in EDs increases.19–21

We used ED occupancy as a marker of ED overcrowding, to assess any confounding effect it may have on the efficiency of care for discharged patients. There was no significant difference in ED occupancy in the two study periods, despite the significant change in ED performance. This is likely due to the large increase in total time for discharged patients being offset by the large increase in the proportion of patients who did not wait.

We chose to sample 6 months of presentations to limit the effect of potential confounding variables such as seasonal variations, staffing changes and chance variability in caseload and casemix between the two periods, and we matched the periods of the year for which we collected data. Junior medical officers change terms every 10 weeks to 3 months, and it is possible that staff relatively new to the ED with little experience in using FirstNet could have affected the results. However, the term changes are similar from year to year, and we would expect the findings to have been controlled by examining the same 6-month period of each year. As patient presentations per triage category were similar in the control and study periods, it is unlikely that differences in patient distribution across categories were responsible for any of the findings of this study.

Staffing levels were lower in the study period than the control period and this could have affected ED performance, as fewer doctors may lead to longer waiting times.22 However, the deterioration in total time performance and DNW rate remained after adjusting for the measured confounders. Specialists in our ED do not have a distinct patient load; they have a supervisory role to ensure quality of care and patient flow. For this reason, we only measured the mean number of patients seen per non-specialist doctor per shift.

Another possible confounder is a learning effect of the new EMR system. FirstNet was operational for more than 3 months before the study period, and all medical staff had undergone the prescribed training before its introduction. As medical and nursing staff may rotate through the ED for variable periods of as little as 10 weeks, if the learning effect is greater than 3 months, then they will have spent a large part of their time in the ED learning the system without ever becoming adept at using it.

Although we performed statistical modelling to attempt to control for the effect of the measured confounding variables, this process is limited, as the functional relationships assumed by our modelling may not match the non-linear nature of the data. Also, it is acknowledged that statistically adjusting for confounders has its own limitations and cannot account for unknown and unmeasured factors.

An important post-hoc finding was the poor quality of data for the first 2 months of the study period, especially with respect to patient disposition. For this reason, we could only use 4 months of data from the control and study periods. In addition, there were missing data points for some time measurements.

Since the second half of 2009, when the study data were collected from FirstNet, there have been upgrades to the software. However, these have been largely cosmetic, such as the design of the discharge summary and the appearance of the tracking screen, or addressing issues of patient safety, such as introducing standard checks when ordering radiology investigations and highlighting the patient row when accessing a patient’s record. It is not clear whether these changes would lead to significant improvements in ED performance.

Overall, we found that implementation of Cerner FirstNet was associated with deterioration of the performance of the Nepean Hospital ED with respect to the measured outcomes.

One of the key issues for high-volume, high-risk workplaces like hospital emergency departments (EDs) is the struggle of conflicting aims. While hospital managers need information systems for data collection and storage, clinicians need efficient clinical documentation, data retrieval and order-entry systems that save time rather than steal it from the patient. The work of clinicians is aided by reliable data but impaired by the delays of real-time input, difficult system navigation, suboptimal presentation of information, and other problems in the user experience of health information technology (HIT).1

Mohan and colleagues’ study of the impact of an electronic medical record information system on ED performance had some limitations.2 It was retrospective and unable to control for all confounders, and therefore could only show a correlation with ED delays, not causation. However, the premise for the study delivers an important message — the work required to use the information system was perceived by the ED staff to directly conflict with time spent with patients.

Another study has shown that the same electronic medical record information system is perceived to have had a negative impact on the care of patients, as well as the productivity and morale of staff, in six EDs in New South Wales.3 The need to be hypervigilant about the accuracy of the information supplied by the electronic health record compounds an already stressful clinical environment, which in turn leads to resentment towards the technology and the people who have imposed it. This makes it “bad” HIT. Unless this is corrected, HIT efforts will overuse precious health care resources, will be unlikely to achieve claimed benefits for many years to come, and may actually cause harm.4,5

The large HIT corporations produce a type of technology that is best categorised as enterprise resource planning (ERP), which has its roots in the manufacturing industry. It is based on the idea that all processes within an organisation can be standardised, and that all processes of the same type should have their information modelled and processed in the same manner. If this high degree of standardisation were considered the best way to process and model information derived from clinical activity, then ERP would be a favoured technology to adopt, as has happened in many places.

However, there is an alternative, almost contradictory, perspective on the nature of clinical work: that it is non-deterministic and performed by a group of diverse staff working in an ecologically stable network of people that has to respond to diverse medical needs and diseases. The ecology model accommodates staff joining and leaving the process, with differing needs emerging at different times, so that the other individuals in the network have to adapt and modify their behaviour and improvise in an unpredictable manner. Amid all this variability is the ever-demanding imperative to improve the processes of care and attention to the patient, while also increasing staff productivity.6

Where the ERP model has been imposed in the clinical setting, staff may be coerced into an approach to their work that is at odds with established best practices. This could only be considered “good” HIT if it brought greater staff productivity with at least no loss (and, preferably, improvement) of patient safety and services and staff morale.

It is not enough just to identify problems: effort must be invested in transforming bad HIT into good HIT. This process must identify and optimise all the operative factors: human behaviour, system design, equipment performance, skills of the IT participants, and the operational policy framework.7 Good HIT should include clinician control of the interface design for content, dataflow and workflow. It includes the ability to change the system in real time, and it incorporates inbuilt data analytical capability, natural language processing, and native interoperability and clinical coding.8 Finally, there must be an appropriate opportunity to test systems for useability, effectiveness and suitability before their release.

There must be a move away from standardised processing models and towards improving the user experience in the clinical setting. Clinicians should not have to shoehorn their activity into predefined, externally imposed work processes that do not reflect actual activity and will not improve efficiency. A true patient-focused system aligns all its components towards the same aim. Like a good clinician, good HIT does no harm — to patients or staff.