Performance data caution

A SURVEY of senior US hospital managers has reinforced concerns about the worldwide trend to publish hospital performance outcomes on websites such as MyHospitals, with claims by the study participants that the measures used are meaningless.

The survey, published in JAMA Internal Medicine, obtained responses from chief medical officers and other leading executives at 380 US hospitals, who were asked about America’s Hospital Compare website. (1)

Hospital Compare is more detailed than Australia’s more recent MyHospitals site, and includes reports on “processes of care” measures such as the percentage of patients hospitalised for acute myocardial infarction treated with β-blockers, as well as condition-specific mortality and readmission rates, patient experience measures and cost and volume measures.

The US survey found a strong belief among survey respondents that many of the measures stimulated “quality improvement efforts, a sense of empowerment that hospitals are capable of bettering their performance, and an understanding that the public is paying attention”.

However, up to 59% of respondents raised concerns about unintended consequences of publicly reported measures such as gaming the system through changing documentation and coding practices, and neglect of more important issues.

Half the respondents did not believe that the measures accurately portrayed the quality of care for the conditions they addressed or could be used to draw inferences about quality at the hospital more generally, and more than half reported that the measures were not meaningful for differentiating hospitals.

The author of an accompanying commentary said the bureaucratic work of measurement and reporting usually fell to non-clinical staff “with little understanding of or authority over processes on the wards”, and argued that performance improvement programs should be clinician-led in order to provoke meaningful change. (2)

Australian experts told MJA InSight the study offered important cautions for the future development of MyHospitals.

The Australian website, launched by the federal Labor government in 2010, currently contains limited data such as waiting times for particular procedures and rates of health care-associated Staphylococcus aureus infections. However, the National Health Performance Authority (NHPA), which maintains the MyHospitals website, has plans for it to report on hospital-standardised mortality and other measures. (3)

Dr Ian Scott, director of the department of internal medicine and clinical epidemiology at Princess Alexandra Hospital, Brisbane, said he did not believe the MyHospitals website was currently driving any significant improvements in Australian hospitals.

“I’m not saying hospital managers or professionals are ignoring it but don’t think they take a lot of notice of it either, because they already have access to the information before it goes online”, he said.

“No hospital wants to be the worst for a particular measure, so they are sensitive to the data, but they were already sensitive to these measures because there has been an intense quality and safety agenda within hospitals for some time.”

Dr Scott applauded the current “softly, softly” approach to publishing data on the website.

“I think there’s a little more reluctance on the part of policymakers and hospital managers in Australia compared to other countries to having mortality and other clinical outcome data put up on a public website because there are concerns people will misinterpret it”, he said.

Dr Alan Wolff, a medical administrator and director of medical services at Wimmera Health Care Group, Victoria, agreed Australian politicians and bureaucrats had been relatively tempered compared with their US and UK counterparts in committing to publishing hospital performance data online.

“Our hospitals know their hospital standardised mortality rates but they are not published”, Dr Wolff told MJA InSight. “There hasn’t been a draconian approach in Australia, but I think it will get tighter.”

Dr Wolff said the key lesson from the latest US study and others like it was to involve clinicians in creating and monitoring performance outcomes.

“It needs to be a bottom-up approach, and it needs to involve a carrot and not a stick — not shaming hospitals into improving performance, which lowers morale among staff”, he said.

“The measureables must to be things that can be changed, like rates of thrombolysis for stroke, rather than just mortality figures.”

Dr Diane Watson, NHPA CEO, said the US study was important research given some US states were more than 15 years ahead of Australia in publicly reporting comparable performance of hospitals.

“In light of this new research in the US, we hope that the measures on the MyHospitals website stimulate quality improvement, clinical engagement and improved care for patients”, she told MJA InSight, saying MyHospitals averaged more than 238 000 unique visitors per month.

Dr Watson said NHPA worked with hospital advisory committees and collaborated with international colleagues to ensure its measures reflected the quality and safety of care in Australia’s hospitals.

1. JAMA Intern Med 2014; Online 6 October
2. JAMA Intern Med 2014; Online 6 October
3. NHPA 2014; Performance indicator reporting, 31 July

(Photo: iStock)