Online symptom checkers: still a long way to go

ONLINE and mobile application symptom checkers get the diagnosis right first time in just a third of instances, but the only Australian-based checker included in a recent study,, Healthdirect, did slightly better than the rest.

In pre-coronavirus disease 2019 (COVID-19) times (November 2018 – January 2019), Australian researchers put 36 freely available online and mobile application symptom checkers under the spotlight to determine their effectiveness in providing diagnostic and triage advice.

The researchers evaluated the symptom checkers using 48 medical condition vignettes (1170 diagnostic vignettes and 688 triage vignettes). They concluded that symptom checkers may provide unsuitable or incomplete diagnostic or triage advice for users in Australia, resulting in inappropriate care advice.

Their findings, published in the MJA, showed that the correct diagnosis was listed first in only 36% of tests, and the first 10 results in 58% of tests. For triage services, the correct advice was provided in 49% of cases, including 60% of emergency and urgent cases, but only 30–40% of less serious case vignettes.

Healthdirect, which was launched in 2015 and is funded by the federal government, was the only Australian-based symptom checker included in the study. The service, which provides only triage advice, did slightly better than other symptom checkers in the MJA study, providing correct triage advice in 28 of 46 vignettes (61%).

Associate Professor Adam Dunn, Head of the Discipline of Biomedical Informatics and Digital Health at the University of Sydney, said it was important to consider the purpose of symptom checkers in determining their utility.

“Is a symptom checker just something you want to Google online to reassure yourself, or is it something you want to actually action and you want to deploy as a health service to try to get people the right care at the right time?” he asked.

He said it was likely that most symptom checkers included in the study were not designed to replace a full triage service.

“The most important thing is that we really shouldn’t be judging all the symptom checkers together. We don’t need 50 symptom checkers; we only need one and we need one good one. And the best performing [triage tool in the study] … happened to be the Healthdirect one, which is likely to be the most appropriate one for the Australian context.”

Associate Professor Dunn said finding the right risk balance was crucial to the effectiveness of online symptom checkers.

“There needs to be balance between sending too many people to health services that they don’t need, and not directing people to health services who need them,” he said. “It’s a real sensitivity and specificity problem. And that’s why it’s such a critical thing right now [during a pandemic], because we need to play with this balance between being risk averse and making sure people get care when they need it versus not overwhelming the health system.”

He said the UK’s artificial intelligence triage service, Babylon Health, provided a recent example of the potential dangers of not getting this balance right.

When a UK doctor tested the service by inputting symptoms related to a heart attack, the symptom checker advised him to stay home.

Dr Marie-Louise Stokes, Chief Medical Officer of Healthdirect Australia, said the Healthdirect symptom checker aimed to provide information and advice to support someone who was unsure about what to do about their symptoms.

“It is not a diagnostic tool and does not replace assessment by a clinician,” she said. “Symptom checker users can have the information and advice sent to them via SMS. A unique reference number enables a nurse on the Healthdirect helpline to bring up their information if they call the helpline.

“The symptom checker contains a service finder so they can find the nearest appropriate health service if advised to seek medical assistance.”

It’s a service that seems to have struck a chord with Australians.

“We have seen a huge adoption of people seeking health information from digital channels,” Dr Stokes said. “In the 2019 financial year, usage of the Healthdirect Symptom Checker doubled, and it is now used over 2 million times a year.”

In an editorial to be published in the MJA on 15 June, Associate Professor Dunn wrote that to be meaningfully integrated into clinical practice, decision support tools should be approved by a national governing body and subject to ongoing surveillance, similar to drugs and medical devices.

He said formal regulation would be challenging because online symptom checkers included disclaimers warning users not to rely on the health advice provided. Instead, he said, health services could recommend services that have ongoing, transparent surveillance, including user surveillance.

“We are never going to be able to regulate these kinds of symptom checkers because they are always going to exist online,” he said. “If we were going to recommend one of them – let’s say the Healthdirect one happens to be the best that we have – local health districts or NSW Health can advise people to use that service.”

Dr Stokes said Healthdirect welcomed standards and systems –- including guidelines, accreditation or regulation – aimed at improving the safety and quality of digital health tools.

She said the service’s symptom checker algorithms were reviewed by clinicians who checked question flows and wording, care advice and dispositions against Australian guidelines.

“We also check whether certain symptoms – for example, abdominal pain – are over-represented in a certain disposition, such as visiting the emergency department,” she said. “If we find that is the case, we look to see if a change to wording (such as ‘severe’ v ‘so severe you can’t walk’) could make it clearer and eliminate ambiguity or confusion for users to ensure they are referred to the most appropriate level of care.”

The agility of many sectors of the health system has been put to the test during the COVID-19 pandemic, and online symptom checkers are no exception.

In February, Healthdirect developed a COVID-19 app that has been accessed 2.5 million times and counting. The COVID-19 Symptom Checker had been updated 31 times so far to take account of changes in guidelines and testing criteria.