Patients are already asking AI. How do we make it safer?

My patients already arrive having asked AI about their symptoms. I am not against this, and I could not stop it if I were. Some arrive better informed than they would have been. The question is not whether patients will ask AI. The question is what is behind the AI they are asking.

People ask ChatGPT 300 million health questions a week, by OpenAI's own count. In July the company opened ChatGPT Health, connected to hospital records and Apple Health, to every US adult on free and paid plans. The fine print says it is "not intended for use in the diagnosis or treatment of any health condition". That wording keeps it outside medical device regulation. It is the company covering itself. People use it to look up their diagnoses anyway.

It is currently only available in the US, but models like this will reach Australia. The delay is a window to set the rules before they arrive, not after.

The same week, the Australian Alliance for AI in Healthcare, led by Professor Enrico Coiera, released its third National Policy Roadmap, built on input from 96 organisations, making 28 recommendations. Its finding: Australia's health system is not ready. A handful of technology giants are racing to become the first place everyone goes with a health question, shaped by commercial algorithms and offshore data we do not regulate. The TGA cannot touch a chatbot, because a chatbot is not classed as a medical device. The Australian organisations and agencies that should be closing the gap are working in silos, doubling up the work in the race for governance. When someone consents to an MRI, they are also consenting to the algorithms inside it, and nobody tells them.

The shortcomings of AI health advice

These models learned mostly from American and European material written in English. They can use the wrong drug names, follow the wrong guidelines, and recommend medicines that are not available here. They know nothing of how Medicare or the PBS works. Aboriginal and Torres Strait Islander health is barely present in the training data, so the tools know least about the population with the greatest need. Women are under-represented in the medical literature the models learned from, and a model trained on biased research inherits the bias. If that is not addressed, these models will amplify the bias rather than correct it. Right now, while the rules are being written, we can still influence which way that goes.

In Australia we strictly regulate commercial influence over health advice. A drug company cannot fly me to a resort to influence what I prescribe, and payments to doctors are publicly disclosed. The United States allows prescription medicines to be advertised directly to consumers. Australia bans it. Yet an algorithm can steer millions of health decisions at once, shaped by commercial interests the patient never sees, with no register, no code of conduct, and no regulator able to look inside. The same risk we regulate in doctors, at far greater scale, with no rules at all.

There is a quieter problem. AI does not take a history. Ask it what to take for back pain and it might suggest ibuprofen, which is reasonable, unless you are on an anticoagulant, and your kidneys are failing, and you did not think to mention it. The model answers the question it was given, without the facts that change the answer. Teaching people what the tool needs to know, and what it cannot do, is part of the answer. National AI literacy must run in parallel, otherwise we are handing Porsches to people who have never driven.

Building safeguards for Australian patients

So what should Australia do? Two things, and they are not alternatives.

Regulate what is already here. These companies have Australian offices, Australian revenue and Australian government contracts, which is where regulation grips. We can demand transparency over training data, require that Australian health data stays onshore on local servers and is never used for training. We require that no model update reaches Australian users before it is tested. Parliament can close the loophole: the law currently asks what a product is marketed as, not what millions of people a week actually use it for.

And offer a better door. Not an Australian ChatGPT. Nobody here has the billions, and we do not need them. We rent an engine under contract and build the door ourselves: a service that answers only from Australian guidelines, Cochrane reviews and PBS information, sends chest pain to triple zero instead of into an essay, and logs and audits every question and answer, the way we already audit hospitals. Over time, we run one of the freely released models on our own computers, customised with Australian guidelines and Australian data. It gets better every year, it is ours, and nobody offshore can switch it off, reprice it or take it away. Build the door so the engine can be swapped, and it outlives every model behind it. We have built a national front door before. It is called healthdirect, millions of Australians use it, and the AI front door is its next job, not a new invention.

None of this makes these tools safe, only safer. The rules reduce the risk. They do not erase it. But my next patient has already asked AI before walking through my door. The only question still open is who we let stand on the other side of that door with them.

Dr Samantha Pillay OAM is a urological surgeon in Adelaide, the first female urologist in South Australia, and a member of the Royal Australasian College of Surgeons' AI in Surgery Advisory Group. She writes on AI, health and equity at samanthapillay.com.