Do no harm: the ethics of AI in healthcare

“I can look at my patient instead of my keyboard.” 

This is just one example of how AI has supported Dr Janice Tan, Deputy Chair of RACGP Digital Health Special Interest Group, who uses AI mostly for paperwork and administrative work, including an AI scribe in most consultations.

Artificial intelligence (AI) has become a common support tool in many medical settings, with even more growth and innovation to be seen because of its rapidly expanding technological capabilities.  

The Australian Commission on Safety and Quality in Health Care (the Commission) recognises that while artificial intelligence (AI) has the potential to support healthcare delivery, it also presents key ethical challenges for clinicians. 

Macquarie University authors of a scoping review, report that, “AI is being used to make influential clinical and operational decisions through tasks such as predictive analysis, diagnostic assistance, or optimisation of administrative processes.

“These decisions may change resource allocation decisions or diagnosis and treatment pathways, raising concerns over liability in instances of harm as lines of responsibility remain unclear in legal guidelines.

“New risks for patient safety have also emerged from AI systems, with the potential for errors arising from data drift, hidden biases, and model ‘hallucination, not adequately addressed by existing governance. In a high-risk industry like healthcare, such errors could have serious widespread or fatal consequences.”

 Associate Professor Ash Hopkins, an NHMRC Investigator and leader of Flinders’ Clinical Cancer Epidemiology Lab, says, “AI holds enormous promise but must be applied responsibly.” 

Understanding the ethics of using AI in healthcare should assist clinicians.

AI ethics principles in healthcare 

The Australian Government’s AI Ethics Principles are a helpful guide for understanding the ethics of AI for all Australians. Meanwhile, Macquarie University’s scoping review examined the frameworks governing AI across the lifecycle within healthcare organisations.

It found the most common ethics principles, such as transparency, fairness and equity, privacy and security, accountability, and beneficence, aligned closely with established AI ethics frameworks. However, subthemes specifically relevant to healthcare, such as consent, confidentiality, and medicolegal liability, were not commonly included. Noticeably, some critical ethics principles, such as contestability, sustainability and cultural safety, were absent from more than three-quarters of the frameworks reviewed.

Accountability

The review found that the oversight mechanism (such as an AI-specific governance committee) was the least common ethics component. 

According to the Commission, a key challenge is maintaining accountability for clinical decisions. While AI can support clinical practice, responsibility for patient care remains with the treating clinician. As outlined by the Commission’s Pragmatic AI Guidance for Clinicians (the Guidance), this means applying professional judgement and critically evaluating recommendations or outputs generated by AI tools.

Dr Tan says she never lets AI make the decision. “It doesn’t diagnose, doesn’t prescribe, and nothing it writes leaves my practice until I’ve read it. For evidence lookups, I go to the source. If a tool can’t show me where something came from or if I am uncertain that the reference is valid, I don’t use it.”

Patient harm from AI-assisted clinical decisions 

This oversight is important, as a situation may arise in which an AI-assisted decision causes harm. In this situation, Professor of Law Jeannie Marie Paterson, and Co- director of Melbourne University’s Centre for Artificial Intelligence and Digital Ethics (CAIDE), says there are three possible parties who may be responsible: the clinician, the clinic, and the manufacturer/developer. “In all cases, liability doesn’t attach merely because there is a poor outcome.

“The liability of the clinician (and the clinic) depends on negligence (a tort) and a breach of s 60 of the Australian Consumer Law. The clinician must have failed to exercise reasonable care and skill in using the tool. There are as yet no cases. It is likely that the clinician would have to exercise independent judgment about whether the tool is suitable for use and more broadly understand the limitations of the tool and the propensity for error. 

“For example, AI clinical assisted decision tools may provide accuracy and confidence ratings - the clinical should understand what these mean and exercise skill in deciding how to use the recommendations.”

Jeannie explains that the liability of the developer will commonly be based in product liability under the Australian Consumer Law, which can apply to software. “The question would be whether the tool has a safety defect and is not as safe as the community reasonably would expect.”

Safety

Associate Professor Rita Matulionyte from Macquarie Law School, Macquarie University, explains that technology companies should provide healthcare organisations and clinicians with accurate information about the tools’ capabilities and limitations (also what they don’t know about the tool’s performance).

“Clinicians should avoid using tools that have not undergone robust evaluation. This means checking TGA approval of AI-enabled medical devices, and for AI tools that do not qualify as medical devices, hospitals and clinics should conduct appropriate evaluations before acquiring them. They could also look for independent research studies evaluating the tool, where available. This should be done regularly, as independent evidence on newer AI tools is still emerging.

“Use of ChatGPT and other general purpose models that were not trained for clinical purposes is not acceptable.

“Clinicians should also monitor the tool’s performance, respond appropriately to safety incidents and report them, and follow best practices for its use.

“Healthcare organisations should have AI governance frameworks, including designated personnel responsible for AI policy and implementation.”

Transparency

She says consent is one of the biggest ethical dilemmas that can pose a challenge for clinicians, particularly for patients with lower health or digital literacy, or who don’t speak English as a first language. 

Professor Matulionyte says, “In our recent paper (currently under peer review), we argue patients should be informed where: 

1. AI plays a significant role in clinical decision-making; or 
2. AI replaces the clinician. 

Patients should also be informed about how their personal data is used by AI tools, potentially through an organisation’s general privacy statement. For AI scribes, a separate explanation of personal data use may be appropriate, as these tools collect substantial amounts of personal information, and patients appear to have particular concerns about this.

But Dr Tan adds, “Then there’s what happens to patient data once it leaves the room, and whether it gets used for purposes the patient or clinician never agreed to, especially with AI functionality and scope creep.”

Privacy

As AHPRA explains, healthcare providers must be especially vigilant to avoid inputting sensitive information into programs such as LLMs and should have policies and procedures and training on the use of LLMs (including Chat GPT). An intentional or unintentional disclosure of health information into LLM technology could be a notifiable data breach under the Privacy Act.

Dr Tan says, “I don’t put identifiable patient information into any tool that hasn’t been through proper privacy and governance review.”

Fairness

According to the Commission, another important challenge is the risk of bias, which can typically arise from the type and quality of data used to train AI systems. AI tools may produce less accurate or less reliable outputs for some patient groups when those groups are underrepresented in the data used to train the system. The Commission’s Guidance highlights the importance of AI systems being trained on data that reflects the diversity of the patient population in which they will be used. Clinicians need to be aware of these limitations and consider whether AI outputs are appropriate for the individual patient and clinical context.

Contestability

According to the authors of the scoping review, all stakeholders require assurance that efficient and accessible processes exist to lodge complaints and receive appropriate and adequate redress. If this process is too difficult, justice is compromised and there is little opportunity for similar harms to be avoided in the future.