Opinion: Don't risk patient trust in the pursuit of better AI
This is an opinion piece by Johan Busse, chair of the Danish Council on Ethics for Data (Dataetisk Råd), published in Ingeniøren. He argues that patient trust must not be put at risk in the pursuit of better AI in healthcare.
The trigger is a case reported by Ingeniøren. More than 100,000 patient messages were used in an AI pilot in Region Hovedstaden and Region Sjælland without informing the patients. The project was meant to help staff draft replies to patient messages. According to Ingeniøren, no data protection impact analysis was done beforehand. The regions earlier rejected criticism of the project's legality, but now concede that their duty to inform patients may have been insufficiently met. Ingeniøren also spoke to six parents whose children were linked to the child and adolescent psychiatry department in Roskilde during the trial. They reacted to not being told, and several want to know whether their child's data was used. Busse says that regardless of the legal assessment, use of patient data must be explainable and justifiable to patients.
His central point is that trust affects treatment. If patients are unsure how sensitive information is used, they may hold back symptoms and worries, which weakens staff's basis for choosing the right care. Busse acknowledges AI's real potential to free up time and improve communication, but says pilots must state from the start what improvement is sought and how it will be judged. Questions include whether staff really save time when they must check replies, whether the answers are correct and useful, and whether a simpler solution or less personal data could do the same. Expected gains must be weighed against risks, and both tracked.
He also calls for patient involvement throughout, as recommended in the council's report on responsible AI in health and welfare technology. This should cover patients with language or digital barriers. Relatives can contribute but cannot replace patients' own views. Patients should be told clearly whether their or their child's data is in an AI project, and staff need to understand the tool's limits and have time to catch errors. Management should bring together clinical, technical, legal and data ethics expertise, understand how suppliers handle data, and define who approves launch and expansion and when unresolved risks require a pause.