Clinicians are using AI without guidance. What are the risks?

There’s no longer a question of when AI will become a regular part of working as a clinician—it already is. A survey from healthcare AI platform Heidi Health found that 86% of clinicians surveyed use the technology daily or on a regular basis, often for administrative tasks like charting.

However, more than 4 in 5 (83%) reported doing so without guidance from their employers. This could be leaving clinics, health systems, and other healthcare institutions vulnerable to HIPAA violations and cyberattacks, experts told Healthcare Brew.

“Healthcare is undergoing what I’d call an unmanaged AI transformation,” Steven Teppler, chief cybersecurity legal officer, partner, and chair of cybersecurity and data privacy at the law firm Mandelbaum Barrett, told us. “Clinicians discover the tools can be used, can save a meaningful amount of time, but the institutional governance surrounding that is lacking, and that’s where the legal, security, and patient safety issues start to come to the fore.”

Heidi Health conducted the survey in May. Respondents consisted of 1,823 healthcare workers from 25 countries, including those working in family medicine, mental health, surgery, and acute care. he respondents reported using a range of AI tools, including ChatGPT, Claude, and Gemini, as well as healthcare-specific tools such as ambient scribes or UpToDate and OpenEvidence, which are generally HIPAA compliant.

Potential unintended fallout. A concern of unmanaged AI use is how protected health information (PHI) is handled, Teppler said. PHI includes any information about a patient’s clinical care such as test results or treatment plans that’s stored in the same record as identifiable information, like the patient’s name and social security number, and is protected under HIPAA.

Is PHI being transmitted through the AI? Is it being used to train AI models? Is it encrypted? These are some of the questions physicians and their employers must consider to avoid potential violations when using an AI tool, Teppler added.

The consequences of breaking HIPAA can be severe, ranging from reputational risks to million-dollar fines and even prison time for selling PHI.

Beyond HIPAA concerns, Teppler said he’s worried about the cybersecurity implications of unguided AI use. Many AI models have browser extensions, for example, which are vulnerable to various cyber threats.

Audrey Adeline, a former member of the Founder’s Office at SquareX, a browser detection and response company, told IT Brew last October that AI browsers “are not security-aware.”

“They’re trained to complete tasks to make people more productive, but it’s very easy for attackers to trick these AI agents to make them think that certain malicious tasks are required to complete whatever prompt the user is telling them to do,” she said at the time.

Cyberattacks have huge consequences for health systems. They can lead to delays in patient care, medical errors like incorrect medication dosing, and medical equipment malfunctions.

The Change cyberattack in 2024, for instance, left providers and health systems unable to process insurance claims and prescriptions or schedule appointments. It cost providers an estimated $100 million per day, we previously reported.

What can be done? Health systems can’t wait for the government to step in and regulate AI, according to Jeffrey Saviano, a senior lecturer on AI strategy, governance, and ethics at the Massachusetts Institute of Technology’s Sloan School of Management. They instead need to take it upon themselves to institute guidance detailing what they will and won’t tolerate.

An important part of a governance framework is transparency, Teppler added. There should be transparency between clinicians and their employers regarding the types of AI technology they’re using as well as transparency between clinicians and their patients when AI is used as part of their care.

However, defining the guardrails around transparency is a challenge in itself, Saviano said. When should clinicians disclose AI to patients? Should they disclose something as small as when AI was used to code a piece of software the clinician is using? Or should they only disclose when a clinician is using AI for direct care, like using an ambient scribe to record their interaction? The answer is nuanced, and every health system will have to take an individualized approach, according to Saviano.

“I have so much empathy for clinicians,” Saviano said. “They’re overworked, they’re trying to do the right thing. I believe that the vast majority are doing it with great intentions, but I don’t think that they understand the risks involved.”