Artificial Intelligence (AI) has made its way into healthcare, from note-taking to voice agents. Conversations around liability and legislation are already happening, and there’s no doubt that AI is starting to change how care is delivered.
Already 75% of big healthcare companies are “experimenting with or planning to scale generative AI across the enterprise,” according to Deloitte Center for Health Solutions.
But less discussed are the potential downsides of this new technology. Just as the implementation of electronic medical records brought pros and cons—like easier collaboration across departments, but more time at the computer—it’s likely AI will see a similar varied trajectory.
“There’s vast amounts of marketing hype [around AI], compared to the actual clinical proof,” James Barlow, a professor at Imperial College Business School in London, told Healthcare Brew. “How do you as a company or developer actually build up an evidence base for the claims you’re making?”
We spoke with experts about the potential challenges AI presents to healthcare, and how the industry can best guard itself against them.
Downsides. To build models, most AI tech is trained using existing data, whether that's published research or a patient’s chart. That means, in order to work well, the tools must have access to accurate and high-quality information. But experts are concerned whether this is happening.
Concerns around bias and equity still plague AI in general, but in healthcare, it could make the quality of care worse, especially for higher-risk patients, Barlow said. “What base are you training the AI on? Are the datasets representative? If they’re not, they might produce biased results.”
Harish Yalamanchili, a surgeon at vascular clinic Coastal Vein Care in California, said understanding what sources data is trained on is “one of the most important things.”
“How careful are you in generating and making sure that your sources are carefully vetted?” he said, adding that “we’re running out of data to build our AI.”
Navigate the healthcare industry
Healthcare Brew covers pharmaceutical developments, health startups, the latest tech, and how it impacts hospitals and providers to keep administrators and providers informed.
(The New York Times reported publishers and other online sources are preventing their information from being used to build AI. In other words, AI needs existing information, but many sources have already been used or are getting restricted.)
There’s concern this could lead companies to “loosen the restrictions a little bit,” on the quality of data used to build platforms, Yalamanchili said.
“AI for me should be a tool that makes your ability to treat people easier,” Yalamanchili said. “The outcomes have to be equivalent to the outcomes without AI, or better.”
Though, he added, he also worries these tools could make it easier to miss errors introduced into a patient’s chart. “If you have a corrupt data point anywhere…how do you know what the reality was before this?” he said.
Bad data or errors could create mistrust between clinicians and patients, Barlow added. There’s also fear about job displacement, he said.
Upsides. In terms of building up guardrails around these challenges, Barlow said a lot of it comes down to trust.
“[Building trust] requires being very open and transparent about the data that’s being used, the quality of the data, how you’re designing the algorithms that use that data, whether you’re using advanced datasets,” he said.
It’s also important to follow any future laws that may emerge, he said. There are currently no “absolute” regulations for AI in healthcare, Yalamanchili added, though organizations like the American Medical Association and Coalition for Health AI have written some.
Yalamanchili said including more clinicians in leadership could also help, as they know what would best support staff and patients.
“The way we’re going about it, and just throwing all these darts at the dartboard, and hoping that one of them hits bull’s-eye,” he said. “We’re talking about actual human lives,” said.