Experts call out obscure privacy policies for AI health bots, limited federal rules

When’s the last time you read an app’s full privacy policy? It’s probably been a while, if ever.

That’s always been a concern for data privacy experts. But the explosion of health and mental health chatbots have taken those concerns to a new level as more consumers discuss sensitive mental health information with AI models, some of which operate outside the healthcare system and don’t have to follow federal patient data protection laws, i.e., the Health Insurance Portability and Accountability Act (HIPAA).

A lot is on the line as a result. Bad actors can leverage personal health information to gain access to health savings accounts or even bank accounts. But keeping track of where data is going is difficult. Even state health insurance marketplace websites have shared personal data with Big Tech via ad trackers, per a May Bloomberg report.

Amid these high stakes, experts are pointing to what they see as obscure data privacy policies and calling for reform.

Dense disclaimers. Wright told us it can be difficult for consumers to find answers to questions about data governance policies when using a mental health chatbot. Privacy policies are often “not user-friendly” and written in “legalese.”

Startup AI companies don’t always train their own large language learning (LLM) models from scratch, either. Increasingly, new AI products are wrappers, meaning they contain an algorithm built on top of frontier LLMs like ChatGPT or Claude, she said.

“Where the challenge happens often is that companies that build the wrapper can’t always answer [data governance] questions on behalf of the frontier models,” Wright said.

Experts want federal help. The Trump administration is pushing to bring more AI into the medical system. But regulations around health AI so far have largely come from the states, with the federal government sticking primarily to nonbinding recommendations, Healthcare Brew previously reported.

Federal government regulations could motivate AI startups to operate with more transparency, the American Medical Association (AMA) said in April 22 public letters to Congress. The letters urged more disclosure, oversight, accountability, and security requirements for health chatbots.

As it is now, many users may not know that LLMs operating outside the healthcare system are not covered entities under HIPAA, AMA CEO John Whyte told Healthcare Brew.

“Chatbots can serve some value, obviously, in the healthcare system, but in its current iteration, there’s probably more risks than there are benefits,” he said.

Help from an unlikely source. Big Tech could also play a role in shaping regulations.

New November 2025 guidelines from Apple—released ahead of the launch of an AI-upgraded Siri—require developers to get users’ “explicit permission” before sharing their data with third-party AI.

The new rule prompted companies like startup Slingshot AI, maker of mental health chatbot Ash, to add a new privacy notice to its user onboarding process early this year, Slingshot co-founder Daniel Reid Cahn told us.

Ash’s previous privacy policy disclosed that it partnered with outside companies but didn’t explicitly name companies like OpenAI or Anthropic, Cahn said. Though Ash isn’t a wrapper, the new notice specifies that user messages may be sent securely to these two companies and other named partners to generate responses, though the partners won’t store or train on these conversations. It then asks the user to check a box if they consent.

“You will see recent disclosures from many AI companies due to [the] AI disclosure policy from Apple,” Cahn said over email. “In any event, the privacy of our users continues to be paramount, and we have zero data retention agreements with our partners.”