How safe are chatbots for health data? It’s hard to know.

Each week, we schedule our rounds with Healthcare Brew readers. Want to be featured in an upcoming edition? Click here to introduce yourself.

Between AI labs putting the healthcare industry in their sights and ChatGPT joining WebMD as a go-to for health anxiety, chatbots are digesting an increasing amount of sensitive personal health information.

But just how safe are these platforms in terms of data privacy? And what do providers using these systems need to know about inputting sensitive data?

Jennifer King, a privacy and data policy fellow at the Stanford Institute for Human-Centered AI, said the general lack of transparency into how AI labs handle personal data makes these hard questions to answer. King and a team of researchers recently investigated the privacy policies of six leading AI companies and published the findings last fall.

We spoke with King about OpenAI and Anthropic’s forays into health-specific chatbots, risks for consumers, and what healthcare providers should be thinking about as they use these tools.

This conversation has been lightly edited for length and clarity.

What are some of the biggest risks if you’re a consumer and you’re starting to turn to chatbots for health questions or putting more sensitive health information in there?

It’s hard to know…There is very little transparency in what happens to that data that I provide the companies. We know they train on the data you provide them by default, but what does that actually mean, and what are the risks? It’s a really hard thing to pick apart.

I did a study last fall where we tried to decompose some of this, and the companies really don’t tell you what they’re doing with that data, but I always want to caution that it doesn’t mean they’re not doing anything. In fact, when we did the study, Microsoft was the most concrete about saying that they were trying to strip personally identifiable information (PII) from documents—or actually, I don’t think they trained on documents at all—but they were trying to strip things out of people’s responses so that they didn’t get pulled into training data. OpenAI has recently come out with a tool that does some version of this that they’ve made available to the public, to developers. So again, I assume they’re using it. But are they? It’s not clear.

The bottom line is that we hope they’re cleaning the data a bit and not just pipelining it into their training. But if they are, and they’re not treating it appropriately, then the risk is that that data is memorized, that it potentially can come back out in contexts that the companies may be trying to prevent, but they can’t prevent everything. It comes down to guardrails.

How reliable is that process to remove PII? Is there a chance they can overlook something?

Oh sure…The low-hanging fruit will be known, structured data types—phone numbers, Social Security numbers, addresses—that’s the kind of thing they should be able to pull out if they want to. Much harder will be anything that I am descriptively saying about myself or my mental state, or a diagnosis. And that could be really hard, because, let’s say I’m not using a specified health context, I’m just chatting with ChatGPT or something, and I say, “I have a breast cancer diagnosis and I’m trying to figure out more about it.” They can try to index off terms like “breast cancer” and strip that or earmark the conversation and not include it. But I think you’re seeing very similar challenges with mental health, and especially in the states where they’re obligated to step in if somebody’s making statements of self-harm…The nuance can be lost, so somebody can have a conversation that really probably doesn’t qualify for that, but then suddenly they’re getting this referral, like, “Don’t engage in self-harm,” where it’s like, “I wasn’t even talking about that.” And of course the opposite might be true, too—that other things slip through the cracks.

It’s best-guess, because we have no access to internal data to see what they’re doing. The more troubling thing may be not so much about training data but more about, you are disclosing a lot of deeply personal information to the companies, and again, we just don’t know precisely what they’re doing with that internally.

With every level of the healthcare system now using different types of AI, what should healthcare workers be mindful of in terms of data privacy?

This is coming up more and more in my world right now…You shouldn’t be using a non-workplace-provided tool. And I don’t think most physicians are, I get the impression. Maybe your small provider could be, but if you’re part of a hospital or research university, you have some internal tool you’re using. And I was talking to a physician friend about it, and he was saying, “It’s actually pretty cool; when you’re working with something that’s trained on health literature, it can be really helpful.”

Whether you should be providing patient data to it that’s not anonymized is a different question. And I just don’t know [the answer.] I will say that from the study I did, we know that in most, if not the majority of enterprise contexts, these companies say they don’t train on enterprise data. So your company inks a contract with Anthropic, Anthropic is not going to pull that data into a general training pool, and I would presume in the healthcare context you do have HIPAA, so they can’t just send that data along down the pipeline without there being potential repercussions.

I know that Epic, the EHR medical records provider, they’re doing two things that I’m trying to dig into. One is that they’re training their own LLM on patient records. And that’s an interesting one because I haven’t been able to dig into the consent issues, because I think they’re doing it broadly across their whole record set…They also sell data. I encountered this from a research colleague on the med school side at Stanford, who said, “We just bought a bunch of patient data anonymized from Epic for the study that we’re doing,’ which actually surprised me. Epic might be an interesting case to look at to see how they’re handling this stuff…But beyond the consumer tools, there might be some interesting things going on more in these very specialized spaces. But broadly, the big commercial tools tend to guarantee that they’re not training on that data for broader model training purposes.

In terms of policy recommendations—or what can be done to improve health data privacy—what would you like to see?

There’s a handful of things I would say: one, just greater transparency, so that we have a better understanding of exactly what’s happening to all this data. My wish list would be that people aren’t forced to be opted in by default, that they can choose, although I’m always wary of forcing it on the user in a way that makes you have to really think about [the choice]. All of the concerns we’ve had about internet usage and social media really still apply here around third-party access to your data.

I just finished a whole paper on data brokers and data broker compliance with California law, and one of the interesting things about the California Data Broker Registry for this year, 2026, is that now the data brokers have to self-report whether they’re selling data to generative AI companies, and there’s about 30 companies in the registry right now that are doing that. So all of the tracking, data broker mass sales of data that we’ve seen in the internet for 20-plus years are now already showing up in the space, too. There’s a lot of the same problems.

Then on top of that, I think more transparency about how this whole training data space is put together, because we know that a lot of the foundation models were built by just insane amounts of web scraping, and that is still going on, but it’s shifting right now—we’re kind of in the mid-process, from what I can tell, of moving from the land-grab bonanza of “scrape everything” to the creation of these companies that straddle a line between data brokers and scrapers. They’re starting to create these specialized data sets to sell to the model developers, and again, very little transparency in that space.