Artificial intelligence (AI) is expected to shake up the future of healthcare as more companies leverage the tool to scan medical records for data and recommend potential treatment pathways.
But the success of those capabilities, health and tech leaders have warned, will rely on the accuracy of the underlying electronic health records (EHR) data.
That’s why researchers with COTA, a New York-based company that uses AI to organize and analyze oncology data, are testing the quality and accuracy of real-world medical records—including mortality rates—for patients with certain types of cancer.
C.K. Wang, an oncologist and COTA’s chief medical officer, told Healthcare Brew that provider EHR data doesn’t “fully reflect the complete death information that is out there” because physicians often aren’t notified when a patient dies outside of the healthcare system—and there’s little incentive for doctors to collect that information. Even when a death is reported, he added, it may not always be recorded properly in an EHR.
Wang cautioned that incomplete patient data could result in biased or incorrect recommendations for physicians using AI-powered software to aid in clinical decisions, like which treatments would be most effective for different conditions.
In response, COTA researchers recently compared the real-world data for 21,500+ patients diagnosed with certain blood cancers—including acute myeloid leukemia, chronic lymphocytic leukemia, diffuse large B-cell lymphoma, follicular lymphoma, marginal zone B-cell lymphoma, multiple myeloma, and myelodysplastic syndrome—diagnosed between January 2015 and December 2020 against the National Death Index (NDI), which is known as the “gold standard” for mortality data.
Navigate the healthcare industry
Healthcare Brew covers pharmaceutical developments, health startups, the latest tech, and how it impacts hospitals and providers to keep administrators and providers informed.
They found varying rates of accuracy across cancer types and age groups, with younger patients (ages 18 to 29) having the lowest sensitivity, according to a report released in December 2023.
Overall, however, the real-world mortality data and NDI death data largely matched up, with a sensitivity of about 88% and specificity of nearly 96%—“meaning that it was relatively complete and that when death information was present, it was highly accurate,” Wang said. Still, additional research is needed to assess the methodology and expand its use.
“Here at COTA, we believe that 100% is an almost unachievable goal, but we want to get there as much as we can. I do believe that getting close to 90% is meaningful [for] sensitivity and specificity,” Wang said. “For any data that is used to train large language models or any AI tools, there needs to be a concerted focus not just on accuracy but [also on] the completeness of the data.”
COTA researchers plan to examine death data for patients with solid tumor cancer next, he added.