Mass General Brigham creates AI testing grounds for hospitals

It seems a new artificial intelligence (AI) tool promising to improve efficiency in healthcare and reduce clinician burnout comes out nearly every day. According to Deloitte, 75% of “leading healthcare companies” are experimenting with generative AI.

In response, Mass General Brigham is getting health systems together to test out and compare notes on emerging AI tools from companies including Microsoft, Google, and OpenAI and also look at how these tools perform in clinical settings. In an announcement on November 13, the Boston-based health system touted the venture—called the Healthcare AI Challenge—as the “first-of-its-kind interactive virtual environment” that lets clinicians assess AI solutions.

Participants will gain access to new AI tools used for specific tasks, like evaluating medical images, which they can test in simulations and provide feedback on performance. That commentary will be publicly available so health systems that aren’t involved in the partnership can learn from the program, according to the news release.

The first in the series of challenges will focus on radiology, as AI has already had a “rapid and meaningful impact” in the specialty, according to Mass General Brigham.

“We are facing an overwhelming influx of FDA-approved AI tools in healthcare, especially in radiology,” Dushyant Sahani, chair and professor of the department of radiology at the University of Washington School of Medicine, said in a statement. “Forming an academic collaborative could play a crucial role in validating and selecting these tools, ensuring they adhere to the highest standards of efficacy and safety.”

Atlanta-based Emory Healthcare as well as the radiology departments at the University of Wisconsin School of Public Health and the University of Washington School of Medicine have signed on as partners for the initiative. Mass General Brigham said in the news release that other health system partners will be announced at a later date.

The American College of Radiology, a professional group that represents radiologists, has also signed on.

Setting standards. The program comes as healthcare professionals try to set standards for incorporating AI into their work. In March, a group of health systems, healthcare organizations, and AI and data scientists formed the Coalition for Health AI, or CHAI, which released draft guidelines for “best practices when designing, developing, and deploying AI within healthcare workflows” in June.

“We need to go beyond collaboratives that come to consensus on how to think about AI,” Alistair Erskine, Emory Healthcare’s chief information and digital officer, said in a statement. “We need healthcare delivery communities to provide real-world experience of the application of AI at the point of care. That is what the Healthcare AI Challenge is designed to do.”