Survey: 57% of Clinicians Say GenAI Threatens Judgment, Not Jobs
By Matt Phillion
New data on generative AI from Wolters Kluwer looking at a survey of healthcare professionals finds that 57% of respondents overall believe generative AI may produce “an erosion of clinical decision-making skills caused by over-reliance” on these technologies. 55% of physicians and 53% of nurses agree.
Only 18% of respondents were aware of any clear generative AI policies or training in their organizations, opening up risks for rogue applications and misuse. Further, only 31% were aware of guidelines that delineated between their responsibilities and generative AI’s responsibilities.
“Our findings point to immediate concerns on how AI applications intended for frontline providers are being adopted to ensure that they are safe and effective,” says Dr. Peter Bonis, chief medical officer with Wolters Kluwer Health. “One is governance. Organizations must have processes in place to ensure that AI applications perform as expected over time, that involves not only vendor and application selection but training, vigilance around regulatory compliance, thoughtful insertion into workflow, and prioritization relative to other initiatives.”
There’s a great need for governance around generative AI right now, but most organizations don’t have that, Bonis says. This also plays into the need for adequate training and education.
“Only 18% of providers who responded said they were aware of any guidance from their institutions, and only 42% said their organizations have clear processes,” says Bonis.
“There is a clear gap between what they need to know and the process that these tools are using,” he adds. “It’s very significant concern and a reality check about these tools.”
There are many guides for safe and effective use of AI, but internally, a lot of things have to go right to make the tools true to their purpose, Bonis explains.
Getting the right teams—experts in workflow design, in technology, in AI itself—to create a coherent strategy for adoption and maintenance of these tools is necessary.
“Gen AI tools need to be considered along with other technology being adopted,” says Bonis. “There are larger organizations who are on the cutting edge who have internal oversight, but many smaller organizations will likely require third parties to help them vet relevant tools and assure safety and effectiveness.”
Hype vs. function
What tools are being discussed when it comes to generative AI needs to be clear from the start of any discussion.
“There’s so much hype relative to the current successful deployment of AI tools,” says Bonis, citing another survey of 43 major US healthcare systems conducted at the end of 2024. AI adoption and perceptions of success varied significantly.
Ambient notes, a generative AI tool for clinical documentation, was the only use case with 100% of respondents reporting adoption activities, but only 53% reported a high degree of success. Imaging and radiology were the most used clinical AI use cases but while 90% of organizations reported at least partial deployment, only 19% reported success. Similarly, many organizations deployed AI for clinical risk stratification (such as for early sepsis detection) but only 38% reported high success. Major barriers to successful adoption identified included immaturity of the AI tools (77%), financial concerns (47%) and regulatory uncertainty (40%).
“Organizations at the cutting edge are deploying the technology and finding all the bumps in the road,” says Bonis. “In many cases, they are not getting the expected value out of it yet, but they can if they take the right steps. We need to better understand how these tools are working and what can be realistically expected.”
There’s a lot of excitement around adoption, Bonis explains but if you look at the literature, he notes, there’s a lot of cautionary tales about how these models can go sideways, or not perform well, and those stories get less attention.
“Patient care is about as high stakes as you can get,” says Bonis. “The ease of use of these models, especially the chat models, is so appealing that there’s an overwhelming temptation to adopt before the full spectrum of safety is understood, and that comes back to governance. The technology is not regulated to the point where it is going to self-police, so organizations need to have policies in place for the evaluation of these tools and how they’ll be used by healthcare professionals.”
Catching hallucinations—or not
The issue of generative AI and hallucinations—the tool providing information that sounds correct but is not—is well known, but there’s also a belief that professionals will recognize and catch those hallucinations, Bonis says.
“Busy healthcare professionals may not investigate the answer the tool gives them,” he says. “Even sometimes the references it cites are hallucinated.”
But what about when the person asking the query is a level of expert that they should recognize incorrect information, but they miss it?
Experts seek information because they want to confirm their understanding or have identified a relevant information gap within or outside of their specialty. When we look at this in detail, experts do not routinely identify subtly or grossly incorrect answers provided by generative AI.
Furthermore, studies of generative AI for clinical use cases have identified many additional shortcomings that are common yet overshadowed by overconfidence and hype. Search reliability varies between sessions and with subtle changes in prompts, various biases (racial and cognitive) have crept into responses, automaticity can lead to actions without adequate thought, the models struggle when messy data are presented serially, responses are sometimes out-of-date, and overly efficient responses can deprecate context. There is also emerging literature suggesting that over-reliance on generative AI has the potential to impede learning and critical thinking.
The survey, in looking at top organizational priorities, found that optimizing workflow is one of the most needed improvements when it comes to AI use.
“But only 63% said they were prepared to use it,” says Bonis. “Organizations must engage with staff about the availability of these tools and how to use them, build more formal processes around governance, and amplify limitations.”
Healthcare professionals have been using all kinds of options for information seeking over the years without a formal governance process. The situation with generative AI is a bit different, Bonis notes.
“With major search engines, you have (over the pre-generative AI years) retrieved links to relevant resources, which to an extent, leaves assessment of credibility up to the user. But with generative AI, you get a compelling and actionable answer, the validity of which is uncertain,” he says.
If one defines “rogue adoption” of technology as adopting it without fully understanding the limitations and risks, there is the potential for that here, and so institutions will need to articulate the preferred tools they stand behind.
“In my view, when a provider is using a technology outside of an officially sanctioned, well-governed process, and making decisions on behalf of patients, there’s a lot of underappreciated risks—potentially exposing PHI, suboptimal care, and possibly patient harm, or even exposing data to targeted marketing or other uses of their time-stamped search queries.” Bonis explains. “Enterprises need to be responsible for governance, and many haven’t gotten their arms around it yet.”
There is so much emphasis on the potential for AI and sentiment to get on board or get left behind. There’s a struggle to move fast enough even as the governance around it hasn’t fully matured.
“AI has a lot of potential to create value in the healthcare domain,” says Bonis. “But we are in a hype cycle with an explosion of tools targeting healthcare.”
There’s a need for enterprises and vendors to lean in responsibly and look for ways to solve healthcare challenges. “We don’t want to hamstring the development of technology,” Bonis says, “but we do need guardrails.”
“There is substantial experimentation taking place at a fast pace,” he adds. “The tools themselves not only have to perform well but deliver enterprise value. We need thoughtful onboarding in a disciplined way to solve problems and have enough governance around that to make sure we’re not getting ourselves into trouble.”
“It’s been highly disruptive. This is a high-stakes domain, and if you introduce errors the effect is immediate. It’s our mission and responsibility to get it right,” says Bonis.
Matt Phillion is a freelance writer covering healthcare, cybersecurity, and more. He can be reached at matthew.phillion@gmail.com.