Focusing on the Misuse of AI Chatbots in Healthcare
By Matt Phillion
According to a report from ECRI, the use of artificial intelligence (AI) chatbots in healthcare tops the list of most significant healthcare risks in 2026.
“It’s becoming very mainstream,” says Francisco Rodriguez-Campos, PhD, a Principal Project Officer in the Device Safety team at ECRI. “I think we’ve seen over the past year that all of them have very public issues, and this is something we need to address.”
With the explosion of AI tools over the past few years, healthcare, like all industries, is seeing rapid adoption and growing reliance on this technology. And while chatbots that rely on large language models like ChatGPT, Claude, Copilot, Gemini, and Grok, produce human-like and confident-sounding responses to the users’ questions, they are not regulated as medical devices and are not validated for healthcare purposes. Despite this, clinicians, healthcare professionals, and even patients use them at an ever-increasing frequency. According to a recent analysis from OpenAI, the maker of ChatGPT, more than 40 million people use their LLM for health information.
“It’s not quite a medical device, but it is really having an impact on patient safety,” says Rodriguez-Campos. “The issue is everyone is using it like it’s the new Google.”
Confident, but not always correct
ECRI notes that chatbots can provide valuable assistance but also can provide false or misleading information that can result in significant patient harm.
“It’s a flawed technology with a lot of potential,” Rodriguez-Campos says. “You can see the places where it has potential in areas like discharge notes, or for noting when you may need to take out medical jargon or use plain language depending on the education of the reader.”
It is for this reason ECRI advises caution when using a chatbot for information that can impact patient care. AI systems generate responses by predicting sequences of words based on patterns learned from their training data rather than truly understanding context or meaning the way a human would. They are also programmed to sound confident and to always provide an answer to the user, even if that answer isn’t a reliable one, ECRI’s report explains.
The ECRI report notes that chatbots have:
- Suggested incorrect diagnoses
- Recommended unnecessary testing
- Promoted subpar medical supplies
- And evented body parts in response to medical questions
All in the confident tone of a trusted medical expert. It’s not verifying that those answers are correct by a trusted expert where risks arise.
“You can imagine where you’re in a fast-paced environment this can be useful. We used to rely on a book on drug interactions, and then we had it on a computer or smartphone, and now the thought is well, AI will help me with this even better when I need to quickly check this or that,” says Rodriguez-Campos. “Things you don’t remember off the top of your head. But what’s different with AI is it hallucinates. It’s always trying to give you an answer and doesn’t say when it doesn’t know.”
In one example, ECRI researchers asked whether it would be acceptable to place an electrosurgical return electrode over the patient’s shoulder blade, which is known to leave the patient at risk of burns. The chatbot stated the placement was appropriate.
“We asked the chatbot if the electrodes were placed right and it says sure, you’re fine—but in reality, that wasn’t right,” says Rodriguez-Campos. “We also saw situation where a patient would say I didn’t understand the instructions my doctor gave me, but I did understand what ChatGPT said” when the instructions did not match up.
AI use and disparities in care
The ECRI report also notes that using chatbots for health decisions can lead to greater risks as higher healthcare costs, closing hospitals and healthcare facilities, and other economic challenges can lead to even more reliance on the tools. With patients turning to technology as a substitute for actual professional advice due to lack of access, more hidden dangers continue to emerge.
“More of the general population are starting to use this instead of a doctor, listening to AI instead of actually looking for professional medical guidance,” says Rodriguez-Campos. “Maybe insurance isn’t covering them, or they can’t get their next appointment for six months. There’s a lot of reasons why someone would be compelled to say, ‘Let me ask the machine.’”
This also highlights the matter of healthcare disparity, according to ECRI experts. Biases embedded in the data these chatbots are trained on can warp how the models interpret information. This can reinforce stereotypes and inequities among patient populations.
ECRI’s report suggested methods for using chatbots more wisely. Better education for not only clinicians but patients and their caregivers can help highlight the limitations on the technology and build in the practice of verifying any information provided by these tools with a trusted, verifiable source.
“The key is education. And this is something that has to trickle down from the leadership level,” says Rodriguez-Campos. “When you are going to adopt AI you start with the promise of better ROI, improved processes, faster care so you can see more patients. In reality, you have to have a human in the loop.”
This trickles down to patients as well.
“It’s fine if you check with the chatbot, but always chick with your clinicians to make sure you don’t do what it’s telling you to do without verifying” with a trusted human source, says Rodriguez-Campos.
It’s not just patients using AI tools at home to watch out for. Organizations deploying LLMs that interact with patients, from checking into the hospital for a visit to picking up prescriptions, it’s important to remember and educate toward the limitations of the technology.
Leadership’s role
Health systems should also look at promoting responsible use of AI tools through proper governance and by providing clinicians with training on appropriate use. They should also regularly audit these tools’ performance for veracity.
“The big thing is governance,” he says. “We’ve been talking about how you need to choose what processes you want to implement with AI, define your metrics, and see if it really works before you deploy. You have to hit the brakes a little and really think about what you’re doing. It’s not magic.”
Is there a future where AI is considered a medical device and is regulated as such?
“I’m not sure, to tell you the truth,” Rodriguez-Campos says. “Some institutions are deploying these tools for the patient, but also starting to look at having an enclosed LLM that lives in their intranet that only has access to information from the inside.”
They can ask patients if they are comfortable having their models look at their data to provide a better solution, but with that comes privacy issues.
“The FDA already has regulations for software and medical devices, and I think that may be a good starting point,” Rodriguez-Campos says.
Many of the guardrails we can expect to see around the technology will involve liability, Rodriguez-Campos notes.
“No matter what the tool tells you, you have to sign off on it. I think clinicians are aware and know they’re on the hook for that,” says Rodriguez-Campos. “Look at imaging. AI has been around in imaging for a while and it’s less of an issue because the radiologist is always looking—they don’t blindly trust the AI output.”
On the clinical side, we must continue to remind users that at the end of the day, the responsibility for decisions comes from the clinician, not the AI tool, and thus so lies the liability.
“It’s not perfect. I’ve even seen some models include a disclaimer that says it shouldn’t be trusted for medical advice,” says Rodriguez-Campos. “I saw one recently that said, ‘This makes mistakes, so please always check.’”
In the end, we need to ensure there is a human in the loop when using these tools to keep patients safe, Rodriguez-Campos notes. In some ways, it’s a bit like the days of patients diagnosing themselves online before an appointment.
“It’s WebMD all over again. The machine’s output sounds right! But we need to proceed with care,” says Rodriguez-Campos. “I spoke with a physician who used it AI to create a treatment plan and he was happy to see the plan came with journal references, only to find those references were made up. That’s the scary part. And now imagine a person with no medical knowledge encountering this. If it sounds too good to be true, it probably is.”
Matt Phillion is a freelance writer covering healthcare, cybersecurity, and more. He can be reached at matthew.phillion@gmail.com.