AI Chatbots for Medical Advice Pose Risks Amid Healthcare Strains

As healthcare systems face long waiting lists and rising costs, many individuals are increasingly turning to AI-powered chatbots such as ChatGPT for medical self-diagnosis and health advice. A recent survey found that approximately one in six American adults use chatbots for health-related guidance at least once a month.

However, a recent study led by researchers at the Oxford Internet Institute highlights significant risks associated with overreliance on these AI chatbots. The study revealed a two-way communication breakdown between users and chatbots, where users often failed to provide key information and received responses that mixed accurate and inaccurate recommendations.

In the study, around 1,300 participants in the U.K. were presented with medical scenarios crafted by doctors. They were tasked with identifying potential health conditions and determining appropriate actions using chatbots powered by models like GPT-4o, Cohere’s Command R+, and Meta’s Llama 3, alongside traditional methods such as online searches and personal judgment.

Results showed that chatbot users were less likely to correctly identify relevant health conditions and tended to underestimate the severity of those they did recognize. This was partly due to participants omitting important details when querying chatbots and receiving answers that were difficult to interpret.

Adam Mahdi, co-author of the study and director of graduate studies at the Oxford Internet Institute, emphasized that current evaluation methods for chatbots do not adequately reflect the complexities of human interaction. He recommends that, similar to clinical trials for medications, AI chatbot systems should undergo rigorous real-world testing before widespread deployment in healthcare.

The findings come amid growing efforts by major tech companies to integrate AI into health services. Apple is developing AI tools for lifestyle advice, Amazon is exploring AI to analyze social determinants of health, and Microsoft is building AI to triage patient messages to care providers. Despite this enthusiasm, both medical professionals and patients remain cautious about AI's readiness for higher-risk clinical applications.

The American Medical Association advises against physicians using chatbots like ChatGPT for clinical decision-making, and leading AI companies caution users not to rely on chatbot outputs for diagnoses. Trusted sources and professional medical advice remain essential for healthcare decisions.

This study underscores the importance of developing AI health tools that can effectively communicate with users, interpret complex health information accurately, and support safer decision-making. As AI continues to evolve, integrating rigorous testing and human-centered design will be critical to realizing its potential in healthcare without compromising patient safety.