As healthcare systems face strain, many turn to AI chatbots like ChatGPT for medical guidance. A survey indicates one in six U.S. adults uses chatbots for health advice monthly. However, an Oxford Internet Institute study highlights that these tools often deliver unreliable advice due to communication challenges.
The study, involving 1,300 U.K. participants, used doctor-designed medical scenarios. Participants identified health conditions and decided on actions, such as visiting a doctor, using chatbots or traditional methods like online searches. Tested AI models included GPT-4o (ChatGPT’s default), Cohere’s Command R+, and Meta’s Llama 3.
Adam Mahdi, study co-author, told TechCrunch, “The study revealed a two-way communication breakdown.” Users often left out key details in queries, resulting in chatbot responses that mixed accurate and misleading advice. This complicated informed health decisions. The study found chatbot users were less accurate in identifying conditions and more likely to underestimate severity compared to those using other methods.
The findings reveal a gap between AI’s healthcare potential and its current limitations. Companies like Apple and Microsoft are developing AI for health applications, but skepticism persists among doctors and patients about its reliability in critical scenarios. The American Medical Association advises against using chatbots like ChatGPT for clinical decisions, and OpenAI cautions against relying on its tools for diagnoses.
Mahdi emphasized the need for improved testing, stating, “Current evaluation methods for [chatbots] do not reflect the complexity of interacting with human users.” Unlike doctors, chatbots cannot ask follow-up questions or interpret non-verbal cues, which are crucial in medical contexts.
The study recommends relying on trusted sources for healthcare decisions, Mahdi noted. The findings suggest that until chatbots better handle human interactions, professional medical guidance remains more reliable than AI advice, according to the researchers.