AI Chatbots Pose Risks When Used for Medical Advice

Security firm ESET warned that the growing use of AI‑driven health chatbots can expose users to inaccurate diagnoses, privacy breaches and unclear liability. The firm highlighted that tools such as Copilot Health, ChatGPT Health and Amazon HealthAI often provide advice that mimics professional medical guidance but lack the nuanced judgment of trained physicians, increasing the chance of misdiagnosis and inappropriate treatment.

A large user study from the Oxford Internet Institute and the Nuffield Department of Primary Care Health Sciences confirmed these concerns. Nearly 1,300 participants who consulted large language models for health scenarios performed no better than those who used traditional online searches or personal judgment. Researchers noted a “two‑way communication breakdown,” with users unsure what information to supply and models delivering a mix of correct and incorrect recommendations. As Dr. Rebecca Payne put it, “AI just isn’t ready to take on the role of the physician.” The study calls for real‑world testing of AI systems before deployment and stresses the need for clear regulatory frameworks to protect patient data and ensure accountability.