MIT Study Finds Explainable AI Misleads Non‑Experts in Diagnosis

Researchers at MIT evaluated how explainable artificial intelligence (XAI) influences diagnostic decisions for skin diseases. The study showed that non‑expert participants improved overall accuracy, but largely because they deferred to the AI’s suggestions, even when the large‑language‑model explanations were incorrect. In contrast, clinicians performed best when they received only the model’s prediction without any explanatory overlay, indicating that detailed explanations can increase automation bias among users with limited medical knowledge.

A separate exploratory qualitative study examined eight licensed psychologists’ reactions to an XAI system that predicts depression. The participants highlighted potential benefits for counselor education, such as greater diagnostic transparency and reflective supervision, while also raising concerns about data quality, ethical issues, and preserving human judgment. Both studies underscore the need to balance AI assistance with user expertise to avoid unintended errors in health‑related decision‑making.