University of Pennsylvania study warns of AI cognitive surrender

Researchers at the University of Pennsylvania’s Wharton School have identified a phenomenon called ‘cognitive surrender,’ where users tend to accept AI-generated reasoning without critical evaluation. A study involving 1,372 participants and over 9,500 tests using a GPT-4o based chatbot revealed that while users accepted correct answers 92.7% of the time, they continued to follow incorrect AI responses in 79.8% of cases.

This tendency is exacerbated by ‘AI hallucinations,’ a process where large language models (LLMs) generate factually incorrect information that appears highly coherent and convincing. Because LLMs function as prediction engines rather than databases, they may invent dates, laws, or web links to complete a statistically probable sentence structure when they lack clear data. This discrepancy between formal linguistic coherence and factual truth poses a significant risk to users who may stop performing independent analysis.