Audit finds flaws in research regarding AI effectiveness in education

A forensic audit of 39 meta-analyses regarding the use of artificial intelligence in education has revealed significant conceptual and statistical flaws. While these studies collectively reported substantial benefits for AI in classrooms, an examination of 14 of these papers found recurring errors, including “impossible statistics” and “hallucinated references.”

One meta-analysis has already been retracted following complaints from Norwegian scholars. Despite these issues, the flawed papers have accumulated over 3,000 citations, contributing to a perception that the effectiveness of AI in education is settled science. Experts, including Justin Reich of the Massachusetts Institute of Technology, have characterized the existing body of meta-analytical evidence as untrustworthy.

Separate research highlights the cognitive implications of AI usage. A study at the MIT Media Lab using EEG measurements found that participants using language models exhibited weaker neural connectivity compared to those using search engines or writing independently. Researchers described this phenomenon as “cognitive debt,” noting that users often struggled to recall or quote what they had written. Additionally, research from Microsoft and Carnegie Mellon University suggests that higher confidence in AI tools correlates with a decrease in critical thinking among knowledge workers.