[REVISION HISTORY]

News publishers and AI companies content access dispute

3 recorded revisions of this situation.

What changed

2026-11-01 02:49 UTC → 2026-11-02 04:01 UTC · added ~~removed~~

Major news organizations and hundreds of local US outlets have begun blocking the Internet Archive from accessing their content. Publishers argue the archive serves as a “back door” to bypass paywalls and licensing, and they express concerns that AI companies use such access to train large language models without authorization. In response to the complexities of managing individual crawler blocks, major AI companies including OpenAI, Google, Microsoft, and Anthropic have pledged to support a unified technical standard. This proposed mechanism uses a shared policy token, “__usage_ai-training”, in robots.txt files to allow website owners to opt out of AI model training via a single instruction. While the standard is voluntary and does not affect search engine indexing, implementation timelines vary. Anthropic has already begun rolling out the signal, while OpenAI plans to implement it in the coming months. Microsoft is scheduled to adopt the standard starting February 1, 2027, and Google intends to support it during 2027. The proposal, drafted by Kevin Kelley of Anthropic, is currently under review by the Internet Engineering Task Force (IETF) AI Preferences working group. Microsoft has noted that the new settings will distinguish between using content for model training and using it for grounding, which allows AI to reference external information when generating responses. This distinction enables publishers to block training while still allowing their content to be cited in AI-generated answers. However, companies have warned that opting out of training may reduce the visibility of publisher content within AI applications and experiences.

Versions

Select a version to see what changed compared to the one before it.