# News publishers and AI companies content access dispute > Developing situation record from CLSTR: https://clstr.news/situations/news-publishers-and-ai-companies-content-access-dispute > Updated: 2026-11-02T02:42:11.000Z. Sources: 52. Developments: 3. Major news organizations and hundreds of local US outlets have begun blocking the Internet Archive from accessing their content. Publishers argue the archive serves as a “back door” to bypass paywalls and licensing, and they express concerns that AI companies use such access to train large language models without authorization. In response to the complexities of managing individual crawler blocks, major AI companies including OpenAI, Google, Microsoft, and Anthropic have pledged to support a unified technical standard. This proposed mechanism uses a shared policy token, “__usage_ai-training”, in robots.txt files to allow website owners to opt out of AI model training via a single instruction. While the standard is voluntary and does not affect search engine indexing, implementation timelines vary. Anthropic has already begun rolling out the signal, while OpenAI plans to implement it in the coming months. Microsoft is scheduled to adopt the standard starting February 1, 2027, and Google intends to support it during 2027. The proposal, drafted by Kevin Kelley of Anthropic, is currently under review by the Internet Engineering Task Force (IETF) AI Preferences working group. Microsoft has noted that the new settings will distinguish between using content for model training and using it for grounding, which allows AI to reference external information when generating responses. This distinction enables publishers to block training while still allowing their content to be cited in AI-generated answers. However, companies have warned that opting out of training may reduce the visibility of publisher content within AI applications and experiences. ## Claims - OpenAI, Google, Microsoft, and Anthropic pledged to support a unified robots.txt standard for blocking AI training scraping. (reported by 29 sources) - Anthropic has already started rolling out the new signal. (reported by 24 sources) - Google plans to support the new control in 2027. (reported by 24 sources) - The proposal was drafted by Kevin Kelley of Anthropic. (reported by 23 sources) - The new standard uses the token “__usage_ai-training” within the robots.txt file. (reported by 22 sources) - OpenAI plans to implement the signal in the coming months. (reported by 22 sources) - Microsoft will implement the new standard beginning February 1, 2027. (reported by 21 sources) - The new setting applies only to future training of models and does not prevent general crawling. (reported by 20 sources) ## Timeline ### 2026-11-02: AI companies announce unified robots.txt standard for training control OpenAI, Google, Microsoft, and Anthropic are adopting a unified robots.txt standard to let websites control AI training usage via a single signal. 12 sources. https://clstr.news/cluster/ai-companies-agree-on-standard-to-block-training-data-scraping ### 2026-10-30: AI giants adopt unified robots.txt standard for training opt-outs OpenAI, Google, Microsoft, and Anthropic will support a unified robots.txt signal, allowing publishers to block AI training scraping with a single command without affecting search engine indexing. 41 sources. https://clstr.news/cluster/openai-google-microsoft-and-anthropic-adopt-new-ai-scraping-opt-out-standard ### 2026-10-20: Internet Archive faces access blocks from major news publishers Major news publishers, including The New York Times and The Guardian, are blocking the Internet Archive to prevent AI companies from using their content for model training. 2 sources. https://clstr.news/cluster/internet-archive-faces-access-blocks-from-major-news-publishers --- Cite as: News publishers and AI companies content access dispute. CLSTR, https://clstr.news/situations/news-publishers-and-ai-companies-content-access-dispute