[CLSTR]
API
< Back to situations

Monitor this situation.

[SITUATION] · [ACTIVE] · [TECHNOLOGY]

3 clusters · 52 sources · 13 days · First seen · Last updated

News publishers and AI companies content access dispute

Overview

Major news organizations and hundreds of local US outlets have begun blocking the Internet Archive from accessing their content. Publishers argue the archive serves as a “back door” to bypass paywalls and licensing, and they express concerns that AI companies use such access to train large language models without authorization.

In response to the complexities of managing individual crawler blocks, major AI companies including OpenAI, Google, Microsoft, and Anthropic have pledged to support a unified technical standard. This proposed mechanism uses a shared policy token, “__usage_ai-training”, in robots.txt files to allow website owners to opt out of AI model training via a single instruction. While the standard is voluntary and does not affect search engine indexing, implementation timelines vary. Anthropic has already begun rolling out the signal, while OpenAI plans to implement it in the coming months. Microsoft is scheduled to adopt the standard starting February 1, 2027, and Google intends to support it during 2027. The proposal, drafted by Kevin Kelley of Anthropic, is currently under review by the Internet Engineering Task Force (IETF) AI Preferences working group.

Microsoft has noted that the new settings will distinguish between using content for model training and using it for grounding, which allows AI to reference external information when generating responses. This distinction enables publishers to block training while still allowing their content to be cited in AI-generated answers. However, companies have warned that opting out of training may reduce the visibility of publisher content within AI applications and experiences.

Entities

Google · OpenAI · Microsoft · Anthropic · Internet Engineering Task Force

Claims

What the coverage asserts, and how many sources carry each claim.

Timeline

  1. [TECHNOLOGY] 12 sources
    AI companies announce unified robots.txt standard for training control

    OpenAI, Google, Microsoft, and Anthropic are adopting a unified robots.txt standard to let websites control AI training usage via a single signal.

  2. [TECHNOLOGY] 41 sources
    AI giants adopt unified robots.txt standard for training opt-outs

    OpenAI, Google, Microsoft, and Anthropic will support a unified robots.txt signal, allowing publishers to block AI training scraping with a single command without affecting search engine indexing.

  3. [TECHNOLOGY] 2 sources
    Internet Archive faces access blocks from major news publishers

    Major news publishers, including The New York Times and The Guardian, are blocking the Internet Archive to prevent AI companies from using their content for model training.

Sources

24-ore.com · 24hod.sk · aimagazine.ca · astratv.gr · batista70phone.com · biznis.ba · bodhi.co.jp · bright.nl · brockallen.com · cafebiz.vn · cafef.vn · classics.itmedia.co.jp · econoTimes.com · ertnews.gr · geekyalgeria.com · genk.vn · googlediscovery.com · ihal.it · infodata.ilsole24ore.com · iussp.org · khabaragency.net · kienthuc.net.vn · linformatique.org · mashdigi.com · media-innovation.jp · mercado.etc.br · news.mynavi.jp · nguoi-viet.com · nucleo.jor.br · plagiarismtoday.com · ppc.land · prodiris.fr · sada-tabuk.com · savedelete.com · sciencemag.cz · searchenginejournal.com · sector.sk · senigallianotizie.it · siliconangle.com · spacemoney.com.br · spravy.pravda.sk · techenet.com · techinside.com · technology.inquirer.net · thanhnien.vn · thewrestlinguniverse.com · tob.news · vtv.vn

This summary has been updated 3 times: see revision history