started · updated
AI giants adopt unified robots.txt standard for training opt-outs
OpenAI, Google, Microsoft, and Anthropic have pledged to support a new, unified technical standard to allow website owners and publishers to block their content from being used for AI model training. The initiative, based on an IETF draft authored by Anthropic’s Kevin Kelley, introduces a specific policy token, “__usage_ai-training”, to be used within robots.txt files.
This standardized signal aims to replace the current fragmented system where publishers must manually create separate blocking rules for every individual AI crawler. The new protocol distinguishes between training preferences and general web crawling: using the opt-out signal does not prevent search engines like Google or Bing from indexing sites or appearing in search results.
Implementation timelines vary among the participating companies. Anthropic has already begun rolling out the signal. OpenAI plans to implement it in the coming months. Microsoft and Google have scheduled their support for early 2027, with Microsoft specifically noting it will begin honoring the new settings on February 1, 2027.
Entities
Anthropic · Google · Internet Engineering Task Force · Kevin Kelley · Microsoft · OpenAI
Claims
What the coverage asserts, and how many sources carry each claim.
- [● 29 SOURCES] OpenAI, Google, Microsoft, and Anthropic pledged to support a unified robots.txt standard for blocking AI training scraping. tob.news · ihal.it · www.prodiris.fr · 24-ore.com · www.batista70phone.com · +23 more
- [● 24 SOURCES] Anthropic has already started rolling out the new signal. tob.news · savedelete.com · www.ertnews.gr · world-today-journal.com · mashdigi.com · +18 more
- [● 24 SOURCES] Google plans to support the new control in 2027. tob.news · savedelete.com · ppc.land · geekyalgeria.com · world-today-journal.com · +18 more
- [● 21 SOURCES] Microsoft will implement the new standard beginning February 1, 2027. tob.news · savedelete.com · www.pravda.sk · world-today-journal.com · www.24hod.sk · +16 more
- [● 20 SOURCES] The new setting applies only to future training of models and does not prevent general crawling. tob.news · www.techinside.com · geekyalgeria.com · www.24hod.sk · mashdigi.com · +14 more
- [● 23 SOURCES] The proposal was drafted by Kevin Kelley of Anthropic. ihal.it · savedelete.com · www.ertnews.gr · geekyalgeria.com · world-today-journal.com · +17 more
- [● 22 SOURCES] The new standard uses the token “__usage_ai-training” within the robots.txt file. ihal.it · savedelete.com · ppc.land · geekyalgeria.com · mashdigi.com · +16 more
- [● 22 SOURCES] OpenAI plans to implement the signal in the coming months. tob.news · ihal.it · 24-ore.com · www.techinside.com · www.theverge.com · +17 more