AI companies announce unified robots.txt standard for training control

AI training robots.txt OpenAI Google data scraping

Major AI developers, including OpenAI, Google, Microsoft, and Anthropic, have announced support for a unified standard to allow website owners to control how their content is used for AI training. The initiative aims to simplify the current process, where publishers must set individual blocking instructions for every different AI crawler.

The proposed standard utilizes a new token, “__usage_ai-training”, within the robots.txt file. By adding this specific instruction, webmasters can express a single preference to block or allow multiple major AI companies from using their site content for model training. This effort is based on a draft submitted to the Internet Engineering Task Force (IETF) AI Preferences working group.

Implementation timelines vary among the companies. OpenAI expects to adopt the signal within the coming months, and Anthropic has already begun implementation. Microsoft plans to apply the new settings starting February 1, 2027, and will also introduce mechanisms to distinguish between model training and grounding (referencing content for real-time answers). Google aims to implement the standard in 2027.

While the standard simplifies technical management, it remains a voluntary protocol. It does not provide new legal tools for rights holders, nor does it prevent crawling for purposes other than training. For general commercial websites, experts suggest that allowing AI training may be beneficial to ensure AI models provide accurate information about their products and services.

Entities

Anthropic · Google · IETF · Internet Engineering Task Force · Microsoft · OpenAI

Claims

What the coverage asserts, and how many sources carry each claim.

[● 3 SOURCES] OpenAI aims to implement the new standard within the next few months. news.mynavi.jp · www.zdnet.fr
[○ 1 SOURCE] Microsoft will allow users to distinguish between AI training and grounding controls. news.mynavi.jp
[● 4 SOURCES] OpenAI, Google, Microsoft, and Anthropic support a unified robots.txt signal for AI training control. news.mynavi.jp · www.zdnet.fr
[● 2 SOURCES] The new standard is based on a draft submitted to the IETF AI Preferences working group. news.mynavi.jp · www.zdnet.fr
[● 3 SOURCES] Anthropic has already begun implementing the new standard. news.mynavi.jp · www.zdnet.fr
[● 2 SOURCES] Microsoft plans to implement the new settings on February 1, 2027. news.mynavi.jp
[● 2 SOURCES] Google plans to support the common specification in 2027. news.mynavi.jp
[● 3 SOURCES] The new standard uses the token “__usage_ai-training” in robots.txt. news.mynavi.jp · www.zdnet.fr