llms.txt file adoption shows limited impact on AI visibility

The llms.txt file, originally proposed by Jeremy Howard to help coding agents parse software documentation, has seen widespread adoption despite limited utility for general AI visibility. An analysis by Ahrefs of 137,000 domains found that while 28% of sites hosted the file, 97% of those files received no requests during the measured period.

Traffic to these files is primarily driven by generic SEO crawlers and specific agentic tools, such as Anthropic’s coding agent and OpenAI’s GPTBot. Most AI systems used for answering general questions do not currently utilize the file.

A second version of the proposal (v2) was released in August 2026 to improve discoverability. This update defines how files cover specific paths and allows pages to announce their llms.txt location via HTML or HTTP headers. Major platforms, including WordPress, OpenAI, and Claude, have implemented the file to support AI-driven documentation and site parsing.