Adoption and utility of llms.txt for AI discoverability

The ‘llms.txt’ file is an emerging method designed to improve AI discoverability by providing curated maps or reading lists for large language models and retrieval systems. Unlike traditional robots.txt files that control access, llms.txt aims to help AI agents interpret key content.

Early analysis indicated that while adoption was growing, the file had limited impact on general AI visibility, with a study finding that 97% of hosted files received no requests during the measured period. Most traffic was limited to specific agentic tools and SEO crawlers rather than general-purpose AI systems. To address discoverability, a second version of the proposal was released in August 2026, allowing pages to announce file locations via HTML or HTTP headers.

As the concept develops, specialized tools have emerged to automate the process, such as a module for Magento Open Source and Mage-OS that generates various feeds including /llms.txt, /llms-full.txt, and /llms.jsonl. While brands are increasingly looking to use these files to increase visibility in tools like ChatGPT, Claude, and Perplexity, the format remains an emerging concept rather than a formal web standard.