LLMs.txt emerges as a tool for AI discoverability

The emergence of the ‘llms.txt’ file is providing a new method for improving AI discoverability for websites, particularly in the ecommerce sector. Unlike robots.txt, which controls crawler access, or XML sitemaps, which list URLs, an llms.txt file acts as a suggested reading list or curated map to help large language models (LLMs) and AI retrieval systems find and interpret key content.

Mage-OS has introduced a module for Magento Open Source and Mage-OS to automate this process. The module generates several types of feeds, including /llms.txt for concise information, /llms-full.txt for extended details like social profiles and category trees, and /llms.jsonl for compact JSON-LD data. It also includes AI crawler directives within robots.txt to manage access for specific AI bots.

While the file is gaining attention among brands looking to increase visibility in tools like ChatGPT, Claude, and Perplexity, it is currently an emerging concept rather than a formal web standard.