llms.txt

The file is meant to be read at retrieval time, giving a model a concise map of a site instead of raw HTML^[1]. Search Engine Land's March 2025 coverage listed Anthropic, Hugging Face, Perplexity and Zapier among the sites already serving one, mostly for their own developer documentation^[4].

A twelve-month monitoring run across more than 3 million sites counted 4,088 llms.txt instances in June 2025 and 36,120 by May 2026, an 8.8x rise; including the companion llms-full.txt and ai.txt formats the total reaches 38,980 sites^[2].

Server logs answer the question directly, and the answer is close to no. Of the requests llms.txt files did receive in the Ahrefs sample, AI bots accounted for 19.5%, the rest being SEO audit tools, general crawlers and unidentified bots^[3]. A 14-day log study on one domain recorded 723 robots.txt fetches by 18 named AI crawler families and zero llms.txt fetches^[5]. A twelve-week test on 83 sites logged 7 reads by OpenAI's crawler, 9 by Anthropic's and none by Perplexity's, against thousands of robots.txt fetches by the same crawlers; Google's fetched it 67 times and Meta's 193^[6]. A 30-day census on one domain found the three largest AI crawlers fetched robots.txt 1,150 times and llms.txt 4 times^[7].

Google Search Advocate John Mueller wrote on Bluesky on 17 June 2025 that no AI system currently uses the file, though consumer models do fetch ordinary pages^[8]. In June 2026 he called the question purely speculative, the file having existed for a long time with no AI system using it^[9].

On the published evidence the file is not a route into an AI answer. Server logs from three independent studies put reads by the named crawlers in single or double digits against thousands of robots.txt fetches over the same window^[6], and Google says no AI system uses it^[8]. Publishing one costs little and blocks nothing, but it is not a substitute for the things that are read: robots.txt at the door, and pages an engine can fetch and quote.