Robots.txt

No registry of user-agent tokens exists, so each operator names and documents its own. OpenAI runs GPTBot for training data and says a Disallow for it signals that a site's content should not be used to train foundation models^[3]. Separate tokens cover search and advertising, while ChatGPT-User fetches a page because a person asked for it, a case where OpenAI warns that robots.txt rules may not apply^[4]. Anthropic divides ClaudeBot, Claude-User and Claude-SearchBot along the same lines and states that its bots honour Disallow and Crawl-delay^[5]. Perplexity documents PerplexityBot as a search-index crawler and says its Perplexity-User token generally ignores robots.txt^[6]. Per-bot detail sits in the bot encyclopedia.

Google added Google-Extended on 28 September 2023 as a control over whether a site's content helps improve Bard and Vertex AI generative APIs^[7]. It now governs training of the Gemini models and grounding in Gemini Apps and Vertex AI, and Google states that blocking it does not affect a site's inclusion in Google Search and is not used as a ranking signal there^[8]. The two controls are separate levers on the same crawl: Googlebot decides whether a page is indexed, Google-Extended decides whether the crawled text feeds Gemini training and grounding. Blocking it therefore leaves Search presence, AI Overviews included, untouched.

Nothing in the protocol enforces anything. A crawler can ignore the rules, spoof its identity or otherwise avoid detection^[9], and Cloudflare reports AI crawlers making more than 50 billion requests a day to its network, just under 1% of all requests it sees, some of them disregarding no-crawl directives^[10]. The first large-scale measurement of compliance, over 130 self-declared bots and 40 days of institutional web logs, found bots are less likely to comply with stricter directives, and that AI search crawlers rarely request the file at all^[11]. Publishers have tightened the file fast. The Consent in Crisis audit of 14,000 domains found that within a year robots.txt came to fully restrict more than 5% of all tokens in the C4 corpus and more than 28% of its most actively maintained sources, with 45% of C4 restricted by terms of service^[12]. See training data versus retrieval for why the two kinds of block have different consequences.

Two efforts extend the file from who may crawl to what the crawl is for. Cloudflare's Content Signals Policy, announced 24 September 2025, adds a content-signal directive carrying yes or no values for three uses: search, generative answers and model training^[13]. Its managed robots.txt already applied the policy to more than 3.8 million domains at launch^[14]. The IETF AI Preferences working group is standardising the same idea as a vocabulary plus a set of attachment methods, robots.txt among them^[15]. Both inherit the weakness of the original: a preference in a file is worth only what the reader chooses to give it. Compare llms.txt, which asks for attention rather than withholding it, and content licensing deals, which replace the request with a contract.