AI assistant fetchers

Two operators write the exemption into their own documentation. OpenAI says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply^[3]. Perplexity is blunter: since a user requested the fetch, Perplexity-User generally ignores robots.txt rules^[4]. The reasoning is that the file governs systematic collection, and a single retrieval on a person's behalf resembles that person opening the page. Anthropic reaches the opposite conclusion for the same behaviour, documenting Claude-User as the agent that visits a site when someone asks Claude a question^[6] while stating that its bots honour do-not-crawl directives in robots.txt^[7]. See robots.txt.

The argument became a public fight in August 2025, when Cloudflare reported that Perplexity modified its user agent and rotated source ASNs to keep fetching domains that had disallowed it^[8]. Perplexity's reply rested on the class distinction, arguing that a modern AI assistant answering one question works differently from traditional web crawling^[9]. Publishers read the same fetch as uncompensated access to content, which is what the block-by-default and licensing moves described under AI crawlers respond to.

The class is small but the fastest growing. Cloudflare recorded ChatGPT-User requests up 2,825% between May 2024 and May 2025, to a 1.3% share of crawler requests^[10], and put the growth of AI user-action crawling across 2025 at more than 15 times^[11]. It remains a thin slice of the total: a vendor crawler panel for January to July 2026 attributes 2.5% of AI bot requests to user-triggered action, against 46.5% for training^[12]. Because each hit stands for one question asked, this class is the traffic a site can read as demand rather than harvesting. See AI visibility measurement and AI referral traffic.

This is the smallest class by volume and the most informative one. It is 2.5% of AI bot requests against 46.5% for training^[12], but every hit stands for a question a person actually asked, which makes it the closest thing to a demand signal in a log file. Blocking it is a decision about being answerable rather than about training: Anthropic documents Claude-User as the agent that visits a site when someone asks Claude a question^[6]. The audit framing is that these assistants now act as recommendation engines^[5], so a site invisible to this class is absent from the recommendation, not merely uncrawled.