AI crawlers

A user-agent string is a claim, not proof. OpenAI publishes a separate machine-readable IP range list for each of its crawlers, so an operator can check where a request actually came from^[2]. The general test is to match the source IP against the operator's published ranges and, where supported, run forward-confirmed reverse DNS: the PTR hostname must end at the operator's own domain and forward-resolve back to the same IP. Googlebot supports both, with 315 published ranges^[3].

Obedience is partial and operator-specific. A study of 130 self-declared bots over 40 days, using controlled robots.txt experiments, found bots comply less as the directive gets stricter, and that whole categories of bot, AI search crawlers among them, rarely check robots.txt at all, so a Disallow line is not a uniform control^[4].

The ratio of pages taken to visitors returned separates the operators by three orders of magnitude. For the 28 days to 21 July 2026, Cloudflare Radar put Mistral at 3,389 pages crawled per referral, Anthropic's ClaudeBot at 2,237, PerplexityBot at 225, GPTBot at 217, Microsoft Copilot at 35 and Google at 4.6^[5]. The figures move fast: the same panel put Anthropic at 56,969 to 1 in January 2026 and 2,363 to 1 in July, as Claude began citing and linking sources^[6]. Volume shifted just as sharply over the preceding year, GPTBot rising from 5% to 30% of AI crawler traffic while ClaudeBot fell 46% and Bytespider fell 85%^[7].

On 1 July 2025 Cloudflare changed the default on its network to block AI crawlers unless they pay for the content, inverting the opt-out norm^[8]. Blocking is not free. A staggered difference-in-differences study of news publishers found that those who blocked GenAI crawlers lost about 23% of their SimilarWeb traffic (ATT -0.262, 95% CI -0.352 to -0.173)^[9]. The pattern is also uneven by publisher type: 60.0% of reputable news sites block at least one AI crawler against 9.1% of misinformation sites, and blocking among reputable sites rose from 23% in September 2023 to nearly 60% by May 2025^[10], which skews what models read. See ChatGPT Search and llms.txt.