Cloudflare

On 1 July 2025 Cloudflare made blocking AI crawlers the default for every domain that signs up, so an owner who wants to be crawled has to say so rather than opt out later^[2]. Cloudflare puts the size of that network at 20% of the web^[1]. More than a million customers had already taken the one-click block introduced in September 2024^[4]. A robots.txt rule is therefore not the only gate an AI crawler has to pass.

Pay Per Crawl entered private beta on the same day, letting a site charge per request using the HTTP 402 Payment Required status: a crawler either presents payment intent in its request headers and gets a 200, or receives a 402 quoting the price^[5]. Each crawler is set to one of three states, Allow for free access, Charge at a flat domain-wide price, or Block with no payment option^[6].

A bot qualifies as verified on two tests: it declares itself deterministically, through a cryptographic Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS; and it behaves, obeying robots.txt and crawl directives, holding reasonable request rates, and not evading owner preferences or attacking sites^[7]. On 4 August 2025 Cloudflare published evidence that Perplexity was using undeclared crawlers that impersonated Chrome on macOS and rotated across networks outside its official range, in order to keep reading domains that had blocked PerplexityBot^[8]. Cloudflare de-listed Perplexity as a verified bot and added managed rules against the pattern^[9].

In its 2025 Radar year in review, Cloudflare put AI bots other than Googlebot at 4.2% of HTML request traffic across the network, with Googlebot itself at 4.5%^[10]. Radar figures for May 2026 put AI crawlers at 20.3% of verified bot traffic, with AI search bots adding a further 6.5%^[11]. On 3 June 2026 chief executive Matthew Prince put automated systems at 57.5% of HTTP requests to web content worldwide, against 42.5% from people^[3].