Training scrapers

The larger operators publish a token that controls training alone. Google-Extended decides whether crawled content may train future Gemini models^[6], and Apple is explicit that Applebot-Extended does not itself crawl anything: it is a preference read at training time, and a page that disallows it still appears in search results^[7]. That separation is the point. A site can refuse the training corpus and keep the index entry an answer engine cites from. See robots.txt.

Common Crawl is the largest shared input. CCBot is a Nutch-based crawler built on Apache Hadoop^[4], feeding a corpus of petabytes gathered since 2008^[5]. Common Crawl publishes a robots.txt block that stops the crawler from crawling a site^[8], which acts forward, on crawls not yet taken. The withdrawal figures below are therefore measured against a corpus, C4, rather than against a live crawler^[9].

Refusals have moved fast and unevenly. An audit of 14,000 web domains found robots.txt restrictions added in a single year fully restricted more than 5% of all C4 tokens and more than 28% of its most actively maintained sources^[9], with terms of service covering 45% of the corpus^[10]. Among news sites, 60.0% of reputable outlets disallow at least one AI crawler against 9.1% of misinformation sites^[11], a gap that skews what a model reads. Blocking is not free: publishers who block generative AI bots see reduced site traffic against those who do not^[12]. Compliance is partial in any case, since bots obey less as a directive gets stricter^[13].

Blocking became the default rather than an act. On 1 July 2025 Cloudflare changed the default on its network to block AI crawlers unless they pay creators for their content^[14], and blocking among reputable news sites had already climbed from 23% in September 2023 to nearly 60% by May 2025^[15]. The counterpart is paid access rather than free access, covered under content licensing deals and AI crawlers.