Search crawlers

Google publishes different robots.txt tokens for the two jobs. GoogleOther is the generic crawler its product teams use to fetch publicly accessible content^[6], while Google-Extended is a standalone token deciding whether crawled content may train future Gemini models^[5]. A site can stay in the index and out of the training corpus. OpenAI documents the same split, running OAI-SearchBot for search and a separate token for model training^[2]. See training scrapers and robots.txt.

Google states that its common crawlers always respect robots.txt rules for automatic crawls^[7], and the protocol assumes exactly that: a crawler reads the file and decides what to follow^[1]. Obedience across the wider population is uneven, and it went unmeasured for most of the protocol's life: the 2025 paper that first tested it at scale, on anonymised institutional web logs, records that the evidence before it was anecdotal^[8]. That study, of 130 self-declared bots over 40 days, found compliance falls as a directive gets stricter, and that AI search crawlers often do not request robots.txt at all^[9], so a Disallow line is a request rather than a control.

Index membership is the precondition for citation. OpenAI states that a site opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link^[2], Anthropic describes Claude-SearchBot as analysing content to improve the relevance of search responses^[3], and Perplexity asks webmasters to allow PerplexityBot so pages can be surfaced and linked^[4]. Blocking the search token of an engine removes the page from the pool its answers are drawn from, which is a different decision from refusing training. See ChatGPT Search, answer engines and AI citation sources.

Search crawling remains the largest share of automated fetching. Cloudflare recorded Googlebot rising from 30% to 50% of crawler requests between May 2024 and May 2025^[10], with GPTBot going from 2.2% to 7.7% on a 305% rise in requests^[11]. Across 2025 Googlebot originated 4.5% of HTML requests on the Cloudflare network against 4.2% for every other AI bot combined^[12]. Retrieval is a minority of declared AI bot purpose: a vendor crawler panel for January to July 2026 put search or answer retrieval at 11.1% of AI bot requests against 46.5% for training^[13]. See Cloudflare and AI crawlers.