CCBot
The Common Crawl Foundation crawler, building the open web archive that seeded the training sets of nearly every large language model - GPT-3 famously included. Blocking CCBot is the single broadest training opt-out on the web, because dozens of labs consume the corpus downstream.
What it feeds
CCBot builds the Common Crawl archive, an open repository of web crawl data maintained by a non-profit foundation whose stated goal is democratising access to web information.[1] Because the corpus is public, dozens of labs consume it downstream, which makes this one crawler the broadest single training opt-out available to a publisher.
How to identify it
The user agent is short and carries a link to the project FAQ.[2] Common Crawl runs the crawler on dedicated address ranges with reverse DNS, apart from IPv6 where reverse DNS is not yet supported,[3] and publishes those ranges as JSON.[4] The foundation warns that crawlers falsely identify themselves as CCBot,[5] so verification matters more here than for most tokens.
Blocking and its consequences
Blocking is a two-line robots.txt rule against the CCBot user agent.[6] It affects future crawls only: archives already published cannot be recalled, and any model already trained on them keeps what it learned.
Verification
Operator documentation: https://commoncrawl.org/ccbot
A user agent is a claim, not a proof: any client can send any name. Where an operator publishes IP ranges, a reverse lookup is the only way to tell a real visit from an impersonation.
Seen by Baseline
Requests from this bot across sites tracked by Baseline, last 30 days. Aggregated over - sites, never reported per site.
References (6)
Last updated 2026-09-05. Written and maintained by Baseline Labs.
