ClaudeBot
The crawler Anthropic uses to collect training data for Claude models. Anthropic documents the token and honours robots.txt, but publishes no IP ranges, so a claimed ClaudeBot hit cannot be verified against an official list - anyone can wear the name.
What it feeds
ClaudeBot gathers web content that may contribute to the training of Anthropic models, which the operator frames as improving their utility and safety.[1] It is separate from the tokens Claude uses to read a page during a conversation or to build search results.
How to identify it
The token appears in logs as ClaudeBot with a version and a contact address. Anthropic publishes the addresses its bots fetch from as JSON at claude.com/crawling/bots.json, which is the only mechanical way to separate a real visit from a client that simply typed the name.
Blocking and its consequences
Anthropic states that its bots honour industry standard robots.txt directives,[2] and supports the non-standard Crawl-delay extension for sites that want to slow rather than stop the crawl.[3] Blocking it is a training opt-out only: Claude can still read a blocked page when a user asks about it, through Claude-User.
Verification
Operator documentation: https://support.claude.com/en/articles/8896518
No IP list published - identity cannot be verified
A user agent is a claim, not a proof: any client can send any name. Where an operator publishes IP ranges, a reverse lookup is the only way to tell a real visit from an impersonation.
Seen by Baseline
Requests from this bot across sites tracked by Baseline, last 30 days. Aggregated over - sites, never reported per site.
References (3)
Last updated 2026-09-05. Written and maintained by Baseline Labs.
