AI agents
AI agents navigate a site and act on it for one person, rather than collecting pages for a corpus or an index. Cloudflare's bot taxonomy defines the class as user-directed agents visiting a page on behalf of a human[1]. The products behind the traffic are recent: Anthropic released computer use in October 2024[2], Perplexity shipped the Comet browser in July 2025[3], and OpenAI released ChatGPT Atlas with an optional agent mode in October 2025[4]. An agent reads a page to act on it, so the page is a surface to operate rather than a document to quote.
How a site sees them
An agent arrives looking like a browser, because it usually is one. Computer use works from screenshots and pixel coordinates, moving a cursor and typing the way a person does[2], and Atlas gives its agent mode a visible cursor inside the ordinary browser window[4]. Claude for Chrome runs as an extension taking actions in the user's own browser[5]. That leaves a site with no reliable string to match on, which is why the identification work moved to signatures rather than user agents. See AI crawlers.
Signed agents
Web Bot Auth proposes that bots sign requests cryptographically, using RFC 9421 HTTP Message Signatures, so a site owner can verify who is calling rather than trust a header[6]. Cloudflare put it into production on 28 August 2025 with ChatGPT agent, Goose from Block, Browserbase and Anchor Browser as the first signed cohort[7], and now verifies a bot only where it declares itself through a signature, a published IP list with a stable user agent or reverse DNS, and obeys robots.txt at reasonable rates[8]. The standards work is unfinished: the original architecture draft expired in March 2026 and was replaced by a protocol draft[9].
Early numbers
Public panels do not yet break agents out. A vendor crawler panel splitting AI bot requests by declared purpose for January to July 2026 has rows for training at 46.5%, mixed purpose at 39.3%, search or answer retrieval at 11.1% and user-triggered action at 2.5%, and none for agent activity[10], so agent traffic sits inside the user-triggered slice or inside undeclared browser traffic. The published measurements so far are safety rather than volume: Anthropic reported prompt-injection attacks succeeding 23.6% of the time against browser use without mitigations and 11.2% with them[11], and browser-specific attacks falling from 35.7% to 0%[12], on a pilot opened to 1,000 users[5].
Benchmark success rates
Benchmarks put agent reliability far below a person's. WebArena, a reproducible environment built from fully functional websites in four common domains, reports its best GPT-4-based agent completing 14.41% of tasks end to end[13], against 78.24% for human testers on the same tasks[14]. The gap has narrowed rather than closed: Branch-and-Browse, a tree-structured browsing agent published in October 2025, reaches 35.8% on WebArena[15], under half the human rate. The task sets are drawn from real sites rather than mock-ups, which is what makes the numbers transferable: Mind2Web collects over 2,000 open-ended tasks from 137 websites spanning 31 domains[16].
Bots in this class
The Baseline Labs bot registry (v1) tracks 14 of these. Each one has its own entry with its user agent, its robots.txt token and how to verify it.
- AmazonBuyForMeAmazon
- ChatGPT AgentOpenAI
- Claude-CodeAnthropic
- CursorCursor
- DevinCognition
- Google-AgentGoogle
- Google-Gemini-CLIGoogle
- GoogleAgent-MarinerGoogle
- Manus-UserButterfly Effect
- NovaActAmazon
- opencodeopencode
- OperatorOpenAI
- TraeByteDance
- TwinAgentTwin
The full bot encyclopedia lists every class side by side.
Seen by Baseline
Requests from bots in this class across sites tracked by Baseline, last 30 days. Aggregated over - sites, never reported per site.
References (16)
- Verified bots - Cloudflare docs
- Developing a computer use model - Anthropic
- Comet (web browser) - Wikipedia
- ChatGPT Atlas - Wikipedia
- Piloting Claude for Chrome - Anthropic
- Web Bot Auth: cryptographic bot identity - Cloudflare Blog
- Sign in to your agent: Web Bot Auth for signed agents - Cloudflare Blog
- Verified bots - Cloudflare docs
- draft-meunier-web-bot-auth-architecture - IETF Datatracker
- Crawl-to-refer ratio: AI crawlers and LLM bots - Seomator crawler panel
- Piloting Claude for Chrome - Anthropic
- Piloting Claude for Chrome - Anthropic
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- WebArena: A Realistic Web Environment for Building Autonomous Agents
- Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
- Mind2Web: Towards a Generalist Agent for the Web
Last updated 2026-09-04. Written and maintained by Baseline Labs.