Skip to content

AI agents

Concept in AI search and SEO
Volatile, last checked 2026-09-04 This page carries measured figures that move quickly.Reviewed 2026-09-04

AI agents navigate a site and act on it for one person, rather than collecting pages for a corpus or an index. Cloudflare's bot taxonomy defines the class as user-directed agents visiting a page on behalf of a human[1]. The products behind the traffic are recent: Anthropic released computer use in October 2024[2], Perplexity shipped the Comet browser in July 2025[3], and OpenAI released ChatGPT Atlas with an optional agent mode in October 2025[4]. An agent reads a page to act on it, so the page is a surface to operate rather than a document to quote.

How a site sees them

An agent arrives looking like a browser, because it usually is one. Computer use works from screenshots and pixel coordinates, moving a cursor and typing the way a person does[2], and Atlas gives its agent mode a visible cursor inside the ordinary browser window[4]. Claude for Chrome runs as an extension taking actions in the user's own browser[5]. That leaves a site with no reliable string to match on, which is why the identification work moved to signatures rather than user agents. See AI crawlers.

Signed agents

Web Bot Auth proposes that bots sign requests cryptographically, using RFC 9421 HTTP Message Signatures, so a site owner can verify who is calling rather than trust a header[6]. Cloudflare put it into production on 28 August 2025 with ChatGPT agent, Goose from Block, Browserbase and Anchor Browser as the first signed cohort[7], and now verifies a bot only where it declares itself through a signature, a published IP list with a stable user agent or reverse DNS, and obeys robots.txt at reasonable rates[8]. The standards work is unfinished: the original architecture draft expired in March 2026 and was replaced by a protocol draft[9].

Early numbers

Public panels do not yet break agents out. A vendor crawler panel splitting AI bot requests by declared purpose for January to July 2026 has rows for training at 46.5%, mixed purpose at 39.3%, search or answer retrieval at 11.1% and user-triggered action at 2.5%, and none for agent activity[10], so agent traffic sits inside the user-triggered slice or inside undeclared browser traffic. The published measurements so far are safety rather than volume: Anthropic reported prompt-injection attacks succeeding 23.6% of the time against browser use without mitigations and 11.2% with them[11], and browser-specific attacks falling from 35.7% to 0%[12], on a pilot opened to 1,000 users[5].

Benchmark success rates

Benchmarks put agent reliability far below a person's. WebArena, a reproducible environment built from fully functional websites in four common domains, reports its best GPT-4-based agent completing 14.41% of tasks end to end[13], against 78.24% for human testers on the same tasks[14]. The gap has narrowed rather than closed: Branch-and-Browse, a tree-structured browsing agent published in October 2025, reaches 35.8% on WebArena[15], under half the human rate. The task sets are drawn from real sites rather than mock-ups, which is what makes the numbers transferable: Mind2Web collects over 2,000 open-ended tasks from 137 websites spanning 31 domains[16].

Bots in this class

The Baseline Labs bot registry (v1) tracks 14 of these. Each one has its own entry with its user agent, its robots.txt token and how to verify it.

The full bot encyclopedia lists every class side by side.

Seen by Baseline

Requests, 30 days
...
Last 7 days
...
Bots seen
...

Requests from bots in this class across sites tracked by Baseline, last 30 days. Aggregated over - sites, never reported per site.

References (16)
  1. Verified bots - Cloudflare docs
    Documentation Retrieved 2026-09-04
  2. Developing a computer use model - Anthropic
    Vendor documentation Published 2024-10-22 Retrieved 2026-09-04
  3. Comet (web browser) - Wikipedia
    Journalism Published 2025-07-09 Retrieved 2026-09-04
  4. ChatGPT Atlas - Wikipedia
    Journalism Published 2025-10-21 Retrieved 2026-09-04
  5. Piloting Claude for Chrome - Anthropic
    Vendor documentation Published 2025-08-25 Retrieved 2026-09-04
  6. Web Bot Auth: cryptographic bot identity - Cloudflare Blog
    Vendor documentation Published 2025-05-15 Retrieved 2026-09-04
  7. Sign in to your agent: Web Bot Auth for signed agents - Cloudflare Blog
    Vendor documentation Published 2025-08-28 Retrieved 2026-09-04
  8. Verified bots - Cloudflare docs
    Documentation Retrieved 2026-09-04
  9. draft-meunier-web-bot-auth-architecture - IETF Datatracker
    Documentation Published 2026-03-02 Retrieved 2026-09-04
  10. Crawl-to-refer ratio: AI crawlers and LLM bots - Seomator crawler panel
    Vendor documentation Published 2026-07-21 Retrieved 2026-09-04 Single-source Volatile, last checked 2026-09-04
  11. Piloting Claude for Chrome - Anthropic
    Vendor documentation Published 2025-08-25 Retrieved 2026-09-04
  12. Piloting Claude for Chrome - Anthropic
    Vendor documentation Published 2025-08-25 Retrieved 2026-09-04
  13. WebArena: A Realistic Web Environment for Building Autonomous Agents
    Academic Published 2023-07-25 Retrieved 2026-09-04
  14. WebArena: A Realistic Web Environment for Building Autonomous Agents
    Academic Published 2023-07-25 Retrieved 2026-09-04
  15. Branch-and-Browse: Efficient and Controllable Web Exploration with Tree-Structured Reasoning and Action Memory
    Academic Published 2025-10-18 Retrieved 2026-09-04 Single-source
  16. Mind2Web: Towards a Generalist Agent for the Web
    Academic Published 2023-06-09 Retrieved 2026-09-04

Last updated 2026-09-04. Written and maintained by Baseline Labs.

George
Online
0%