The web just went majority-machine — can AI even read your site?

Bots passed humans online — 18 months early

On 3 June 2026, Cloudflare CEO Matthew Prince posted data from Cloudflare Radar showing that bots had surpassed human-generated HTML traffic for the first time. The split: 57.5% bots, 42.5% humans. Prince had forecast this would happen at SXSW — his estimate was roughly 18 months out. "Agentic traffic growing so fast that bots have now passed human traffic online for the first time in the Internet's history," he wrote. He added he remains "stunned by the rate of growth" of nonhuman traffic. Independently, NBC News reported a marginally different read of 57.4% bots / 42.6% humans from the same Cloudflare data, and Prince acknowledged the figures are "a bit messy… but clearly on the other side now." The Imperva Bad Bot Report 2026 (published April 29, 2026) separately measured 53% automated traffic across all web requests in 2025 — a different basket that includes API and app calls, so the two figures shouldn't be averaged, but they point the same direction.

Most of those bots are not AI — and most AI bots don't send traffic back

The 57.5% is a total bot figure, not an AI-only figure. Cloudflare's own breakdown (May 2026) puts AI crawlers at roughly 20.3% of verified bot traffic, with AI-search bots adding around 6.5% on top. The rest is legacy search indexers, uptime monitors, feed fetchers and security scanners — bots that have been around for decades. And even within the AI slice, 51.8% of AI-crawler requests are for model training — not for surfacing answers in search. Only 9.3% of AI-crawler requests are for search purposes. Training crawlers return effectively zero referral traffic: Cloudflare's data shows ClaudeBot hitting pages at a crawl-to-refer ratio of roughly 70,900:1 (June 2025) and GPTBot at 1,600:1. Google's combined crawler sits at a comparatively modest 9–10:1. The machine majority is real; its composition matters enormously for what you should actually do about it.

Source: Cloudflare Radar, May–June 2026; crawl-to-refer ratios from Cloudflare blog data, June 2025.

69% of AI crawlers can't run JavaScript — and they're not pretending otherwise

Vercel and MERJ analysed more than 500 million GPTBot fetches and found zero evidence of JavaScript execution. GPTBot downloads JS files roughly 11.5% of the time; ClaudeBot does so around 23.8% of the time — but neither actually runs them. A React, Vue or Angular single-page app serves these crawlers a blank <div id="root"></div> and nothing else. The one meaningful exception is Gemini: because it rides Google's crawler infrastructure, it does render JavaScript. Google's own guidance on AI features confirms rendering capability as a prerequisite for citation, making Googlebot-family crawlers the only AI path that reliably sees client-rendered content. That leaves 69% of AI crawlers locked out of the majority of modern web apps.

Source: Vercel/MERJ — The Rise of the AI Crawler.

What this data does — and doesn't — tell you

Three things worth holding alongside the headline numbers:

Requests ≠ engagement. The 57.5% figure counts HTTP requests, not attention, time on page, or money spent. By any engagement or revenue metric, humans still dominate the web by a wide margin. Prince flagged this himself, noting the data covers HTML traffic and calling it "a bit messy." It is a leading indicator of direction, not a claim that AI agents are outspending human shoppers.

Single-vendor view. Cloudflare routes roughly 20% of web traffic. Its sample is large and diverse, but it is not the whole internet — high-traffic properties that bypass Cloudflare are invisible to this dataset.

Crawlability is necessary, not sufficient. Getting your HTML in front of an AI crawler is the floor, not the ceiling. Being readable doesn't guarantee citation. Model training data has a cutoff; search-mode retrieval is selective; and the crawl-to-refer ratios above show that most AI crawlers are not in the business of sending traffic back. Fixing your rendering is a prerequisite — it is not a citation strategy on its own.

Server-side rendering is table stakes for AI visibility

The practical implication is direct: if your marketing site, landing pages, or product pages are pure client-side React or Vue, the crawlers that matter — GPTBot, ClaudeBot, PerplexityBot — are reading a blank page. Static site generation (SSG) or server-side rendering (SSR) are no longer a performance optimisation; they are the minimum bar for being legible to the machines now majority on the web. Beyond rendering, the Vercel/MERJ study found that crawlers followed structured signals (sitemaps, canonical tags, clean HTML) more reliably than JavaScript-heavy navigation. Google's AI features documentation echoes this: structured, crawlable markup is the prerequisite for appearing in AI-generated answers.

See how AI reads your site