BaselineLabsAIScraper
The background crawler run by Baseline Labs. It fetches a site's pages for Rolling Schema, which keeps that site's schema.org markup in step with its content, and for background research crawls. It works slowly from a queue and backs off when a site shows strain. It does not collect text for model training.
What it does
BaselineLabsAIScraper is the crawler behind Rolling Schema. When a site is connected to Rolling Schema, it re-reads the site's pages on a schedule so the structured data Baseline Labs publishes for them stays accurate. It runs from a queue on our own servers and never in response to a click, which is why it is slow and steady rather than quick.
How fast it goes
- One connection to a site at a time.
- One request every 5 seconds. A robots.txt Crawl-delay can slow it further, never speed it up.
- At most 25 pages in a batch, with at least 5 minutes between batches for the same site.
- A daily cap of 1,000 requests per site, shared with GeorgeBot. The verified owner of a site can change it.
- After 3 errors in a row (a 403, a 429, a timeout or a 5xx) it stops the batch, and a site that keeps failing is left alone for hours.
- It honours Retry-After, waiting up to 5 minutes.
Where it comes from
Requests come only from four static addresses: 94.130.130.190 and 204.168.236.25 (IPv4), 2a01:4f8:13b:3968::2 and 2a01:4f9:c014:c18f::1 (IPv6). A request using this name from anywhere else is not us.
How to block or slow it
It reads robots.txt at the start of every batch. To keep it off a site entirely:
User-agent: BaselineLabsAIScraper
Disallow: /To slow it down instead, give it a Crawl-delay in seconds. Anything below 5 is ignored.
User-agent: BaselineLabsAIScraper
Crawl-delay: 30Blocking it stops Rolling Schema from updating that site's markup.
To let both Baseline Labs identities through a firewall or a host's bot filter, allow user agents containing BaselineLabs.
Verification
Operator documentation: https://baselinelabs.ai/bots/baselinelabsaiscraper
Requests come only from four static addresses: 94.130.130.190 and 204.168.236.25 (IPv4), 2a01:4f8:13b:3968::2 and 2a01:4f9:c014:c18f::1 (IPv6). A request using this name from anywhere else is not us.
A user agent is a claim, not a proof: any client can send any name. Where an operator publishes IP ranges, a reverse lookup is the only way to tell a real visit from an impersonation.
Seen by Baseline
Requests from this bot across sites tracked by Baseline, last 30 days. Aggregated over - sites, never reported per site.
Last updated 2026-09-15. Written and maintained by Baseline Labs.
