IndexNow and sitemaps

A single sitemap file is capped at 50,000 URLs or 50MB uncompressed, and a larger site splits across a sitemap index^[3]. Google reads the lastmod element only when the value is consistently and verifiably accurate, checked against the page's real modification^[4]. The priority and changefreq elements are ignored outright^[5]. Google accepts a sitemap through Search Console, the Search Console API, a line in robots.txt, or WebSub^[6]. Microsoft says Bing revisits a known sitemap at least once a day^[7] and uses lastmod to decide which URLs to recrawl first^[8].

Microsoft Bing and Yandex published IndexNow on 18 October 2021^[9]. A site hosts a key file, then posts up to 10,000 URLs per request to any participating endpoint^[10]. Engines that adopt the protocol agree to pass submitted URLs to each other, so one submission reaches the whole set^[11]. Microsoft reported in August 2022 that 7% of newly clicked Bing result URLs had come through IndexNow^[12], and in March 2024 put the figure at 17% against 2.5 billion submitted URLs^[13].

A Google spokesperson said in November 2021 that the company would be testing the potential benefits of the protocol^[14]. No adoption followed: Google is absent from the IndexNow endpoint list^[1], and its recrawl documentation still offers the URL Inspection tool for a handful of pages and a sitemap for the rest, with crawling taking anywhere from a few days to a few weeks^[2].

Neither mechanism is specific to AI. AI crawlers fetch the same files as search crawlers, and Cloudflare measured Googlebot at 50% of crawler requests in May 2025, ahead of Bingbot at 8.7%, GPTBot at 7.7% and ClaudeBot at 5.4%^[15]. Vercel recorded 569 million GPTBot requests and 370 million from Claude across its network in a month, with neither crawler executing the JavaScript it fetches^[16]. An IndexNow submission reaches Bing and the products built on its index, but not crawlers outside the protocol, which find pages through links and sitemaps^[1]. Both Google and Bing read the sitemap lastmod value, and both condition it on the date being accurate^[4][8], which ties discovery to content freshness.