GPTBot

GPTBot collects web content for the training corpora behind OpenAI foundation models, and OpenAI describes the purpose as making those models more useful and safe.^[1] It is a training crawler rather than a retrieval agent: a page it fetches is not thereby made citable in a ChatGPT answer, which is the job of OAI-SearchBot and ChatGPT-User.

The token appears in logs inside a browser-shaped string ending in the crawler name, version and a documentation URL.^[2] The version number moves, so match on the token rather than the whole string. OpenAI publishes the ranges GPTBot fetches from as JSON at openai.com/gptbot.json, and an address outside that file is an impersonation whatever the user agent says.

OpenAI treats a disallow as an explicit signal that the content should not be used in training.^[3] Blocking it costs nothing in live AI visibility, because ChatGPT search retrieval runs on separate tokens, and gains nothing in the short term either: models already trained on a page do not unlearn it.

Operator documentation: https://platform.openai.com/docs/bots

Published IP list: https://openai.com/gptbot.json

A user agent is a claim, not a proof: any client can send any name. Where an operator publishes IP ranges, a reverse lookup is the only way to tell a real visit from an impersonation.