Skip to content

LLM Visibility Tools

Generate the three files that tell AI systems how to read your site

Point the generator at a domain and it builds an llms.txt content index, a robots.txt block with per-crawler AI directives, and the meta tags that opt pages out of training datasets. Copy, download, done.

Generate yours free

Free with an account. No credit card, no credits spent.

What it generates

llms.txt

A markdown index of your site's pages with titles and descriptions, in the format set out by the llmstxt.org spec - plus an llms-full.txt variant carrying the page text itself.

robots.txt directives

Rules for 14 named AI crawlers. Allow everything, block everything, or set each bot's access individually with its own path.

Meta tags

The noai and noimageai head snippets, for opting individual pages out of AI training datasets rather than the whole domain.

All three at once

One run produces the full set. Copy any panel to the clipboard, download a single file, or take the lot as a bundle.

How it works

1

Enter a domain

Just the hostname. Nothing has to be installed, and the site does not need to be yours to inspect what a generated file would look like.

2

We find your pages

The generator reads your sitemap.xml (following a sitemap index, or whatever path your robots.txt points at) and takes up to 40 URLs from it.

3

We read each page's own words

Every discovered page is fetched for its <title> and meta description. No model writes the descriptions - the file says what your pages already say about themselves.

4

Review, then publish

Edit anything you want to change, then upload llms.txt to the root of your domain and paste the robots rules and meta tags where they belong.

What llms.txt actually is, and what it is not

llms.txt is a plain markdown file at the root of a domain that lists your important pages with a line of description each. The idea, proposed at llmstxt.org, is the same one behind robots.txt and sitemap.xml: give a machine a short, curated map instead of making it infer your site's shape from navigation and links.

Being straight about the state of adoption: it is fetched today mainly by coding agents and MCP clients such as Cursor and Claude Code, rather than by the big answer engines. No major search or answer engine has confirmed it as a ranking or retrieval input. We wrote up the evidence in our review of whether llms.txt affects Google Search, and the honest summary is that it is a two-minute hedge, not a growth lever.

The other two files are different: they do something today. AI crawler directives in robots.txt are honoured by the major operators and are the actual mechanism for deciding who may read your site and for what. The noai meta tags are the page-level version of the same choice.

If you want the part of this that moves visibility, it is not any of these three files - it is whether your pages are readable and well-described in the first place. Site Pulse grades that, and Schema Audit checks the structured data engines lean on to understand a page.

Frequently asked questions

Where does the llms.txt file go?
At the root of your domain, so it resolves at https://yourdomain.com/llms.txt - the same place robots.txt lives. The fuller variant goes alongside it as llms-full.txt.
What is the difference between the concise and full versions?
The concise file is an index: page titles, URLs and a description line each. The full file keeps the same structure but carries the page text as well, so a client that reads it has your content without fetching every page. Concise is the one most people publish; full is useful when you are feeding a specific agent.
How many pages does it include?
Up to 40, taken from your sitemap. That is a curation limit rather than a technical one - an index of every URL on a large site stops being a map and becomes another crawl. Edit the file down further if the top 40 is still more than you want to point at.
What if my site has no sitemap?
The generator checks the standard sitemap paths and any location declared in your robots.txt. If none of those resolve, there is nothing to index from - publishing a sitemap first is worth doing on its own merits, and then this run will work.
Which AI crawlers can I control?
Fourteen, including GPTBot, ChatGPT-User and OAI-SearchBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended, PerplexityBot, CCBot, Bytespider, cohere-ai, FacebookBot, Applebot-Extended, Amazonbot and Diffbot. Custom mode gives each one its own path, so you can block a bot from /pricing while leaving your blog open.
Should I block AI crawlers?
It depends what you want. Blocking training crawlers keeps your content out of future model weights; blocking retrieval crawlers also keeps you out of the answers those engines give about your category, which for most businesses is the thing they are trying to get into. The split matters - the two are separate bots for most operators, which is exactly why the generator lists them individually instead of offering one switch.
Does any of this cost credits?
No. Generation is free with an account - it reads public pages and writes text files, with no model calls and no paid data behind it.
Will publishing llms.txt improve my rankings?
There is no evidence that it will, and we would rather say so. Treat it as cheap insurance against future adoption. The work that measurably changes how AI engines treat a site is content and structure - see Site Pulse and the Visibility Scanner for that side of it.

Build your three files in about a minute

Generate yours free

LLM Visibility Tools

Generate the three files that define how AI systems see your site - a content map, crawler permissions, and per-page directives.

Discovering pages…

llms.txt

A structured markdown index of your site's pages. Today it's fetched mainly by coding agents and MCP clients (Cursor, Claude Code) rather than answer engines - and it's a two-minute hedge for when they do adopt it. Read our evidence review. Also generates llms-full.txt with complete page text.

robots.txt

Per-crawler directives for 14 major AI bots. Allow all, block all, or pick exactly who gets in - with per-bot path control in Custom mode.

Meta tags

HTML <head> snippets using noai and noimageai to opt pages out of AI training datasets.

George
Online
0%