llms.txt vs robots.txt vs Sitemap.xml: What AI Reads

Fetch all three files from your own domain and you get three very different things back. robots.txt returns instructions a crawler either honours or ignores. sitemap.xml returns a list of URLs a crawler may schedule. llms.txt returns a markdown briefing that, as of late 2026, no major AI provider has publicly confirmed reading.

That difference is the whole answer, but it gets lost because all three live at the domain root, all three are plain text, and all three arrived through the same SEO conversation. They sit at different layers of the stack, and only one of them changes what a crawler does today.

Everything below expands on that table, including the parts people get backwards.

robots.txt: The Only File That Actually Changes Behaviour

robots.txt is the one file of the three that changes behaviour, because it is the only one crawlers are built to check. The syntax was standardised in RFC 9309 in 2022, and every major crawler — Google, Bing, OpenAI, Anthropic, Perplexity — reads it before fetching pages.

For AI purposes it works per user agent:

# Block model training, allow search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

OpenAI’s crawler documentation explains why those are separate entries: “Each setting is independent of the others — for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training.”

That separation is the practical value. You can be visible in AI search while opting out of training, or the reverse. Nothing else on this page gives you that control.

The Google-Extended nuance. Google’s crawler documentation describes Google-Extended as a standalone token that controls whether crawled content may be used to train Gemini models and for grounding. It is not a search crawler — Googlebot still crawls you, and AI Overviews draw on the normal Search index. Blocking Google-Extended does not remove your pages from AI Overviews or AI Mode; it only limits Gemini training and grounding use. Google’s AI features documentation states the position plainly: “There are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary.”

What robots.txt cannot do. It cannot remove a page from an index if that page was already crawled, and because it blocks fetching, it also blocks Google from ever seeing a noindex tag on that URL. It is a crawling instruction, not an indexing or legal control. It is also voluntary — a crawler that ignores it faces no technical obstacle, which is why enforcement increasingly happens at the CDN and not in the file.

sitemap.xml: Discovery, Not Permission

A sitemap tells crawlers what exists and when it last changed. The sitemaps.org protocol defines the format, and it is explicit that the scheduling hints are advisory — changefreq is “considered a hint and not a command.” A sitemap grants no access and provides no ranking benefit. It shortens the gap between publishing and being found.

Size limits matter on large sites: 50,000 URLs and 50 MB uncompressed per file, with sitemap index files to group them.

For AI specifically, be careful about the claim you make. Search-oriented AI crawlers exist to answer queries, so they benefit from the same discovery infrastructure as search engines — but no major AI vendor documents reading your sitemap for training. The honest position is:

• Sitemap → search engines: well documented, definitely read.
• Sitemap → AI search crawlers: probable by inheritance, not formally specified.
• Sitemap → training crawlers: not something to rely on. GPTBot does not need your sitemap to find pages that are already linked.

Which means the sitemap’s AI value is second-hand: it gets pages indexed faster, and indexed pages are what AI search features cite. That is worth having, but it is not a lever you pull for AI visibility.

llms.txt: A Proposal With Momentum and No Confirmed Readers

This is the file that needs the most precision, because the marketing around it has outrun the evidence.

What it is. A markdown file at /llms.txt (or any subpath, like /docs/llms.txt) that summarises a site and links to its most useful content in a machine-readable shape. It was proposed by Jeremy Howard on September 3, 2024, and the spec was revised to v2 on August 10, 2026 — still described on its own page as “A proposal to standardise” the practice.

The format is minimal. The H1 is the only required section:

# Example Project

> One-line summary of what this site is and who it serves.

Optional paragraph of context.

## Documentation
- [Getting started](https://example.com/docs/start/): install and first run
- [API reference](https://example.com/docs/api/): endpoints and auth

## Optional
- [Changelog](https://example.com/changelog/): release notes

Links take the form [name](url), optionally followed by : notes. A section headed ## Optional is the convention for content an agent can skip when context is tight.

What it is for. The spec is aimed at agents that fetch content to answer a question — coding assistants reading API docs, chat tools summarising a product page. It is a briefing for something already planning to read you. It is not crawl control, and it is not a sitemap.

What the evidence says. A SE Ranking analysis of 300,000 domains, reported in November 2025, found that roughly one in ten domains published an llms.txt file, and that citation rates did not change based on whether a site had one. Removing the feature from their model actually improved its accuracy. The study’s conclusion was that llms.txt “doesn’t seem to directly impact AI citation frequency. At least not yet.”

The confusion worth naming. OpenAI, Anthropic and Google all publish llms.txt files for their own developer documentation. That is those companies acting as publishers, making their docs easier for agents to consume. It is not evidence that their models read your file. This distinction — publishing the format versus consuming it — accounts for most of the misplaced confidence around llms.txt.

Where the momentum is real. The spec author notes that thousands of sites now publish one, documentation platforms generate it automatically, and Chrome’s Lighthouse audits for it as part of its agentic browsing checks. That is tooling momentum, not proof of consumption. It means the file is becoming a convention; it does not mean an agent has read yours.

What it cannot do. It cannot block a crawler. It cannot enforce a usage policy. It does not replace robots.txt, and it does not make a page rank or get cited. If you want to restrict AI access, the only file on this list that does anything is robots.txt.

Where Each One Fails

The Only Way to Know for Sure: Read Your Logs

Every argument above is about what providers say they read. Your access log shows what actually requested your files, from which user agent, on which date. That is the only direct evidence available to you, and it takes one command.

First, see which AI agents are hitting the site at all:

grep -Ei 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|PerplexityBot|Google-Extended' \
  /var/log/nginx/access.log \
  | awk '{print $1, $7}' | sort | uniq -c | sort -rn | head -20

Then answer the question this article is titled after:

grep -c 'llms\.txt' /var/log/nginx/access.log
grep -c 'robots\.txt' /var/log/nginx/access.log

On most sites the second number is large — every crawler checks robots.txt as a matter of course. The first is usually zero. If an agent is fetching /llms.txt, you will see it in the log with a user agent and a timestamp, and that is worth more than any vendor blog post.

Two caveats when reading the output. Hits from a CDN may not appear in origin logs at all, so a zero can mean “not requested” or “handled upstream” — check the CDN’s own log export. And a single fetch is not adoption: one curious crawler requesting the file once does not mean the practice works, only that the file was found.

Run this monthly. If llms.txt requests start appearing across providers, that is the signal the spec has been adopted, and it will show up in your logs before it shows up in an announcement.

Where Each File Lives in WordPress

robots.txt. WordPress serves a virtual robots.txt — there is usually no physical file to edit. SEO plugins hook into it and let you append rules; a physical file in the web root overrides the virtual one, which is a common reason plugin changes “stop working.” Note that Settings → Reading → “Discourage search engines” is not a robots.txt setting in the usual sense: it writes a blanket Disallow and a sitewide noindex, and it is the most common accidental cause of a site vanishing. If you are choosing a plugin to manage this, the free WordPress SEO plugins comparison covers which ones expose robots.txt editing and sitemap control.

sitemap.xml. Core generates /wp-sitemap.xml since WordPress 5.5. SEO plugins replace it with their own index. Running both is fine as long as you submit the right one in Search Console — submitting a stale leftover sitemap is the usual mistake.

llms.txt. WordPress has no core support. You need either a real file at the domain root (uploaded, not a page) or a route that serves it with the correct content type. Creating a WordPress page with the slug llms.txt will not work: it resolves to /llms-txt/ or /llms-txt-2/, and even at the right path it returns HTML. If you want it generated rather than uploaded, an SEO plugin with an AI/llms module is the simplest route — our DefiniteSEO setup guide shows the plugin configuration patterns these modules follow.

Verify All Three in One Minute

for f in robots.txt sitemap.xml llms.txt; do
  printf "%-12s %s  %s\n" "$f" \
    "$(curl -s -o /dev/null -w '%{http_code}' https://example.com/$f)" \
    "$(curl -sI https://example.com/$f | grep -i '^content-type' | tr -d '\r')"
done

Read the output like this:

• All three 200 — files exist. Content type tells you whether llms.txt is being served as HTML by mistake.
• llms.txt 404 — normal on most WordPress sites; nothing generates it by default.
• sitemap.xml 404 — check /wp-sitemap.xml, the core location.
• robots.txt 200 but rules missing — a physical file is overriding the virtual one.

What to Do This Week

Work in this order, because the evidence is strongest at the top:

1. Confirm the AI crawlers you care about are allowed. curl your robots.txt and check for a blanket Disallow or a CDN rule that blocks bot traffic.
2. Confirm pages are indexed. An AI feature can only cite a page in the index. If pages are missing, fix that before touching any of these files.
3. Make the content extractable. Clear headings, direct answers, structured data. This is the part that moves citations, and it is covered in semantic SEO and entity structure and in AI Overviews and WordPress.
4. Keep the sitemap accurate. Exclude noindex pages, don’t fake lastmod, submit the right URL in Search Console.
5. Add llms.txt last, if at all. Twenty minutes of work, real but unproven upside, zero downside beyond the time. Do it because it costs little, not because it works.

If a service is selling llms.txt as the key to AI visibility, that is worth scepticism — the AI SEO services comparison covers what legitimate providers actually do. And if your URL structure is inconsistent, no file fixes it: structure your WordPress pages first.

FAQ

Does ChatGPT read my llms.txt?

No provider has publicly confirmed reading third-party llms.txt files at inference time, and a 300,000-domain analysis found no relationship between having the file and being cited. OpenAI publishes an llms.txt for its own documentation, but that is publishing, not consuming.

Should I block GPTBot to protect my content?

That is a business decision, not an SEO one. Blocking GPTBot opts out of training use while OAI-SearchBot can stay allowed for search visibility — OpenAI documents the two as independent settings. Blocking both removes you from both uses.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended governs Gemini training and grounding, not Search crawling. AI Overviews and AI Mode draw on the normal Search index, and Google states there are no additional requirements to appear in them.

Is a sitemap still worth having for AI search?

Yes, indirectly. It speeds up discovery and indexing, and AI search features cite indexed pages. It is not a direct AI signal, and no vendor documents reading sitemaps for training.

Which file should I create first?

robots.txt is already there and is the only one with enforcement effects — audit it before adding anything. A sitemap should already exist in WordPress. llms.txt is optional and last.

Can llms.txt replace robots.txt?

No. llms.txt has no enforcement mechanism at all. If you need crawlers to stop, only robots.txt communicates that, and even then compliance is voluntary.

My llms.txt returns an HTML page. Why?

You probably created a WordPress page with that slug. Pages render HTML and live at a different path. Serve a real text file at the domain root, or add a route that returns text/plain, and confirm with the curl content-type check above.