llms.txt: What It Is, How to Write One and How to Validate It

In short: llms.txt is a proposed Markdown file at /llms.txt that gives language models a short map of your site: a # Site name title (the only required part), an optional one-line summary in a > blockquote, and ## sections listing your most useful pages as Markdown links. It does not control crawling, it is not a Google ranking signal, and the major AI crawlers do not document reading it. It is still cheap to publish and useful to tools and agents pointed at your site. Write it in ten minutes, serve it as plain text with status 200, and validate it.

Few web files have attracted as much attention for as little documented effect. Plugins add llms.txt with one click, guides promise it will "get you into ChatGPT", and many site owners publish one without knowing what it is supposed to contain. This guide sticks to what the proposal defines, what the AI providers have and have not said, and how to write a file that is correct and actually useful. It also explains exactly what our free llms.txt checker validates, and how to fix each problem it reports.

What llms.txt is

The llms.txt proposal was published by Jeremy Howard on September 3, 2024. It describes itself as "a proposal to standardise on using an /llms.txt file to provide information to help agents use a website," and the page notes it is now in its second version and open for community input.

The idea: HTML pages are full of navigation, scripts and layout that waste a model's limited context. A short, human-written Markdown file can tell a model what the site is and where the useful content lives. The proposal says llms.txt "would mainly be useful for inference rather than training," in other words when a model or agent is answering a question and reads your site live.

The format, section by section

The proposal defines the sections in a fixed order:

Each entry in a file list is a Markdown list item with a required link [name](url), optionally followed by a colon and a note about the page. The proposal also suggests offering a Markdown version of individual pages at the same URL with .md appended, for example page.html.md, or with the extension replaced (page.md).

The file usually lives at the site root, but the proposal allows it at any path, covering the pages under that path, which suits documentation that lives under /docs/. The Next.js documentation, for example, points readers to an index at /docs/llms.txt.

What llms.txt is not

• Not access control. robots.txt, defined in RFC 9309, tells crawlers what they may fetch. llms.txt permits and forbids nothing. The proposal itself says the two "have different purposes" and that llms.txt can complement robots.txt.
• Not a sitemap. A sitemap lists every indexable URL for search engines. llms.txt lists a curated handful, with context. The proposal says a sitemap "isn't a substitute" for llms.txt.
• Not a Google requirement. Google's guidance on AI features says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features."
• Not documented by the big AI crawlers. The crawler documentation of OpenAI, Anthropic and Perplexity explains robots.txt tokens and IP ranges; as of October 2026, none of these pages mentions llms.txt.

Why publish one anyway

Three reasons hold up without exaggeration:

1. Agents and coding assistants read it when pointed at a site. If a developer pastes your docs URL into an assistant, a clean index of Markdown links is easier to work with than your rendered navigation. This is why documentation platforms adopted it first; llmstxt.org lists generators in Mintlify, GitBook, Yoast SEO, AIOSEO, Wix and others.
2. It forces a useful editorial exercise. Writing one sentence about what you do and picking the ten pages that matter is the same work that makes a home page clear to people and to the search crawlers that do feed AI answers.
3. It costs almost nothing. A static file, no ongoing maintenance beyond keeping links current.

What it will not do is fix a site that AI crawlers cannot reach. If OAI-SearchBot gets a 403 from your firewall, no text file changes that. Check access first with the AI crawler checker, and read our AI crawlers guide for the details.

Copy-ready llms.txt examples

A software product

# Tallyboard

> Tallyboard is time tracking and invoicing software for agencies of 5 to 200 people. Plans start at $12 per user per month; a 14-day trial needs no card.

Tallyboard runs in the browser and on iOS and Android, and integrates with QuickBooks, Xero and Slack.

## Product
- [Features](https://www.example.com/features): timers, timesheets, approvals, invoicing
- [Pricing](https://www.example.com/pricing): plans, limits and annual discount
- [Integrations](https://www.example.com/integrations): QuickBooks, Xero, Slack, Jira

## Help
- [Getting started](https://www.example.com/help/getting-started): set up a workspace in 10 minutes
- [API reference](https://www.example.com/docs/api.md): REST API, authentication and rate limits

## Optional
- [Changelog](https://www.example.com/changelog)
- [Company](https://www.example.com/about)

A local business

# North Street Bakery

> Family bakery in Springfield, Illinois, baking sourdough, pastries and custom cakes since 2009. Open Monday to Saturday.

## Visit
- [Location and hours](https://www.example.com/visit): 120 North Street, opening hours, parking
- [Contact](https://www.example.com/contact): phone, email and order form

## Order
- [Custom cakes](https://www.example.com/cakes): flavors, sizes, prices and lead time
- [Catering](https://www.example.com/catering): office breakfast boxes

A documentation site

# Lumen SDK

> Lumen is an open-source JavaScript SDK for building offline-first apps. MIT license.

## Docs
- [Quick start](https://docs.example.com/quick-start.md): install and first sync in five minutes
- [Configuration](https://docs.example.com/config.md): every option with defaults
- [Conflict resolution](https://docs.example.com/conflicts.md): how merges work

## Examples
- [Todo app](https://docs.example.com/examples/todo.md)

## Optional
- [Architecture notes](https://docs.example.com/internals.md)

Rules that make the file useful: write the summary as facts (what, for whom, where, price), use absolute URLs of canonical pages that return 200, keep notes short, and link to Markdown versions where you have them.

How the Searvia llms.txt checker works

The llms.txt checker runs the same engine as our website SEO checker. You enter a domain or a page address; it reads that page (your home page if you enter only a domain) and the site files around it, runs 38 checks in about 20 seconds and puts the three checks this tool is about at the top. Here is exactly what each one does.

llms.txt file (low priority). The checker follows the page to its final address, then requests /llms.txt on that host, following redirects. The file counts as present only if:

• it answers with status 200,
• it is not an HTML page (a text/html content type, or a body starting with <!doctype or <html>, means your server returned a web page, usually a "not found" page or the home page), and
• it is not empty.

The check passes when the file, after an optional byte order mark (BOM) and leading whitespace, starts with # , which is the H1 the proposal requires. The report also counts the Markdown links in the form [text](url). It warns, rather than fails, when the file is missing or has no title, because llms.txt is optional.

The checker does not validate the blockquote, section order, or whether linked pages exist, and it only looks at the root of the host, not /docs/llms.txt. It does not look for llms-full.txt.

AI crawler access (high priority). The same report shows whether robots.txt lets 10 AI and search crawlers and 2 AI training opt-out tokens fetch the page, and whether your firewall turns away a request that identifies as GPTBot (a warning, not proof, since the request does not come from OpenAI's addresses). It is in this tool's focus because an llms.txt that crawlers cannot reach is pointless.

XML sitemap (medium priority). The checker tries the Sitemap: lines in robots.txt, then /sitemap.xml and /sitemap_index.xml, up to four addresses. It passes on the first one that answers 200 with a <urlset> or <sitemapindex> root and reports how many <loc> entries it found. The sitemap and llms.txt do different jobs, and you want both.

The overall A+ to F grade covers all 38 checks, weighted by priority, and the report can be shared with a link.

Reading the result

A pass with 0 links means the file has a title but no Markdown links, so it gives tools nothing to follow. A pass with 300 links means you pasted a sitemap; trim it to the pages that matter.

How to fix every llms.txt problem

1. No llms.txt file was found

1. Write the file using one of the examples above. Save it as UTF-8 plain text named exactly llms.txt, lowercase.
2. Put it where your server serves the site root:
   • Static sites and most frameworks: the public/ or static/ folder (Next.js, Vite, Astro, Nuxt, Hugo all serve that folder at the root).
   • WordPress: Yoast SEO and AIOSEO can generate the file; otherwise upload it to the WordPress root folder next to wp-config.php.
   • Shopify: you cannot upload files to the root, so use an app, or upload the file to Files and create a URL redirect from /llms.txt to it. The checker follows redirects, so this works as long as the target is served as text.
   • Wix: llmstxt.org lists Wix among platforms that support the file; check your site's SEO settings.
3. Open https://yourdomain.com/llms.txt in a private window and confirm you see raw Markdown, not your website.
4. Run the llms.txt checker again.

2. llms.txt exists but does not start with a "# " title

The first non-blank characters must be # followed by a space. Common causes:

• The file starts with the summary or a blank list; add # Your site name as the first line.
• The heading is written #Name without a space, or as Name with === underneath. Use # Name.
• An invisible byte order mark (BOM) at the start of the file. Our checker ignores it, but some other parsers then do not see # as the first characters. Save the file as "UTF-8 without BOM" in your editor to be safe.
• A plugin added a comment or a generated-by line above the title. Move it below, or remove it.

3. The address returns a web page instead of the file

If you uploaded a file but the checker still says it was not found, request the headers:

curl -I https://yourdomain.com/llms.txt

• Status 200 with Content-Type: text/html: your app's catch-all route answers every unknown URL with the home page or a "not found" page with status 200. Serve the file before the app routes, or configure the static file handler.
• Status 404: the file is not where the server looks. Check the folder and the file name.
• Status 301 or 302 to the home page: a redirect rule catches unknown paths. Add an exception.
• Status 403: a security rule blocks .txt files or unknown paths. Allow /llms.txt.

Then set the content type to plain text. In Nginx:

location = /llms.txt {
    default_type text/plain;
    charset utf-8;
}

In Apache, AddType text/plain .txt in the site configuration or .htaccess does the same. text/markdown is also fine; anything except HTML passes.

4. AI crawler access fails or the sitemap is missing

These come with the llms.txt report because they matter more. If AI search crawlers are blocked in robots.txt or by the firewall, fix that first; our AI crawlers guide has the steps. If no sitemap was found, publish one of your indexable pages, list it in robots.txt with a line such as Sitemap: https://www.example.com/sitemap.xml, and submit it in Google Search Console and Bing Webmaster Tools.

Keep it current

An llms.txt with dead links is worse than none, because it sends tools to error pages. Three habits keep it accurate:

• Generate it from your data when you can. A build step that writes the file from your navigation or docs index never drifts.
• Review it when pricing, plans or the product line change. The summary line is the part most likely to go stale.
• Link only canonical pages. No redirected URLs, no tracking parameters, no noindex pages.

Worked example (hypothetical)

The following is a hypothetical example, not real data. Fernway Travel, an invented tour operator, installs a plugin that generates llms.txt. The checker says "No llms.txt file was found." curl -I shows status 200 and Content-Type: text/html: the site's caching layer served a cached home page for the new address. After purging the cache, the file appears, but the checker now says it "exists but does not start with a "# " title": the plugin writes a comment line first. The team switches the plugin to manual mode and writes the file by hand:

1. # Fernway Travel, then a summary: small-group walking tours in Portugal and Spain, 6 to 12 guests, April to October.
2. ## Tours with eight tour pages and their length and price range as notes.
3. ## Plan your trip with booking terms, what's included and the FAQ.
4. ## Optional with the blog and the about page.

The checker passes with 14 links. The team also finds, in the same report, that its sitemap was missing from robots.txt and adds the Sitemap: line.

A 10-minute llms.txt checklist

1. Check AI crawler access first; llms.txt cannot help if crawlers are blocked.
2. Write # Name, a one-line factual summary, and two or three ## sections of key links.
3. Use absolute, canonical URLs that return 200.
4. Upload to the root, serve as plain text, save without BOM.
5. Validate with the llms.txt checker.
6. Put a reminder to review it when your product or pricing changes.

How Searvia helps

The free llms.txt checker validates the file together with AI crawler access and your sitemap in one shareable report. The AI search readiness checker runs the same engine with the focus on everything assistants need: crawler access, content without JavaScript, identity markup, answer-style content and freshness. Inside a Searvia account, AI visibility asks your customers' real questions to AI assistants every week and shows whether you are mentioned, recommended or cited, and its Check access button tests robots.txt rules, firewall blocks and llms.txt for each AI bot. That is the measurement that tells you whether any of this work pays off. For the wider strategy, see our guide to generative engine optimization.