What is llms.txt? How it works and how to write one

What llms.txt is

The idea came from Jeremy Howard of Answer.AI in 2024. Websites are built for browsers: menus, scripts and layout wrap the useful text. A language model working inside a limited context window does better with a short, clean description of what a site contains and where the good material lives.

The file is plain markdown, served as text at the root of the domain. By convention it starts with a single top-level heading naming the site, followed by a short blockquote summary, then optional notes, then sections of links. Each link is a markdown link with a short description after it. A section called Optional can hold links that are useful but safe to skip when space is tight.

Many sites also publish a longer companion, often called llms-full.txt, that includes the full text of key pages in one file. That is a convention, not part of the core proposal.

What it does and does not do

Some developer tools and coding assistants read llms.txt, especially for documentation: a tool pointed at your docs can load the file to find the right pages. That is a real use, and it is why documentation sites adopted it first.

It is not a search ranking signal. Google's guidance on its AI features says no special files or markup are needed to appear in them, and none of the major AI search engines publishes a statement that it ranks or retrieves pages because of llms.txt. Little public evidence shows the large AI crawlers fetching the file. Expect it to help tools that deliberately look for it, and nothing more.

That makes it worth the ten minutes it takes, as long as the effort stays proportionate. The pages the file points to matter far more than the file does: if those pages are thin, unclear or blocked, a tidy llms.txt will not rescue them.

Related reading: AI crawlers, robots.txt and llms.txt: what to allow and why and What server logs reveal about AI crawlers.

A template you can copy

Start with the site name as a heading and a single sentence saying what you do and for whom. Then add a section for your most useful pages, such as product overview, pricing, documentation and your best guides. Keep each description factual and specific, like "Pricing: plans, limits and what each includes".

Example structure: a heading with your brand name; a blockquote with one sentence about what the product does; a section titled Product with links to the main product pages; a section titled Guides with your five best articles; a section titled Company with About and Contact; and an Optional section for changelog and legal pages.

Use absolute URLs, check that every link returns a normal page, and keep the file short. Fifteen to thirty good links beat three hundred.

How to publish and check it

Place the file at the root so it loads at /llms.txt, serve it as text or markdown, and make sure robots.txt does not block it. Do not put it behind a login, a redirect chain or a script that renders it in the browser.

Then check three things: it loads with a 200 status, the content type is plain text or markdown, and every link in it works. OClever's free llms.txt generator builds the file from a short form, and any free site checker can confirm that the file loads and that robots.txt allows the main AI crawlers.

Update it when your key pages change. A stale file that points to removed pages is worse than none.

What matters more than llms.txt

Make sure AI and search crawlers are allowed in robots.txt, that important pages load without needing a login or heavy scripts, and that key facts appear in plain text rather than inside images or interactive widgets.

Write pages that answer real questions directly, keep facts consistent across your own pages and your profiles elsewhere, and earn mentions on sites that engines already cite for your topic. Those steps move both classic rankings and AI answers. llms.txt is the small finishing touch after them.