AI crawlers, robots.txt and llms.txt: what to allow and why

Three jobs, three kinds of crawler

Training crawlers collect pages that may be used to train or improve models. Blocking them is a decision about how your content is used for model development.

Search or index crawlers build the index that powers an engine's search features and citations. Blocking them is likely to reduce how often your pages are retrieved and cited in that engine's answers.

User-initiated fetchers visit a page because a person asked an assistant to look at it or to research something. They behave more like a browser acting for a user, and vendors document whether they follow robots.txt.

Because the three jobs are separate, each major vendor publishes separate user-agent names. Decide for each job, not for 'AI' as a whole.

The main crawlers, by vendor

OpenAI documents GPTBot (collection of data that may be used in training), OAI-SearchBot (supports search features in ChatGPT) and ChatGPT-User (visits triggered by a user's request).

Anthropic documents ClaudeBot (training data collection), Claude-SearchBot (improving search result quality) and Claude-User (fetches initiated by a user's question).

Perplexity documents PerplexityBot (builds the index for its search results) and Perplexity-User (fetches triggered by a user's question).

Google uses Googlebot for Search, which also underlies AI Overviews and AI Mode eligibility. Google-Extended is a separate robots.txt token that controls whether content Google crawls can be used for Gemini model training and grounding; it does not affect inclusion in Search.

Vendors update these lists. Always check the vendor's current documentation before changing rules.

Related reading: What is llms.txt? How it works and how to write one and What server logs reveal about AI crawlers.

How to write the rules

robots.txt works per user-agent. A rule that applies to everyone is written as 'User-agent: *'. To treat one crawler differently, add a group for it, for example: User-agent: GPTBot followed by Disallow: / to block it from the whole site, or by Allow: / to say it is welcome.

Be careful with broad rules. A blanket 'Disallow: /' under 'User-agent: *' blocks every crawler that obeys robots.txt, including search engines. Review the file as a whole, and test changes with a robots.txt tester before publishing.

Remember that robots.txt is a request, not an access control. Well-behaved crawlers follow it. It will not stop someone who ignores it, and it is not a way to protect confidential content.

A sensible default for most brands

If your goal is to be recommended by AI engines, allow the search and user-initiated crawlers of the engines your customers use, so your pages can be retrieved and cited. Whether to allow training crawlers is a separate business and policy decision. Some publishers allow them to be represented in future models; others block them to control reuse of their content.

Whatever you decide, write it down and apply it deliberately. A common accident is a legacy rule written years ago, or added by a plugin or CDN setting, that now blocks a crawler nobody meant to block.

Beyond robots.txt: things that also block crawlers

Bot protection and firewall rules on a CDN may challenge or block unfamiliar user agents even when robots.txt allows them. Check your CDN's bot settings and look in your server logs for blocked requests from AI crawler user-agents.

Pages that require JavaScript to display their content may appear empty to crawlers that do not run scripts. Test by fetching a page without JavaScript and checking that the main content is present in the HTML.

Login walls, cookie walls and aggressive interstitials can also hide content. If an assistant cannot read a page, it cannot cite it.

What about llms.txt?

llms.txt is a proposal for a plain text file at the root of a site that gives language models a short, curated guide to the most useful content, in a simple format. It is easy to publish and harmless to include.

At the time of writing, major engines have not confirmed that they read llms.txt as a ranking or retrieval signal, so do not expect it to change your visibility by itself. Treat it as a low-cost extra, written for clarity, and keep your real effort on crawlable, clear pages and accurate third-party mentions.

How to check it is working

Look at your server or CDN logs and filter for the user-agent names above. You want to see requests from the crawlers you intended to allow, with successful responses on your important pages, and none from those you meant to block.

OClever's crawler analytics reads those logs and shows, per crawler, which pages were fetched and with what response. The Agent Readiness checks test your robots rules, bot challenges, schema, sitemap and llms.txt from the outside, so you can catch a silent block quickly.