Book a call Free AI Visibility Check

Technical guide

AI crawlers and llms.txt: what to allow, what to skip

Which AI bots read your site, what each one is for, how to set robots.txt, and where llms.txt fits.

  • Checked against OpenAI, Anthropic, Perplexity and Google docs
  • Google Search ignores llms.txt
  • Our own robots.txt and llms.txt as examples

What are AI crawlers?

Bots that AI companies send to read websites. They do three jobs: collect training data, build an index for AI search, and fetch a page when someone asks a question. You can allow or block each job separately.

One site, three kinds of AI bot

Your website
  1. Training crawler

    future AI models

    • GPTBot
    • ClaudeBot
    Your call
  2. Search crawler

    AI search indexanswers with links

    • OAI-SearchBot
    • Claude-SearchBot
    • PerplexityBot
    Allow if you want to be cited
  3. User fetcher

    one page, right now, for one question

    • ChatGPT-User
    • Claude-User
    • Perplexity-User
    Allow if you want to be cited

Google Search ignores llms.txt: it neither helps nor harms visibility there.

Google Search Central, July 2026

Three jobs, three decisions

"AI crawler" sounds like one thing. In practice, each big AI company runs several bots, and each bot has a different job. That matters, because the right answer to "should I block it?" depends on the job.

  • Training.

    The bot collects pages that may be used to train future models. Blocking it is about whether your content goes into the next model. It does not decide whether you show up in answers today.

  • AI search.

    The bot builds the index that AI search answers draw on, with links back to sources. Blocking it is the closest thing to removing yourself from those answers.

  • User fetch.

    A person asks a question, and the assistant opens your page right then to answer it. This is not crawling in the usual sense: one page, one moment, one question.

Table: the main AI bots by job. Checked against each company's documentation on 13 November 2026.
CompanyTrainingAI searchUser fetch
OpenAI GPTBotOAI-SearchBotChatGPT-User
Anthropic ClaudeBotClaude-SearchBotClaude-User
Perplexity none listed: Perplexity says PerplexityBot is not used to train foundation modelsPerplexityBotPerplexity-User
Google Google-Extended (a robots.txt token, not a separate bot)Googlebot (Google Search, including its AI features)not listed separately

Names change. Companies add bots and retire old ones, so check the source pages linked at the end before you edit your own file.

What blocking each bot actually does

Here is what each company says happens when you block its bots, in plain words.

  • OpenAI.

    Block OAI-SearchBot and your site stops appearing in ChatGPT search answers. Block GPTBot and you signal that your content should not be used to train OpenAI's models. The two settings are independent: you can stay in search and stay out of training. ChatGPT-User acts on a user's request, and OpenAI says robots.txt rules may not apply to it.

  • Anthropic.

    Block ClaudeBot and Anthropic excludes your future content from model training. Block Claude-SearchBot and your content is not indexed for Claude's search. Block Claude-User and Claude cannot open your pages when a user asks about them. Anthropic says blocking by IP address is unreliable; use robots.txt.

  • Perplexity.

    PerplexityBot decides whether Perplexity can show and link your site in its answers, and Perplexity recommends allowing it. Perplexity-User fetches a page because a user asked, and Perplexity says it generally ignores robots.txt for that reason.

  • Google.

    There is no separate AI crawler for Google Search. Its AI features, such as AI Overviews, work from the normal Search index, so a page needs to be indexed and eligible to show with a snippet. Google-Extended is different: it controls whether content Google already crawls is used to train Gemini models and to ground answers in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

The pattern: blocking training bots is mostly a decision about your content's future. Blocking search bots and user fetchers is a decision about being cited now.

A robots.txt you can copy

robots.txt is a plain text file at the root of your domain, for example https://example.com/robots.txt. Bots read it before crawling. Say you want two things: to be cited in AI answers, and to stay out of model training. This version does both.

robots.txt
# AI search and user-requested fetches: allowedUser-agent: OAI-SearchBotUser-agent: ChatGPT-UserUser-agent: Claude-SearchBotUser-agent: Claude-UserUser-agent: PerplexityBotUser-agent: Perplexity-UserAllow: /# Model training: your call. This version says no.User-agent: GPTBotUser-agent: ClaudeBotUser-agent: Google-ExtendedDisallow: /# Everyone else, including GooglebotUser-agent: *Allow: /Sitemap: https://example.com/sitemap.xml

A few things to know before you paste it:

  • If you're happy to be used for training, delete the middle block. Your site is then open to every bot, which is what we do on citedlift.com.
  • Blocking Google-Extended has a side effect. It covers grounding in Gemini Apps as well as training. If being cited in the Gemini app matters to you, think twice before blocking it.
  • Blocking training is not retroactive. It applies to future crawls, not to models that already exist.
  • Never block Googlebot to stay out of AI Overviews. It takes you out of Google Search as a whole.

Why bots may still miss your site

A clean robots.txt is the first gate. Two more can quietly shut AI bots out.

robots.txt

CDN or firewall

Your CDN or firewall. Bot protection runs before robots.txt is even read. Since 1 July 2025, Cloudflare asks every newly added domain whether to allow AI crawlers, and many sites turned on a one-click AI bot block before that. If your site sits behind Cloudflare or a similar service, open its bot settings and check what is blocked.

Cloudflare, 1 July 2025

HTML without JavaScript

JavaScript. In a study published in December 2024, Vercel and MERJ found that the major AI crawlers, including GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, download JavaScript files but don't run them. The exceptions were Gemini, which uses Googlebot's infrastructure, and AppleBot. So if your text only appears after JavaScript runs in the browser, those bots see an empty page. Your key content, prices and FAQ should be in the HTML the server sends.

Vercel and MERJ, December 2024

How to check your site

You don't need a tool for a first check.

  1. Open your robots.txt in a browser. Look for Disallow: / under any of the bot names in the table above, and under User-agent: *.

  2. Check your CDN or hosting bot settings. Look for anything called "AI bots", "AI scrapers" or "bot fight mode", and see what it blocks.

  3. Look at the raw HTML. Right-click your page, choose "View page source", and search for a sentence from your main text. If it isn't there, AI crawlers that don't run JavaScript won't find it either.

  4. Check Google Search Console. Your key pages should be indexed. That is the entry ticket to Google's AI features.

  5. Read your server logs. Search for the bot names. Anyone can fake a user agent, so OpenAI and Perplexity publish IP address lists you can match against.

If the bots can reach your pages, the next question is what AI answers actually say about you. That is a measurement job, not a robots.txt job.

What llms.txt is

llms.txt is a proposed file that gives AI tools a short, curated guide to your site. Jeremy Howard proposed it in September 2024 on llmstxt.org. It is a proposal, not a standard that AI companies have signed up to.

The format is plain Markdown at /llms.txt:

llms.txt
# Your Company> One or two sentences: what you do, for whom, where.## Services- [Service A](https://example.com/service-a/): what it is and who it's for- [Pricing](https://example.com/pricing/): packages and starting prices## Optional- [Blog](https://example.com/blog/): guides and notes
  • H1 an H1 with your company or site name (the only required part);
  • blockquote a blockquote with a short summary;
  • optional paragraphs with key details;
  • H2 H2 sections with lists of links to your most useful pages, each with a short note;
  • Optional an optional section called "Optional" for links a tool can skip when it's short on space.

The proposal also suggests offering clean Markdown copies of your pages, for example page.md next to page.html.

Should you publish llms.txt?

Short answer: if it takes you an hour, yes. If it takes you a week, no.

What we know:

Google Search ignores llms.txt: it neither helps nor harms visibility there.

(Google Search Central documentation, updated 10 July 2026)

The crawler documentation from OpenAI, Anthropic and Perplexity doesn't mention llms.txt at all (checked 13 November 2026). So no major AI search engine has said it reads the file.

Why we still publish one:

  • It costs little. A small Markdown file, written once, updated when your offer changes.
  • It forces clarity. Writing a one-paragraph summary of your company is a useful exercise. If you can't, AI answers will struggle too.
  • Some tools do use it. Coding assistants and AI agents that a person points at your site can read it. That use is real, even if it isn't search.

What it won't do:

it won't replace crawlable pages, it won't fix a blocked bot, and it won't make an AI answer mention you. Put robots.txt, CDN settings and server-rendered HTML first.

What we do on citedlift.com

We use our own site as a test bed, so here is the setup, as of 13 November 2026:

  • robots.txt allows every bot, training ones included. We want our ideas to travel, and we'd rather be in the model than out of it.

    See our robots.txt
  • llms.txt gives a short profile: who we are, services, packages with starting prices, and our research. A longer version sits at /llms-full.txt.

    See our llms.txt
  • Every page is static HTML. Text, prices and FAQ are in the page source. Nothing important waits for JavaScript.

One more thing from our own research. When we tested AI answers about companies drawn from the public ESA BIC Noordwijk incubatee and alumni listings, we logged the sources the answers cited:

That's one sample, tested on two days, so treat it as a pattern in that group, not a rule. But it's a useful reality check. Letting AI crawlers in is the floor. Most of what AI answers say about you comes from other sites, which is why GEO and SEO differ in where the work goes. Read the full study.

1.2%

Only 1.2% of 1,798 logged citations pointed to the companies' own websites.

Our own study, 25 and 26 August 2026

What we don't know yet

  • Whether any AI search engine reads llms.txt. Google says it doesn't. The others haven't said either way.
  • How user fetchers treat robots.txt in practice. OpenAI and Perplexity say their user-triggered fetchers may not follow it. Anthropic says its bots honour robots.txt. Behaviour can change without notice.
  • How much training access matters for being named later. Blocking training keeps your future content out of future models. Whether that changes how often you're mentioned is not documented anywhere we've seen.
  • Whether the JavaScript finding still holds everywhere. The Vercel and MERJ study is from December 2024. Crawlers can change. Server-rendered HTML is the safe choice either way.

FAQ

What are AI crawlers?

Bots that AI companies use to read websites. Some collect training data, some build the index for AI search answers, and some open a page when a user asks a question. Each has its own name, so you can allow or block them one by one in robots.txt.

What is the difference between GPTBot and OAI-SearchBot?

GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot decides whether your site can appear in ChatGPT search answers. Blocking GPTBot keeps you out of training; blocking OAI-SearchBot keeps you out of ChatGPT search answers.

Should I block AI crawlers?

Block training bots if you don't want your content in future models. Allow search bots and user fetchers if you want AI answers to cite you. If being found matters to your business, keep those open.

Does blocking Google-Extended remove me from AI Overviews?

No. Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI. AI Overviews work from the Google Search index, which Googlebot crawls. Google says Google-Extended does not affect inclusion or ranking in Google Search.

Does Google use llms.txt?

No. Google says Google Search ignores llms.txt, and that it neither helps nor harms visibility there.

Do I need llms.txt?

It's optional. It is cheap to make and some AI tools and agents read it, but no major AI search engine has said it uses the file. Make sure bots can reach and read your pages first.

Why can't AI bots see my content when robots.txt allows them?

The usual reasons are a CDN or firewall that blocks AI bots, or content that only appears after JavaScript runs. Many AI crawlers don't run JavaScript, so the text has to be in the HTML.

Sources

  1. OpenAI, Overview of OpenAI crawlers checked 13 November 2026 developers.openai.com
  2. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? checked 13 November 2026 support.claude.com
  3. Perplexity, Perplexity crawlers checked 13 November 2026 docs.perplexity.ai
  4. Google, Google's common crawlers (Google-Extended) checked 13 November 2026 developers.google.com
  5. Google Search Central, AI features and your website updated 10 July 2026 developers.google.com
  6. llmstxt.org, The /llms.txt file, Jeremy Howard 3 September 2024 llmstxt.org
  7. Vercel, The rise of the AI crawler, with MERJ 17 December 2024 vercel.com
  8. Cloudflare, press release 1 July 2025 cloudflare.com
  9. CitedLift, ESA BIC Noordwijk AI visibility study tests on 25 and 26 August 2026 bearleap.space

More figures on AI search and buyers, with sources: AI search statistics.

Bots can read you. Do AI answers name you?

The Free AI Visibility Check runs 10 buyer prompts for your category and shows the results within 48 hours.