Block the trainers, keep the librarians.

One company, three different bots

Most AI providers document three kinds of visitor. A training crawler collects pages that may be used to build future models. A search crawler builds the index an assistant searches when it answers with sources. A user fetcher opens a specific page because someone in a chat asked it to.

Sources: OpenAI crawler documentation, Anthropic’s crawler policy, Perplexity’s bot documentation and Google’s list of common crawlers. These pages change; read the current version before you edit a production file.

Blocking GPTBot does not hide you from ChatGPT search

This is the most common mix-up. OpenAI states that each setting is independent: a site can allow OAI-SearchBot to appear in search results while disallowing GPTBot. So “we blocked GPTBot and disappeared from ChatGPT” would need another explanation. Check OAI-SearchBot, your firewall and whether the page is simply not relevant to the question.

The reverse is also true. Allowing GPTBot does not buy you a citation. OpenAI does not describe training access as a ranking factor for ChatGPT search, and we have not seen any evidence that it works as one. If someone tells you it does, ask for the source.

Google and Bing work differently

There is no separate “AI Overviews bot” to allow. Google’s AI optimization guide says a page must be indexed and eligible to appear in Google Search with a snippet to be shown in its generative AI features. In practice that means normal Googlebot access and snippet controls. Google-Extended is a separate opt-out for training and Gemini grounding, and Google says it is not a Search ranking signal.

Search Console also has its own generative AI control for Search. It is worth a look if your pages never appear in AI Overviews even though they rank.

Microsoft announced its Copilot controls through existing Bing tags rather than a new crawler. In its 2023 announcement, NOARCHIVE keeps a page out of chat answers and model training, while NOCACHE limits answers to the URL, title and snippet. Search Engine Journal reported in February 2026 that Bing’s revised guidelines still describe NOARCHIVE as keeping content out of Copilot answers. We could not load the official guidelines page to confirm the wording, so check it before relying on the exact effect.

What llms.txt does (so far, not much)

llms.txt was proposed in September 2024 as a Markdown file that gives language models a tidy summary of a site. It is a proposal, not a standard that the major assistants have committed to.

Google is unusually direct about it: creating an llms.txt file “will neither harm nor help” your visibility in Google Search, because Google Search ignores it. Ahrefs measured requests to llms.txt files across 137,210 domains in its own analytics in May 2026 and reported that 97% of the files it found received no requests at all. That is vendor data from SEO-aware sites, so treat it as a strong hint rather than a census.

So: if a file takes ten minutes and helps a developer tool you actually use, fine. If an agency sells it as a citation strategy, it is selling a text file.

Your robots.txt may not be the real gatekeeper

robots.txt is a request, not a lock. The standard, RFC 9309, says its rules are “not a form of access authorization”. The opposite problem is more common for marketing teams: something in front of the website blocks bots that robots.txt welcomes.

Cloudflare, for example, began asking new domains whether to allow AI crawlers in July 2025, and its AI bot settings have changed since, including how they treat bots that mix search and training. If someone ticked a box during setup, your robots.txt might say yes while the firewall says no.

• Open your live robots.txt and list every AI user agent it mentions.
• Check your CDN or hosting firewall for AI-bot rules, bot-fight modes or challenges on public pages.
• Look at your server or CDN logs for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot. Are they getting 200 responses, or 403s and challenges?
• Write the decision down per bot, with the reason and the date. Future you will want to know why.

A sensible default for most brands

If your goal is to be recommended in AI answers, a reasonable starting point is: allow the search crawlers, decide about training crawlers on its own merits, and leave normal search engine access untouched. Publishers with valuable archives may reasonably choose differently; licensing is a business decision, not an SEO trick.

Then measure instead of guessing. Access only makes a citation possible. Whether you are actually mentioned depends on whether your page answers the buyer’s question better than the alternatives. Our guide to improving AI brand visibility covers that part, and the free baseline shows how to check before and after.