Allow the search bots, decide on the training bots, and fix the robots.txt mistakes that hide your site from AI answers.
Robots.txt for AI crawlers is no longer a one-line decision. OpenAI, Anthropic, and Perplexity each run separate bots for training, search indexing, and user requests, and each bot reads its own rule. Get one line wrong and your pages drop out of ChatGPT or Claude answers while you think you only opted out of model training.
This guide gives you the file, the checks, and the mistakes to avoid. The numbers come from a 10,000-domain robots.txt study published in October 2026 and from the crawler documentation that OpenAI, Anthropic, Perplexity, and Google publish themselves.
Each vendor splits its crawling into training, search, and user-request bots, and only the search and user bots decide whether you show up in answers. The names look alike, so teams block the wrong one.
Each vendor treats these as independent settings. Blocking the training bot does nothing to the search bot.
Allow every search and user-request bot: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, and Perplexity-User, plus Googlebot and Bingbot. These are the bots that fetch your pages at the moment a buyer asks an AI tool about your category.
Your pricing page, comparison pages, integration docs, and case studies are the pages buyers' questions pull in. If a search bot can't fetch them, the engine answers from someone else's page. Our crawl-to-refer ratio analysis shows how lopsided the traffic exchange is, and you can still decide the exchange is worth it, because being absent from the answer costs you more than the crawl does.
About half of the sites that block a training bot keep the matching search bot open, and the other half lock out both. AICrawlerCheck fetched robots.txt files from the Tranco top 10,000 domains on October 5, 2026, and 4,962 returned a valid file.
Among those 4,962 files, GPTBot was blocked on 14.3% and ClaudeBot on 13.4%. OAI-SearchBot was blocked on 7.4% and Claude-SearchBot on 7.3%. Of the 710 files that block GPTBot, 347 (48.9%) still allow OAI-SearchBot. Of the 665 that block ClaudeBot, 307 (46.2%) still allow Claude-SearchBot.
That leaves 10.8% of sites blocking at least one search crawler. Some did it on purpose. Many copied a "block all AI bots" snippet from a blog post and never checked what it covered. Cloudflare Radar's September 28, 2026 snapshot of 4,242 files shows the same pattern from a different sample: GPTBot fully disallowed on 8.11% of files and ClaudeBot on 6.91%. The percentages differ because the samples differ, and the direction matches.
Give every AI bot its own User-agent group, put the search and user bots on Allow, and make your training decision explicit. This version fits most B2B sites that want citations:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Training bots: change to "Disallow: /" if you opt out
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
Sitemap: https://www.example.com/sitemap.xmlOne rule trips up teams that already have a "User-agent: *" group. Under RFC 9309, a crawler follows the most specific group that names it and ignores the wildcard group. If your wildcard group disallows /admin/ or /staging/, copy those lines into each bot group above.
Monitor your progress with Nobori: track whether the engines you just opened up start citing your pages.
Block them only if you have a licensing or legal reason, because the vendor documentation does not tie training access to search inclusion. OpenAI says "Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models." That sentence says nothing about ChatGPT search.
Google works the same way. Its crawler documentation says Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search." The token controls whether Google uses your content for Gemini training and grounding in other products.
If your legal team wants out of training sets, block GPTBot, ClaudeBot, and Google-Extended, and leave the search bots alone. If nobody has raised the question, leaving training bots open costs you nothing you can measure today. Write the decision down with a date so the next person who edits the file knows it was deliberate.
Most mistakes are old SEO problems that now cost you AI visibility. The AICrawlerCheck study of 4,962 valid files counted these:
A 403 or an HTML page at the robots.txt address is the nastiest case. The bot can't read your rules, so it falls back on its own default behavior, and you won't see an error anywhere in your CMS.
Monitor your progress with Nobori: a drop in citations right after a deploy often traces back to a robots.txt change.
Yes, through Googlebot, not through Google-Extended. Google's AI features documentation says "robots.txt directives for Googlebot is the control for site owners." If Googlebot can't crawl a page, that page can't appear in AI Overviews or AI Mode.
Eligibility is short. A page must be "indexed and eligible to be shown in Google Search with a snippet," and Google lists no additional technical requirements. To limit how much of your page appears, use nosnippet, data-nosnippet, max-snippet, or noindex. Google warns that preview controls can take "several days to several months" to apply after a recrawl.
What this means for you: never block Googlebot from the pages you want cited, and treat nosnippet as a citation-reducing setting. A nosnippet tag on a pricing page removes the text Google could quote.
Often they don't. OpenAI's documentation says that for ChatGPT-User, "robots.txt rules may not apply" because a person started the visit. Perplexity says Perplexity-User "generally ignores robots.txt rules" for the same reason.
The AICrawlerCheck study found 287 files that block ChatGPT-User. Those owners likely think they've stopped ChatGPT from opening their pages, and the vendor says the rule may not hold. If you truly need to stop user-triggered fetches, you'd need a firewall or CDN rule, and that also stops buyers from getting your page quoted in a live answer.
For most B2B teams, the right move is to leave these bots on Allow and spend the effort on the pages they fetch.
Plan for about a day for OpenAI and Perplexity, and longer for Google. OpenAI says "it can take ~24 hours from a site's robots.txt update for our systems to adjust." Perplexity says it "may take up to 24 hours for our systems to reflect changes."
Citations move slower than crawl access. A bot has to fetch the page, the engine has to index it, and a buyer's prompt has to pull it in. Edit the file, wait two days, confirm the bots fetch your pages in your server logs, and then watch citations over several weeks. Changing the file again every few days makes it impossible to tell which edit did what.
Run four checks, in this order, and fix each before you move to the next.
curl -I https://yourdomain.com/robots.txt. You want a 200 status and a text/plain content type. Repeat for www and for every subdomain, since each host serves its own file and Anthropic's instructions say to set rules per subdomain.curl -A "OAI-SearchBot" -I https://yourdomain.com/ and repeat for each bot in your file. A 403 or a challenge page means your firewall or CDN is blocking a bot your robots.txt allows.Monitor your progress with Nobori: once the four checks pass, citation tracking shows whether access turned into answers.
Pick 20 to 30 buyer prompts before you change the file, record which of your pages get cited today, and re-run the same prompts weekly for six weeks. Access is a prerequisite and citations are the outcome, so measure the outcome.
Group the prompts by page type: pricing, comparison, integration, and how-to. If pricing prompts start citing your pricing page and nothing else changes, the access fix worked. If nothing moves, the problem sits in the content, not the file. Resist adding an llms.txt file as the next fix. Our llms.txt test found no citation lift from it.
Nobori tracks your brand across ChatGPT, Google AI Overviews, Perplexity, Gemini, and Claude, refreshed daily. Load the 20 to 30 prompts from your pre-change baseline, and you can watch which engines start citing which pages after the robots.txt edit.
The citation view also shows who gets cited when you don't. If a competitor's pricing page keeps winning the prompts where yours is missing, you know the next fix is page content, not crawler access, and Nobori turns that gap into a task for your team.
Run the baseline first. A robots.txt fix with no before-and-after record is a guess, and a tracked one is evidence.
See if AI engines are citing you → nobori.ai