Your robots.txt Blocks ChatGPT's Search Crawler: How to Check Every Bot by Name

Summary

17.1% of sites blocking GPTBot also block OAI-SearchBot; among self-written rules the error rises to 31.3%, while Cloudflare’s managed file affects only 0.3%.
AI crawlers fall into three categories — training bots, search/answer bots, and user-fetch agents — and each requires a different robots.txt decision.
PerplexityBot is a search crawler, not a training bot; 4.6% of sites block it and 43.1% of those also block Perplexity-User.
Audit both layers: fetch robots.txt, label every AI bot, remove Disallow: / from answer bots, check CDN/WAF rules, and schedule quarterly re-checks.
For teams that need implementation support, talk to Synscribe about technical SEO audits, AI search optimization, and hands-on fixes.

When site owners write their own robots.txt rules to block GPTBot, OpenAI's model-training crawler, nearly one-third of them also block OAI-SearchBot — the crawler that powers ChatGPT's live search results. That finding comes from a Trakis analysis of 13,938 readable robots.txt files drawn from the Common Crawl archive (CC-MAIN-2026-39, September 2026). The data and code are published on Zenodo under DOI 10.5281/zenodo.23416952 (data CC BY 4.0, code MIT). Trakis sells an SEO audit tool and disclosed that relationship alongside the research.

The error is not universal. It is concentrated in one specific population: sites that wrote their own rules. That distinction matters more than the headline average.

This article gives web teams a bot-by-bot map and a two-layer audit they can run in ten minutes. The goal is to help sites opt out of model training — if that is the intention — without disappearing from AI-powered search answers.

The Three Categories Every `robots.txt` Rule Must Reflect

A common misconception is that being in a model's training data is what gets a site cited in AI answers. It is not. Modern AI assistants retrieve live information from the web through dedicated search crawlers. Being cited depends on being accessible to those crawlers, not on being present in training data.

There are three distinct categories of AI bot:

Training bots. Collect content to build or refine a foundational model. Blocking them is the standard way to opt out of contributing to future model versions.
Search and answer bots. Index the live web to ground AI responses in current information. Blocking them removes a site from AI-powered answers entirely.
User-fetch agents. Visit a specific URL when a user directly instructs an AI assistant to open it.

Some vendors state these agents do not reliably honour robots.txt. The Trakis study cites OpenAI stating that robots.txt rules "may not apply" to ChatGPT-User, and Perplexity stating that Perplexity-User "generally ignores" robots.txt.

The table below maps every major AI bot to its category.

Vendor	Bot Name	Category	Notes
OpenAI	`GPTBot`	Training	Trains future GPT models
OpenAI	`OAI-SearchBot`	Search / Answer	Powers ChatGPT's live search. Allow for visibility.
OpenAI	`ChatGPT-User`	User-Fetch	Fetches a URL when a user asks ChatGPT to open it
Anthropic	`ClaudeBot`	Training	Trains future Claude models
Anthropic	`Claude-SearchBot`	Search / Answer	Powers Claude's live search. Allow for visibility.
Anthropic	`Claude-User`	User-Fetch	Fetches a URL when a user asks Claude to open it
Google	`Google-Extended`	Training	Controls use of content for Gemini and future Google model training
Google	`Googlebot`	Search / Answer	Google's primary search crawler. Unaffected by `Google-Extended` rules.
Perplexity	`PerplexityBot`	Search / Answer	Perplexity's search crawler, not a training bot. Blocking it removes the site from Perplexity answers.
Perplexity	`Perplexity-User`	User-Fetch	Fetches a URL when a user asks Perplexity to open it
Common Crawl	`CCBot`	Training	Public web archive. Blocked by 11.7% of sites in the study.
Apple	`Applebot-Extended`	Training	Trains Apple Intelligence models. Blocked by 9.1% of sites.
Meta	`Meta-ExternalAgent`	Training	Trains Meta's AI models. Blocked by 9.7% of sites.

Across the 13,938 sites in the Trakis dataset, training bots are blocked at roughly three times the rate of search and answer bots: GPTBot at 11.4%, ClaudeBot at 10.4%, and CCBot at 11.7%, versus OAI-SearchBot at 3.5%, Claude-SearchBot at 3.2%, and PerplexityBot at 4.6%.

Cloudflare's Managed File vs. Self-Written Rules

The headline figure from the Trakis study — 17.1% of GPTBot blockers also block OAI-SearchBot — is an average of two populations that behave very differently. Treating it as a single number obscures where the real problem sits.

Of the 1,331 sites that block GPTBot by name, 45.8% use Cloudflare's managed robots.txt. That managed file is generated automatically when Cloudflare's Bot Preference Sync feature is enabled with at least one training bot set to Block. It names training bots only and leaves answer bots unnamed, so answer crawlers stay allowed.

The result: only 0.3% of sites using Cloudflare's managed file also block OAI-SearchBot. The managed configuration is correct by design.

Sites that wrote their own rules produce a very different outcome. Of the 722 sites with self-written rules blocking GPTBot:

31.3% also block OAI-SearchBot
47.0% also block ChatGPT-User

This pattern repeats for Claude. Of ClaudeBot blockers overall, 14.2% also block Claude-SearchBot. Among those with self-written rules, the figure rises to 29.2%.

The most likely cause is copy-pasted blocklists or broad wildcard patterns that catch both training and answer bots without distinguishing between them.

Larger sites are more exposed. Among the top 10,000 domains in the study, 34.9% of self-written rules incorrectly block OAI-SearchBot. For sites ranked 100,000 to 1,000,000, the figure is 24.5%. Scale and legacy rule sets appear to compound the error.

The Perplexity Trap

Perplexity's bot structure is the exact opposite of OpenAI's and Anthropic's, and this is where the most consequential misunderstanding occurs.

OpenAI separates its bots clearly: GPTBot trains models, while OAI-SearchBot powers search. Anthropic does the same with ClaudeBot and Claude-SearchBot.

Perplexity does not publish a separate named training bot. PerplexityBot is their search and answer crawler.

Blocking PerplexityBot does not protect a site from training data inclusion. It removes the site from Perplexity's answer engine.

The Trakis research found that 4.6% of all 13,938 sites block PerplexityBot. Of those, 43.1% also block Perplexity-User — a pattern consistent with treating PerplexityBot as a training bot when it is not.

Google-Extended is the clean control case. It controls only Gemini training, while Googlebot is Google's separate search crawler. Only 0.6% of sites that block Google-Extended also block Googlebot.

The Google distinction is well understood. The Perplexity distinction is not.

The goal for robots.txt is to apply the same clarity to every vendor: identify each bot's category, then write rules that target only the category the site intends to restrict.

Your Two-Layer, 10-Minute Audit

A robots.txt file is a request, not an enforcement mechanism. A site can allow OAI-SearchBot in the file and still block it at the CDN or WAF. The audit must cover both layers. Neither reveals what the other contains.

The Trakis study also found that Cloudflare's managed robots.txt appeared and disappeared between archive snapshots on the same domains, meaning settings can change silently after a plan change or configuration update. That makes periodic re-checks necessary, not optional.

Work through these steps in order.

Step 1: Fetch Your `robots.txt`

Visit yourdomain.com/robots.txt directly in a browser. Copy the full contents. Look for every User-agent: group that names an AI crawler.

Step 2: Categorise Every AI Bot in the File

Using the table earlier in this article, label each named bot as Training, Search/Answer, or User-Fetch. Write the category next to each User-agent: entry before reading any of the Disallow: lines below it.

Step 3: Check for Incorrect `Disallow` Rules

For any bot labelled Search/Answer or User-Fetch, check the Disallow: directive. A Disallow: / blocks the bot from the entire site.

Pay specific attention to:

OAI-SearchBot
Claude-SearchBot
PerplexityBot
ChatGPT-User
Perplexity-User
Claude-User

If any of these carry Disallow: /, the site is blocking AI answer crawlers. Remove the directive or replace it with Allow: / to restore access.

Step 4: Audit the Edge Layer

This layer is not visible in robots.txt. It requires a separate check.

For Cloudflare Users

These settings enforce blocks at the network edge and override whatever the robots.txt file states. A permissive robots.txt combined with a WAF rule blocking OAI-SearchBot by user-agent string still blocks the crawler.

For Other CDN or WAF Users

Check your firewall rules, bot management settings, and any custom rules filtering on user-agent strings. Export and review them specifically for the six bot names listed in Step 3.

Step 5: Log Results and Schedule a Re-Check

Record what each named bot is currently set to at both layers, and note the date. The Trakis findings show that managed rule sets can change between reviews. Re-run this two-layer audit quarterly, and again after any CDN plan change or new AI crawler announcement.

If you would rather not run that schedule by hand, audit with Synscribe. Its crawler monitoring checks both the robots.txt and edge layers continuously and flags silent changes before they cost you AI citations.

Audit Checklist

Use this as a quick reference after completing the steps above.

Fetched yourdomain.com/robots.txt and copied the full contents
Labelled every named AI bot as Training, Search/Answer, or User-Fetch
Confirmed OAI-SearchBot, Claude-SearchBot, and PerplexityBot are not under a Disallow: / rule
Confirmed ChatGPT-User, Perplexity-User, and Claude-User are not blocked at robots.txt level
Reviewed Cloudflare AI Crawl Control and Bot Fight Mode (or equivalent WAF settings) for edge-layer blocks
Logged the result for each bot with a date
Set a calendar reminder for a re-audit in three months

Stop Blocking the Bots That Cite You

The mechanics are straightforward once the bot categories are clear. Training bots and answer bots carry different names for a reason. A robots.txt file that treats them the same removes a site from AI-powered search without achieving anything additional in terms of training opt-out.

The broader context matters too. Technical SEO elements like robots.txt determine whether a site is findable and citable in both traditional and AI-powered search. Getting the bot taxonomy right is the prerequisite for everything that follows.

If you would rather not run that two-layer check yourself, talk to Synscribe. A technical audit of your crawler rules takes less time than losing another quarter of AI citations.

Frequently Asked Questions About AI Crawlers

What is the difference between GPTBot and OAI-SearchBot?

GPTBot is OpenAI’s training crawler; OAI-SearchBot powers ChatGPT’s live search and citations. Block GPTBot to opt out of model training. Allow OAI-SearchBot if you want your site eligible to appear in ChatGPT answers.

Should I block AI crawlers in robots.txt?

Only block the bots whose role you want to exclude. Block training bots such as GPTBot, ClaudeBot, or CCBot to opt out of training, but keep search/answer bots like OAI-SearchBot and Claude-SearchBot allowed if AI search visibility matters.

Why do so many self-written robots.txt rules block AI search crawlers?

Copy-pasted blocklists and broad wildcard rules often catch training and answer bots together. In the Trakis study, 31.3% of self-written GPTBot rules also blocked OAI-SearchBot, versus 0.3% of Cloudflare managed-file users.

Is PerplexityBot a training bot?

No. PerplexityBot is Perplexity’s search and answer crawler, not a named training bot. Blocking it removes your site from Perplexity answers without opting out of model training. Use the same caution with Perplexity-User, which is a user-fetch agent.

How do I audit my robots.txt for AI crawler mistakes?

Fetch yourdomain.com/robots.txt, label every named AI bot as training, search/answer, or user-fetch, then check for Disallow: / on answer bots. Repeat at the CDN/WAF layer, because edge rules can override a permissive robots.txt.

Does Cloudflare's managed robots.txt block AI search bots?

No, by design. Cloudflare’s managed robots.txt names training bots such as GPTBot, ClaudeBot, and CCBot but leaves answer bots unnamed, so they remain allowed. Only 0.3% of sites using it also blocked OAI-SearchBot in the Trakis dataset.

How often should I recheck AI bot rules?

Quarterly is a practical baseline, plus after any CDN plan change, WAF update, or new AI crawler announcement. Managed robots.txt files have appeared and disappeared between archive snapshots, so settings can change silently.

Will allowing OAI-SearchBot put my content in training data?

No. OAI-SearchBot is used to retrieve live web information for ChatGPT search results; GPTBot is the training crawler. Allowing OAI-SearchBot keeps your site eligible for citations without granting training access through that bot.

What if I want to opt out of AI training completely?

Block the named training bots: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, and Meta-ExternalAgent. Do not block OAI-SearchBot, Claude-SearchBot, or PerplexityBot unless you also intend to disappear from those AI answer engines.