AEO

Should You Block AI Crawlers? GPTBot, ClaudeBot and robots.txt Explained

Blocking AI bots sounds like protecting your content. For most businesses it quietly removes them from the answers their customers are reading. Here is which crawlers do what, and the rules we recommend.

When generative AI arrived, many site owners did the cautious thing and blocked every AI bot they could name. Website builders and security services made it a one-click setting, and in some cases switched it on by default.

Now many of the same businesses are asking why ChatGPT recommends their competitors and never them. Often the answer is in a file most of them have never opened. When we audited our own site this month, we found our CDN adding AI rules on top of our robots.txt that we never wrote. This guide explains what each AI crawler does, what blocking it really costs, and how to set rules that match what you want.

You can't be recommended by an AI assistant that isn't allowed to read your website.

The three kinds of AI crawlers

"AI crawler" is a catch-all. The major AI companies now run separate bots for separate jobs, and each one has its own name, or user agent, that you can allow or block on its own.

AI crawlers by job

Three jobs, three different costs if you block them

Training

Collect pages that may be used to train future AI models.

  • GPTBot
  • ClaudeBot
  • CCBot
  • Google-Extended
  • Applebot-Extended
  • meta-externalagent
Block: future models may know less about you

User fetch

Open a page in real time because a person asked about it or shared the link.

  • ChatGPT-User
  • Claude-User
  • Perplexity-User
Block: the assistant can't read your page
Googlebot and Bingbot are regular search crawlers, but they also feed Google's AI Overviews and AI Mode and Microsoft Copilot. Google-Extended and Applebot-Extended are permission tokens you use in robots.txt, not separate crawlers.

AI crawler list: who's who

Each company documents its bots: OpenAI, Anthropic, Perplexity and Google. Here's the short version.

Bot (company)What it doesIf you block it
GPTBot
OpenAI
Collects content that may train OpenAI's modelsYour pages aren't used for training
OAI-SearchBot
OpenAI
Finds pages to show and link in ChatGPT searchOpenAI says you won't appear in ChatGPT search answers
ChatGPT-User
OpenAI
Visits a page when a ChatGPT user's request needs itChatGPT can't open your page for that user
ClaudeBot
Anthropic
Collects content that may train Claude modelsYour pages aren't used for training
Claude-SearchBot
Anthropic
Improves the quality of Claude's search resultsYou may appear less in Claude's answers
Claude-User
Anthropic
Fetches a page when a Claude user asksAnthropic says this can reduce your visibility in user-directed search
PerplexityBot
Perplexity
Indexes pages for Perplexity's answers. Perplexity says it isn't used for model trainingYou can drop out of Perplexity answers
Googlebot
Google
Crawls for Google Search, including AI Overviews and AI ModeYou disappear from Google. Never block it
Google-Extended
Google
Controls use of your content for Gemini training and groundingNo effect on Google Search, rankings or AI Overviews
Applebot-Extended
Apple
Controls use of your content for Apple's AI trainingApplebot still crawls for Siri and Spotlight
CCBot
Common Crawl
Builds an open web archive that many AI models train onYour pages stay out of future archive snapshots
meta-externalagent
Meta
Collects content for Meta's AI models and productsYour pages aren't used for Meta's AI

Microsoft is the odd one out. Copilot draws on the Bing index, and there's no separate AI crawler to block, so blocking Bingbot removes you from Bing and weakens your presence in Copilot. Don't.


What blocking really costs you

Blocking search and user bots has an immediate cost. When someone asks ChatGPT, Claude or Perplexity for a recommendation, the assistant searches its index and may open pages live. If its bots can't reach your site, it has to rely on what other websites say about you, or it simply names a competitor whose site it could read.

Blocking training bots has a slower, less certain cost. Your content won't be used to train future models. That's a reasonable choice for publishers whose articles are the product. For a business that wants to be known, it may mean future models know less about what you do, though nobody can measure that effect precisely today.

Blocking Google-Extended costs you nothing in Google Search. Google says it doesn't affect inclusion or ranking in Search. AI Overviews and AI Mode are controlled by Googlebot, so blocking Google-Extended won't remove you from them. Google has also said it intends to support AIPREF, a shared machine-readable signal for AI training preferences, in early 2027.

What we recommend

Pick your policy

Three robots.txt policies, and who each one suits

Most businesses

Allow everything

Search, user and training bots can all read your site. You get the most visibility in AI answers today and in future models.

Local services, clinics, firms, B2B, e-commerce

Publishers

Search yes, training no

Allow search and user bots so you can be cited and linked, and block the training bots so your content isn't used to build models.

News sites, paid content, original research

Rarely right

Block all AI bots

No AI tool can read your site directly. AI answers about you rely on other websites, and competitors fill the gap.

Private or legally restricted content

robots.txt rules you can copy

Your robots.txt file lives at yourdomain.com/robots.txt. Each bot follows the group that names it most specifically, and only falls back to the User-agent: * group when no group names it.

Option 1: allow everything (recommended for most businesses)

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Option 2: allow AI search, block AI training

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

In option 2, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User and PerplexityBot aren't named, so they follow the * group and are allowed. Changes can take a day or so to be picked up; OpenAI, for example, says its systems need about 24 hours to adjust after you update robots.txt.

Content signals

Cloudflare's Content Signals Policy, published in September 2025, adds a line to robots.txt that states how your content may be used: search, ai-input (using it in AI answers) and ai-train. Our own file says Content-Signal: search=yes, ai-input=yes, ai-train=yes. These signals state a preference. They don't block anything, and it's up to each AI company whether to honour them.


Check your firewall, not only robots.txt

robots.txt is a polite request. The bigger risk for many businesses is a block they never chose, set at their host, CDN or security plugin, which turns bots away before they ever read robots.txt.

  • Cloudflare: since July 1, 2025, new domains on Cloudflare block known AI crawlers by default. Cloudflare can also manage your robots.txt for you and add AI rules at the top of it. Look in your dashboard's AI Crawl Control and bot settings.
  • Website builders: some, such as Squarespace, have a setting to block AI crawlers. Check your SEO or crawler settings.
  • Security plugins and firewalls: broad rules against "bots" or unfamiliar user agents can catch AI crawlers, and occasionally Googlebot too.

That last kind of mistake is expensive. A firewall rule meant for AI scrapers that also blocks Googlebot removes you from Google Search, AI Overviews and AI Mode all at once.

How to test your setup in 10 minutes

  1. Open yourdomain.com/robots.txt in a browser. Read every group and look for rules you didn't write.
  2. Check your CDN or host dashboard for AI bot, bot fight or crawler settings, and note what each one is set to.
  3. Run a quick request as an AI bot, shown below. A 200 response means the page was served. A 403 means something is blocking that user agent.
  4. Look in your server or CDN logs for visits from GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. Seeing them is the real proof they can reach you.
  5. Repeat after any change to your hosting, security plugins or CDN plan.
curl -I -A "OAI-SearchBot" https://yourdomain.com/
curl -I -A "ClaudeBot" https://yourdomain.com/

Treat the curl test as a quick signal, not proof. Some firewalls check a bot's IP address as well as its name, so a test from your own computer can be treated differently from the real crawler.

Bots that ignore the rules

The major AI companies say their crawlers follow robots.txt. User-triggered fetchers are a grey area: Perplexity says its Perplexity-User fetcher generally ignores robots.txt, and OpenAI says robots.txt rules may not apply to ChatGPT-User, because in both cases a person asked for the page. Some crawlers ignore the rules altogether, and Bytespider, operated by ByteDance, is the one most often reported. For those, robots.txt isn't enough and a firewall block is the right tool. On our own site, Bytespider is the only bot we block, and we block it at the CDN.

Mistakes to avoid

  • Blocking every AI bot, then paying for "AI visibility" work that can't succeed
  • Blocking GPTBot and assuming you're out of ChatGPT search, or the reverse. They are separate bots
  • Blocking Google-Extended to stay out of AI Overviews. It doesn't do that, and blocking Googlebot would remove you from Google
  • Editing robots.txt but never checking the CDN settings that override it
  • Publishing an llms.txt file while the bots that might read it are blocked. Our guide to llms.txt explains what that file does and doesn't do

Once AI crawlers can reach you, measure what they send back. Our guide on tracking AI traffic in GA4 shows how, and our technical SEO checklist covers the rest of your crawl setup.

Is your site quietly blocking AI search?

Digital Relevance checks robots.txt, CDN and firewall settings as part of every AI Search engagement, then helps U.S. and Canadian businesses get cited and recommended by ChatGPT, Perplexity, Gemini and Google's AI features. Start with a free AI Search Snapshot, or see what our AI Search plan covers.

Get My Free AI Search Snapshot

Frequently Asked Questions

Should I block GPTBot?

For most businesses, no. GPTBot collects content that may be used to train OpenAI's models, and allowing it means future models can learn about your business. Publishers whose content is their product may choose to block it. Blocking GPTBot does not remove you from ChatGPT search, which uses OAI-SearchBot.

Does blocking AI crawlers remove me from ChatGPT?

Blocking OAI-SearchBot does: OpenAI says sites that block it won't appear in ChatGPT search answers. Blocking only GPTBot keeps your content out of training but still lets ChatGPT search find and cite your pages.

Does Google-Extended affect AI Overviews?

No. Google-Extended controls whether your content is used for Gemini training and grounding. Google says it doesn't affect inclusion or ranking in Google Search. AI Overviews and AI Mode use Googlebot, which you should never block.

Do AI crawlers respect robots.txt?

The major AI companies, including OpenAI, Anthropic, Google and Perplexity, say their crawlers follow robots.txt. Fetchers triggered by a user's request may not, and some crawlers, such as Bytespider, are widely reported to ignore it. Use a firewall rule for bots that ignore robots.txt.

How do I know if Cloudflare is blocking AI crawlers?

Check AI Crawl Control and the bot settings in your Cloudflare dashboard, and look at your robots.txt for rules Cloudflare added. Since July 1, 2025, new Cloudflare domains block known AI crawlers by default. A request with an AI bot's user agent that returns 403 is another sign.

AI Crawlers robots.txt GPTBot ClaudeBot Technical SEO GEO AEO Cloudflare