Robots.txt for AI Bots: Are You Blocking GPTBot, ClaudeBot, and PerplexityBot?
This is the single most common AI search visibility problem we find in an audit, and the easiest one to fix. A misconfigured robots.txt, often inherited from a default CMS template or a security plugin set to maximum caution, quietly blocks the exact crawlers AI search depends on. Here is which bots to check for, the syntax to look for, and the fix.
- A blocked AI bot is a complete, confirmed visibility killer for that platform, no amount of content quality compensates for it.
- GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, and CCBot are the names to check for specifically, not just a generic wildcard rule.
- The most common accident is a security-plugin default that disallows all bots except a named allowlist that predates AI crawlers.
- Always re-check the live, deployed robots.txt after a fix; a change committed to source control that never reaches production changes nothing.
Why this is the first thing to check, before anything else in this article
Every other AI search optimization technique, schema markup, content extractability, entity authority, depends on a crawler being able to reach your page in the first place. If GPTBot is disallowed in your robots.txt, ChatGPT’s browsing and retrieval features cannot access your site at all, regardless of how well-written or well-structured your content is. This is not a ranking penalty that content quality can offset; it is a hard access boundary that the crawler respects by design.
We list this check first in most audits because it is disproportionately common and disproportionately overlooked. Businesses invest in content, schema, and entity work, all real and valuable, while an inherited robots.txt quietly blocks the very crawler that work was meant to reach. Fixing it takes minutes; not knowing about it can cost months of invisible content.
The AI bots that matter
| User-Agent | Operator | Powers |
|---|---|---|
| GPTBot | OpenAI | ChatGPT training and, per OpenAI's documentation, browsing/retrieval features |
| ChatGPT-User | OpenAI | Real-time browsing triggered directly by a user's ChatGPT request |
| ClaudeBot | Anthropic | Claude's training data and web-connected features |
| PerplexityBot | Perplexity | Perplexity's search index and answer citations |
| Google-Extended | Gemini and AI Overviews' generative features, separate from standard Googlebot search indexing | |
| Applebot-Extended | Apple | Apple Intelligence features |
| CCBot | Common Crawl | The open dataset many AI labs, including smaller ones, train on |
Note the distinction between Googlebot and Google-Extended specifically: they are separate, independently controllable user-agents. Blocking Googlebot removes you from Google Search entirely; blocking Google-Extended only affects your content's use in Google's generative AI features, while your standard search ranking stays untouched. Confusing the two is a common and costly mistake.
How to check your current file
Open yourdomain.com/robots.txt directly in a browser. This is the raw file being served right now, not a preview, not a staging version, and not what your CMS settings page claims is configured. Read through it looking for two things: any line naming one of the bots above with a Disallow rule, and any blanket User-agent: * block with Disallow: / that would catch every bot not given its own specific exception.
The most common accidental block
The pattern we see most often isn’t a deliberate decision to block AI crawlers; it’s a security or performance plugin, or a hosting provider’s default configuration, written before AI crawlers existed as a category, that disallows all unrecognized bots by default and only allowlists a handful of legacy names.
A common accidental blanket block
User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: /
This file looks reasonable at a glance, it explicitly allows the two major search engine bots, but every AI crawler falls under the wildcard Disallow: / rule because none of them are named. GPTBot, ClaudeBot, and PerplexityBot are all silently blocked here, and nothing about the file signals that unless you know to check for each name specifically.
The fix
Corrected robots.txt, AI bots explicitly allowed
User-agent: * Disallow: / User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: /
- 01
Add an explicit allow rule for each AI bot you want indexed
A dedicated
User-agentline withAllow: /overrides the wildcard rule for that specific bot, regardless of where it sits in the file. - 02
Keep sensitive paths disallowed where needed
Admin panels, checkout flows, and internal search results pages should stay disallowed for every bot; the fix here is about your public content pages, not opening everything.
- 03
Deploy to the actual production server
Confirm your CMS or hosting setup serves this file at the true root path with no authentication or redirect in front of it.
Deliberate blocking is a valid choice, accidental blocking isn’t
Some publishers, particularly those whose business model depends on licensing content or on ad-supported pageviews, deliberately block AI training crawlers as a policy decision. That is a legitimate call, and it is a different situation from what this article addresses. The goal here is confirming that whatever your robots.txt currently does is the result of a decision, not an inherited default nobody has looked at since a plugin was installed. If you do choose to block certain bots deliberately, understand the tradeoff clearly: it removes you from that specific platform’s citations entirely, which is worth weighing against the broader shift covered in our AI search optimization guide.
Verifying after deployment
Reload yourdomain.com/robots.txt after publishing the change and confirm the live file matches what you intended, not just what you committed. CMS caching layers, CDN edge caches, and staging-to-production sync delays are common reasons a fix looks correct in the codebase but hasn’t taken effect on the version bots are reading. This same verification discipline applies to the other technical checks in our tracking and technical audit checklist: confirm on the live site, not the source file.
We include this check as a standard first step in the SEO and content authority service, before any content or schema work begins, precisely because everything downstream depends on it.
Frequently Asked Questions
Visit yourdomain.com/robots.txt directly in a browser and read the raw file. Look for the specific user-agent names GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, and CCBot, and check whether any of them are disallowed, either by name or by a blanket wildcard rule with no specific allow exception.
Blocking GPTBot prevents OpenAI's crawler from accessing your site for both training data collection and, per OpenAI's own documentation, the retrieval that powers ChatGPT's web browsing and search features. A blocked GPTBot means your content cannot be cited in ChatGPT's browsing-enabled answers, regardless of how well the content itself is written.
No. Googlebot is Google's standard crawler for search indexing, and blocking it removes you from Google Search entirely, a far more severe action. Google-Extended is a separate, specifically named user-agent that controls whether your content is used for Google's AI features including Gemini and AI Overviews' generative training and retrieval, independent of your standard search indexing.
Some publishers choose to block AI training crawlers on principle or licensing grounds while keeping search-oriented bots allowed, and that is a legitimate business decision. The point of this guide is to make sure any blocking is a deliberate choice, not an accident inherited from a default CMS configuration or a copy-pasted robots.txt template.
Ready to fix what’s costing you conversions?
We’ll review your acquisition funnel, show you what’s underperforming, and hand you a clear, prioritised plan, whether or not you choose to work with us.
Get a clear diagnosis of your acquisition performance.
- 30-minute review
- Free, with no obligation
- Clear findings and priorities
- You keep the findings
30 minutes · Free · No obligation · You keep the findings