GPTBot, ClaudeBot & PerplexityBot: AI Crawlers Explained
GPTBot, ClaudeBot, and PerplexityBot are the crawlers that let ChatGPT, Claude, and Perplexity cite your content. Block them and you get zero AI citations.
· 3 min read
GPTBot, ClaudeBot, and PerplexityBot are the web crawlers deployed by OpenAI, Anthropic, and Perplexity, respectively, to index web content. They follow the standard robots.txt protocol, and whether you let them in directly decides whether your content can be cited in answers from ChatGPT, Claude, and Perplexity.
Why it matters for AI visibility
The impact of blocking these crawlers is binary and measured: zero citations if the crawler is blocked, no matter how good your content is. This is criterion G23 in the LightSpot methodology, rated HIGH severity.
Each crawler is identified by its User-Agent in robots.txt:
- GPTBot (OpenAI): user-agent
GPTBot. Powers the real-time indexing that feeds ChatGPT when it has web access ("Browse with Bing" mode), and also feeds some of OpenAI's model training. - ClaudeBot (Anthropic): user-agent
ClaudeBot. Anthropic uses it for the web crawl that feeds Claude when it pulls from online sources. - PerplexityBot (Perplexity): user-agent
PerplexityBot. Perplexity uses it to index and refresh its web knowledge base in real time — Perplexity being an answer engine that runs almost entirely on real-time RAG.
The historical context matters. In 2023-2024, during the public debate over using web data to train models, many sites added Disallow: / directives for GPTBot or ClaudeBot to their robots.txt. In doing so, they also blocked indexing for real-time answers — an unintended side effect for most of them. Those directives are still in place on many sites, without their owners even realizing it.
One extra checkpoint: CCBot is the crawler for Common Crawl, a nonprofit that produces snapshots of the web used as training data by many language models. Blocking it doesn't affect real-time citations, but it can affect whether your content shows up in models trained on Common Crawl.
A concrete example
Here's a robots.txt snippet that blocks citations for ChatGPT and Claude but allows Perplexity:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
To allow every AI crawler across the whole site (the recommended setting if you want to be cited):
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: CCBot
Allow: /
Note: if no directive is set for a User-Agent in robots.txt, the default behavior is to allow access. So the problem only affects sites that added explicit Disallow directives — often copied from guides published back in 2023 during the model-training debate.
For the full picture, see the complete GEO guide.
FAQ
- What are GPTBot, ClaudeBot and PerplexityBot?
- They're the web crawlers deployed by OpenAI, Anthropic and Perplexity respectively to index web content. They follow the standard robots.txt protocol, and whether you let them in directly decides whether your content can be cited in answers from ChatGPT, Claude and Perplexity. Each is identified by its user-agent: GPTBot, ClaudeBot, PerplexityBot.
- What happens if I block these crawlers?
- The impact is binary: zero citations if the crawler is blocked, no matter how good your content is. No amount of structural or sourcing work makes up for a block in robots.txt. This is criterion G23 in the LightSpot methodology, rated HIGH severity.
- How do I check whether my site blocks AI crawlers?
- Look at the directives in your robots.txt file. If no directive is set for a given user-agent, the default behavior is to allow access. So the problem only affects sites that added an explicit Disallow directive for GPTBot, ClaudeBot or PerplexityBot.
- Why do so many sites block these crawlers without knowing it?
- It's a leftover from 2023-2024. During the public debate over using web data to train models, many sites added a Disallow: / for GPTBot or ClaudeBot, often copied from guides published at the time. In doing so, they also blocked indexing for real-time answers — an unintended side effect for most of them, and those directives are still in place today.
- Should I allow CCBot too?
- Yes, if you want the widest reach. CCBot is the crawler for Common Crawl, a nonprofit whose web snapshots serve as training data for many language models. Blocking it doesn't affect real-time citations, but it can affect whether your content shows up in models trained on Common Crawl. Allowing it is part of the recommended setup, alongside GPTBot, ClaudeBot and PerplexityBot.
Nicolas Meridjen
Fondateur de LightSpot.ai — outil d'audit de visibilité IA (46 critères SEO + GEO)
Je construis LightSpot.ai et j'analyse comment les moteurs de recherche IA (ChatGPT, Perplexity, Google AI Overviews) choisissent les sources qu'ils citent. J'écris sur le GEO et le SEO à partir de données d'audit réelles.
On the same topic
AI Citation: What It Is and How to Get Cited by AI Engines
An AI citation is when a generative engine — ChatGPT, Perplexity, AI Overviews — picks your content as a source in its answer and links back to your page.
Google AI Overviews: What They Are and How to Get Cited
AI Overviews is Google's AI-generated answer at the top of search results. It cites two to four sources picked for how extractable and trustworthy they are.
What Is an Answer Engine? Definition & Examples
An answer engine generates a synthesized answer with cited sources instead of a list of links. Key examples: Perplexity, ChatGPT web, Google AI Overviews.