Blocking AI Crawlers: What Cloudflare's New Default Actually Changes

On September 15, 2026, Cloudflare changed the default settings applied to new domains on its network. Training crawlers and agent bots are now turned away on pages that display ads, while search crawlers remain allowed. Many companies sitting behind that network never had to make the call themselves, because it was made on their behalf. It still deserves to be understood, since the line between these families of bots partly determines whether you appear in answers from ChatGPT, Claude or Perplexity. What follows is a reading of that trade-off, bot by bot, based on the documentation each provider publishes.

Key takeaways

• Cloudflare's new default, in force since September 15, 2026, turns away training and agent bots on ad-bearing pages while letting search crawlers through.
• Every AI provider runs several distinct bots, one for model training and another for search, so a blanket block behaves very differently from a targeted one.
• Refusing OpenAI's training crawler does not remove you from ChatGPT's search feature, whereas refusing its search crawler does, according to OpenAI's own documentation.
• The gap between how much these bots crawl and how little traffic they return fuels the urge to close everything, yet Cloudflare itself says its published ratios are probably overstated.
• Configuration is worth verifying server-side, because a setting inherited from a host or a plugin often differs from what the company believes it decided.

What changed in Cloudflare's defaults on September 15, 2026?

Cloudflare now sorts automated traffic into three purposes, search, agent activity and training, and turns away the latter two by default on pages monetized with advertising. Search stays open with no action required from the site owner.

In its announcement post, the provider defines each purpose in a single sentence. Search covers "any behavior that collects or indexes your content, so it can answer questions about it later". Agent refers to "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now". Training means "a crawler taking your content to train or fine-tune a model". These controls are offered to every customer, including those on the free plan.

One technical detail deserves attention from whoever owns your infrastructure, because it touches ordinary search rankings. Cloudflare notes that multi-purpose crawlers such as Googlebot, Applebot and BingBot "will be blocked by customers who have selected to block Training". Defaults are enforced by the most restrictive applicable rule. A box ticked to guard against model training can therefore shut out the crawler that feeds your Google visibility. Site owners were able to record a different preference in their security settings before the change took effect.

Three families of bots, three different consequences

Every AI provider runs several crawlers, and their names resemble each other closely enough to be confused. The decision to block is never about "AI" in general, but about a specific user agent whose documentation states what it does and what refusing it costs.

OpenAI's documentation is the most explicit about what a refusal costs: sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. The same page presents GPTBot as the crawler that gathers content for training generative models. Two bots, two purposes, two separate decisions.

Anthropic follows the same logic in its help page for site owners. ClaudeBot is described as the bot collecting web content that could potentially contribute to training. Claude-SearchBot navigates the web to improve search result quality, while Claude-User reaches a site when an individual puts a question to Claude. Each can be refused separately in robots.txt.

Perplexity differs on one point worth knowing. Its documentation states that PerplexityBot exists to surface and link websites in search results, and that it is not used to crawl content for foundation models. The company recommends allowing it explicitly. Perplexity-User, by contrast, answers a user's request and generally ignores robots.txt rules, much as ChatGPT-User does on OpenAI's side.

Google's case calls for careful reading, because fear of a ranking penalty holds up many decisions. Google states that Google-Extended does not impact a site's inclusion in Google Search, nor is it used as a ranking signal. The token governs whether Gemini models may use your content, for training as well as for grounding. The documentation says nothing explicit about AI Overviews, which is a reason to treat that question as open rather than settled. We cover the token in more detail in our definition of Google-Extended.

Why the crawl imbalance makes people want to shut everything down

The case for blocking rests on a straightforward observation: AI crawlers read a great deal and send very little back. The relationship between the two can be measured, and the published figures are striking.

Cloudflare introduced that measurement, the crawl-to-refer ratio, in a post published on July 1, 2025. Over the June 19–26, 2025 window, it recorded roughly 70,900 crawl requests for every single referred visitor on Anthropic's side. The same measurement placed Mistral at 0.1 to 1, meaning ten times more referrals than crawl requests. The distance between those two extremes already shows there is no single pattern of AI crawler behavior.

The limit of the metric comes from Cloudflare itself, and it changes how the number should be read. The post explains that traffic referred by Claude's native app carries no referrer header, and that the same probably holds for other native apps. Referral counts therefore cover only the providers' web tools, which leads Cloudflare to write that the calculations "may overstate the respective ratios", without being able to say by how much. A ratio of that magnitude remains a signal of imbalance rather than an exact ledger of lost revenue.

The measurement sheds useful light on an economic argument, the cost of serving content against the traffic received in return. It sheds much less light on the question at hand, which is visibility inside the answers. A company selling software or a service is not primarily chasing visitors referred by ChatGPT. It wants to be named and cited in the answer, including when the reader clicks nothing at all. That kind of traffic largely escapes conventional analytics, as we showed when looking at AI traffic that hides in Google Analytics.

What are other sites actually blocking?

The observed trend is not a blanket block but a sorting exercise: refuse training, let search through. The available counts all point the same way, with amplitudes that vary by sample.

A census published by TechnologyChecker analyzed 4,223 robots.txt files drawn from the Cloudflare Radar API, in a snapshot dated July 27, 2026. GPTBot was disallowed by 696 domains and allowed by 299, slightly more than two refusals for every permission. OAI-SearchBot, the search crawler from the same provider, was disallowed by 250 domains and allowed by 266, near parity that contrasts sharply with how the training crawler fares.

That work comes with a methodological caveat worth holding on to. It is a census run by a content publisher against a public API, over a sample of a few thousand files. It is neither peer-reviewed research nor a Cloudflare publication. The proportions illustrate a direction of travel rather than measuring the state of the web. They do line up with the default Cloudflare chose for itself, which lends them some plausibility.

What we recommend for a company website

In most cases, a company website is better off refusing training and leaving search open. Training returns nothing measurable in the short term, whereas search determines whether you appear in the answers your prospects are reading today.

Two situations separate clearly enough to guide the decision. In the first, your content is your product, as it is for a news publisher or a research firm. Refusing training is fully defensible there, and negotiating a licensing agreement becomes a subject in its own right. In the second, your content exists to make an offering known that you sell elsewhere, which describes most small businesses and agencies. The trade-off then shifts toward search, and closing those crawlers means stepping out of the channel where your customers now ask their questions.

The robots.txt file remains where this preference is expressed, and a shared vocabulary has emerged for stating it. On September 24, 2025, Cloudflare published its Content Signals Policy, which defines three signals for the purpose. The search signal covers building a search index, ai-input covers feeding content into a model at answer time, and ai-train covers training or fine-tuning. The default in the provider's managed robots.txt is search=yes, ai-train=no, with ai-input deliberately left out because, as the company puts it, it does not know its customers' preference on that point and does not want to guess.

One clarification from the same post avoids an expensive illusion. Content signals "express preferences" and "are not technical countermeasures against scraping", and some companies may simply ignore them. A well-written robots.txt file therefore works as a statement of intent, while actual enforcement belongs to firewall rules or bot management. The same holds for the llms.txt file, whose adoption by the engines remains partial.

How to check what your configuration really lets through

A configuration is observed, not inferred from intent. Between a setting inherited from a host, a security plugin and a decision taken two years ago, the gap between what a company believes it allows and what it actually allows is common.

The first check takes seconds: open your robots.txt address and read the lines naming the user agents listed above. The second happens in your hosting dashboard, where Cloudflare now groups AI bots by purpose. The third runs through your access logs, which show which crawlers reach your pages and which receive a refusal. Our definition of AI crawlers lists the exact names to search for in those logs.

An automated audit does that work systematically across every page rather than the homepage alone. It is one of the things our AI visibility audit verifies, alongside data structure and content freshness. Citation tracking supplies the other half of the answer, showing whether your brand actually appears in engine responses after a configuration change.

What we would do in your position

Start by looking at your current configuration before deciding anything, because the September 15, 2026 change may have applied without anyone noticing. Then separate the two decisions that tend to get merged: what you accept for model training, and what you accept for search. The first rests on a judgment about value and business model, and no universal advice applies. The second is a more direct calculation, and for any company selling something other than its content, leaving search open remains the sensible choice.

Finally, verify the real effect of your settings rather than the intended one, by watching over a few weeks whether your pages keep appearing in engine answers. An over-broad configuration takes one line to correct, provided you know it is there.