What Is Google-Extended? The Robots.txt AI Token

Google-Extended is a User-Agent token, separate from Googlebot, that Google introduced in 2023. It lets site owners independently control whether Google can use their content to train and ground its generative AI systems — chiefly Gemini and the Vertex AI services — without touching the classic SEO crawl that Googlebot handles.

Why Google-Extended matters for AI visibility

The split between Googlebot and Google-Extended is one of the most consequential calls a site can make about how it shows up in Google's AI products.

Blocking Googlebot in robots.txt drops the site out of Google's classic index — a drastic move that hits SEO visibility across the board. Blocking Google-Extended is far more surgical: it keeps your classic SEO ranking intact while removing your content from the AI uses Google has declared for this token (Gemini training, Vertex AI grounding).

Google documented Google-Extended as a way for content publishers to manage their relationship with Google's AI products separately from classic search. Google's systems have recognized and respected the token since it launched.

Whether you block or allow Google-Extended comes down to strategy:

• Allow (or specify nothing): your content stays available to the Google systems that use this token for training and generative grounding. This is the default behavior.
• Block: your content stays indexed and ranked normally in classic SEO, but it's off-limits to the AI uses documented for this token. Some publishers — news outlets, agencies — make this call for copyright or editorial-policy reasons.

One thing worth keeping in mind: the relationship between Google-Extended and AI Overviews isn't clearly or publicly documented by Google. AI Overviews may draw on the existing index that Googlebot builds, on real-time grounding, or on a mix of both. Blocking Google-Extended doesn't necessarily mean you're shut out of AI Overviews, and allowing it doesn't guarantee you'll be cited there. It's wise not to treat those two effects as the same thing.

A concrete example

An online newspaper wants its articles to stay reachable by Google for SEO ranking, but doesn't want its content used to train the Gemini models. Here's the matching robots.txt setup:

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

This setup lets Googlebot crawl and index every page for organic ranking, while blocking Google-Extended for AI uses.

On the flip side, a site that wants to make sure its content stays available to Google's AI systems leaves Google-Extended allowed (or simply absent from any Disallow directive), alongside an extractable content structure and well-formed JSON-LD data.

For the full picture, see the complete GEO guide.