Have it both ways: stay discoverable in search while disallowing AI training

Less than 1% of websites want to block search engines, but 17% want to block AI training bots. 👇 The catch is that major crawlers like Googlebot, Applebot, and Bingbot use a single bot for both jobs, making it impossible to block AI training without nuking your search SEO.

Mixed-use crawlers forced site owners into an all-or-nothing trade-off. Ad-supported and subscription sites rely on search engines for human traffic, but AI training bots scrape that same content to give answers without ever sending a visitor. Because tech giants bundled both functions into one crawler, opting out of AI training meant disappearing from search results entirely.

Cloudflare built a new "Disallow AI Training" control that splits search indexing from AI training, letting sites stay visible in search while refusing model training.

A static `robots.txt` file alone cannot solve this because it can't verify who is making a request or stop bots that ignore the file. Cloudflare fixes this at the network edge by classifying incoming bot traffic into three distinct behaviors: Search, Training, and Agent.

When you enable "Disallow AI Training", Cloudflare's edge automatically updates your site's `robots.txt` with specific opt-out directives like `Google-Extended` or `Applebot-Extended`.

The traffic then routes based on bot type: Incoming request → Edge behavior classification → Enforced preference.

Crawler operators meeting Cloudflare's "Accountable" criteria—including Apple, Google, and Microsoft—honor these directives, parsing your pages strictly for search indexing while skipping model training.

For dedicated training-only bots from companies like OpenAI, Anthropic, Meta, and Amazon, Cloudflare skips `robots.txt` entirely and hard-blocks the connection at the edge. Since those companies use separate crawlers for search, blocking their training bots leaves your search rankings intact.

Selecting the full "Block" setting instead causes Cloudflare to drop mixed-use bots completely at the edge, stopping both AI training and search indexing.

Site owners can now control how their content is used without managing individual user-agent strings or tanking search traffic. Apple, Google, and Microsoft have all confirmed that opting out of AI training through these settings will not hurt a site's search ranking.