Cloudflare's new Bot Preference Sync automatically aligns your robots.txt file with your AI bot policies for Search, Agent, and Training. Easily manage which bots access your content without maintaining static files.
When your `robots.txt` file disallows a crawler but your firewall doesn't enforce a block, AI crawlers treat the mismatch as an excuse to ignore your rules entirely. 👇
Maintaining two separate layers of protection is a constant headache. Site owners write static rules in `robots.txt` to state their preferences, but if those preferences don't match what the edge firewall actually blocks, crawlers use the disagreement to justify bypassing your rules.
Cloudflare built Bot Preference Sync to fix this gap across all plan tiers from Free to Enterprise. It automatically generates and prepends matching `robots.txt` directives based on the AI bot policies set in your dashboard.
Instead of manually editing a static file, you pick policy rules in the dashboard for three distinct traffic categories: Search, Agent, and Training. Search and Agent categories can be set to Allow, Block on pages with ads, or Block everywhere, while Training can be set to Disallow.
When you adjust a policy, Cloudflare pulls updated crawler identifiers from its BotBase database and prepends the necessary `User-agent` and `Disallow` rules directly into your `robots.txt`. Your site's existing text file remains untouched below the new header, preserving custom rules.
The system also addresses mixed-use crawlers that run both search indexing and model training under a single user agent. If you set Training to Disallow, Cloudflare allows mixed-use bots to crawl for search indexing only if they meet strict transparency rules. They must honor `robots.txt` preferences, offer page-level reporting on training usage, and prove that disallowing training doesn't penalize search rankings. Crawlers that refuse to meet these standards are blocked at the edge.
The result is a direct alignment: Dashboard policy → Dynamic `robots.txt` → Edge enforcement. Site owners with ad-supported content can also pick a publisher default during onboarding, keeping their pages in search results while shutting out AI model training by default.
NayaRead shows a summary and links to the original article. The full article is hosted on blog.cloudflare.com.