Bot operators now have a home in the Cloudflare dashboard to manage submissions. This update adds submission status tracking, submission editing, and a behavior model so operators can accurately declare how their bots use content.
Bot submissions to Cloudflare have grown 7x since 2023, but until now every single entry was reviewed by hand while operators waited in a total black box. 👇
When bot operators submitted their crawlers to Cloudflare's directory, they had no way to track progress, see rejection reasons, or update details like changed IP endpoints without filling out the whole form again. Manual reviews couldn't scale with the surge in automated traffic, leaving legitimate bot operators stuck in support queues just to check their status.
Cloudflare built BotBase for Operators to fix this, adding full submission tracking, editable directory entries, a standardized three-part taxonomy, and an automated verification pipeline.
The new system replaces the old black-box workflow with real-time status tracking inside the Cloudflare dashboard. Operators can see whether a submission is waiting for review, accepted, or rejected. If an entry is rejected, the system gives specific, actionable reasons why; if Cloudflare adjusts a bot's classification during review, those changes are explicitly shown.
Instead of forcing every crawler into a single generic label, the intake form now requires a three-part declaration: - Behavior: What the bot actually does, such as search indexing, SEO analysis, or AI model training. - Content use: How the bot uses what it reads, matching declarations directly to site preferences like `Content-Signal: search=yes, ai-train=no`. - Operator type: Who runs the infrastructure, distinguishing between direct operators and intermediaries carrying third-party API traffic.
To clear the review bottleneck, Cloudflare replaced its manual rubric checks with an automated verification pipeline. When a submission comes in, automated systems check if the entry is a duplicate and ensure the user-agent pattern is specific enough to avoid overlapping other registered bots.
Next, the pipeline automatically validates the bot's identity. It fetches declared IP lists, verifies reverse DNS records, or checks Web Bot Auth signatures on the fly.
If all automated checks pass, the bot is tracked in the directory immediately. If a check flags an anomaly, the entry is routed to human reviewers with the specific failure pre-labeled, rather than landing as a blank entry in a queue. Operators can also update live endpoints or authentication methods anytime without re-submitting from scratch.
The new pipeline turns a manual bottleneck into an automated workflow, giving operators faster approvals, clear feedback on rejections, and a way to keep identity details current so site owners can safely allow their traffic.
NayaRead shows a summary and links to the original article. The full article is hosted on blog.cloudflare.com.