Cloudflare Classifies AI Crawlers by Purpose
Era: The LLM Era · 2020s–Present
On 1 July 2026, Cloudflare announced separate controls for Search, Agent, and Training traffic. From 15 September, new Cloudflare domains will block Training and Agent crawlers by default on ad-bearing pages while allowing Search by default; site owners can change the settings, and multi-purpose crawlers follow the strictest applicable rule.
Cloudflare
Sources and support
Your site, your rules: new AI traffic options for all customers
Cloudflare · Source type: primary-announcement
Documents Search, Agent, and Training controls and September 15 defaults.
Published: ·Accessed: