Cloudflare Classifies AI Crawlers by Purpose
Era: LLM Era · 2020s–Present
On 1 July 2026, Cloudflare announced separate controls for Search, Agent, and Training traffic, and said that from 15 September new Cloudflare domains would block Training and Agent crawlers by default on ad-bearing pages while allowing Search by default; site owners can change the settings, and multi-purpose crawlers follow the strictest applicable rule.
Cloudflare
Sources and support
Your site, your rules: new AI traffic options for all customers
Cloudflare · Source type: Primary announcement
Documents Search, Agent, and Training controls and September 15 defaults.
Published: ·Accessed: