Cloudflare to Block Mixed‑Use AI Crawlers on Ad‑Supported Pages Starting Sep 2026

Cloudflare announced that, beginning 15 September 2026, it will block web crawlers that combine traditional search indexing with AI‑training or agent functions on pages that display advertisements. The default setting will apply to new Cloudflare customers, new sites created by existing customers and free‑plan accounts unless owners change the controls in the Cloudflare dashboard. Search‑only crawlers will remain allowed, while any bot whose declared purpose includes AI‑related activity will be denied access to ad‑supported pages.

The company said the move targets large search providers that use a single crawler for both web search and AI data collection, specifically naming Google’s dual‑purpose Googlebot. Cloudflare also introduced a "Pay Per Use" model to replace its previous "Pay Per Crawl" service, allowing publishers to receive payments when their content is used in AI‑generated answers; the pilot involves Ceramic.ai and You.com.