Cloudflare mandates AI crawlers to pay for publisher content

Cloudflare announced that, beginning 15 September 2026, its default settings will block “mixed‑use” crawlers that combine search, agent use and model‑training functions from accessing pages that host ads. The change applies to new Cloudflare customers, new sites added by existing customers and all existing free‑plan users, unless site owners adjust the settings themselves.

The move is intended to give publishers greater control over how their content is used by AI services and to create a revenue stream. Cloudflare is expanding its earlier “Pay Per Crawl” system into a “Pay Per Use” model, allowing publishers to be compensated when AI companies actually derive value from their content. Initial partners for the pilot are Ceramic.ai and You.com. "Now that the majority of traffic on the Internet is non‑human, we must go further and act faster so that a sustainable ecosystem can emerge," said Cloudflare co‑founder and CEO Matthew Prince.

Cloudflare specifically called out the world’s largest search engine – a reference to Google – noting that its bot ecosystem accesses roughly twice as much information as other AI firms. Google has responded in the past with tools such as Google‑Extended that let publishers opt out of AI training usage while remaining searchable. The new policy aims to separate traditional search crawling from AI‑specific activities, reducing bandwidth waste and giving publishers the choice between visibility and free content use.