Cloudflare AI Bot Blocking May Disrupt Google and Bing Search Indexing

Cloudflare’s new AI Crawlers & Scrapers feature classifies certain bots as “search + training”. When the AI‑training block is enabled, legitimate search crawlers such as Googlebot and Bingbot receive HTTP 403 responses, preventing them from accessing sitemaps and indexing sites. The issue was first reported by members of the r/SEO Reddit community, who observed the 403 errors after enabling the block and saw them disappear when the setting was turned off.

Google’s John Mueller responded to the reports, offering to investigate the problem. Cloudflare’s official documentation states that, starting 15 September, the platform will block mixed‑use bots on pages with ads while continuing to allow pure search bots. Site owners are advised to review and adjust their Cloudflare settings before the change takes effect to avoid loss of visibility in Google and Bing search results.