Cloudflare will filter out web crawlers that serve AI companies

Cloudflare will block web crawlers that index sites for search engines while simultaneously training AI models starting September 15, 2026. New customers and existing subscribers default to allowing only search access but blocking agent use on pages containing ads unless they opt out or pay per crawl under the updated Pay Per Use feature.