AIThe Registerabout 2 hours ago
Cloudflare to block cynical search-and-scrape bots from ad-supported web pages
3 min read
Cloudflare will block crawlers that combine search indexing with AI data harvesting from ad-supported customer sites by default. Apple, Google, and Microsoft's Bing operate such mixed-use crawlers. Each offers an AI opt-out directive that may allow them to bypass the block.
Level
Hype check
Tap to vote and see what everyone thinks.
In this storyCloudflare
