AIHacker Newsabout 3 hours ago
AI answer bots evade site blocking rules
1 min read
A study of the top 10,000 sites' robots.txt files found 38% of GPTBot rules were written in late 2023. Most sites block training crawlers but ignore answer-time bots like OAI-SearchBot. Cloudflare announced a pay-per-fetch system for live AI answers. Only 144 sites explicitly allow answer-time bots.
Level
Hype check
Tap to vote and see what everyone thinks.
