RoboticsDan Luuabout 11 years ago
The googlebot monopoly
4 min read
Bell Labs blocks all web crawlers except Googlebot, msnbot, and its own corporate bot in its robots.txt file. This prevents the Internet Archive and other search engines from indexing its pages. The practice makes it harder for competing search engines to operate and reduces web archival access.
Level
Hype check
Tap to vote and see what everyone thinks.