Internet vs Web: Layers Explained

Google indexes roughly 4% of the internet; the remaining 96% lies in unindexed depths. The internet is infrastructure plus protocols like IP, HTTP, and SMTP, while the web is a collection of hyperlinked HTML documents. Crawlers use robots.txt to respect access rules.