Why does the wayback machine pay attention to robots.txt

Honestly, half of the internet is being missed because some honest do-gooder decided that the garbage that is robots.txt should be followed by this archival service. This needs to stop. The wayback m...

Posted by: Fizscy