Re: Why does the wayback machine pay attention to robots.txt

Dear Fizscy, "Honestly, half of the internet is being missed because some honest do-gooder decided that the garbage that is robots.txt should be followed by this archival service. This needs to stop...

Posted by: TechLord