Re: Why does the wayback machine pay attention to robots.txt

Its my understanding that the robot.txt file is to prevent search engines from crawling a website. Why the archive thinks it's a useful tool for site owners and the archives website is a mystery? It m...

Posted by: user001