Re: Why does the wayback machine pay attention to robots.txt

I completely agree that the internet archive shouldn't follow robots.txt for several reasons. When a hostmaster adds a robots.txt, it blocks the whole site on the internet archive from being viewed, i...

Posted by: Hobbyboy