Re: Why does the wayback machine pay attention to robots.txt

I kind of do agree with you about leaving this open for discussion. Archiving a site is not a big deal, to be honest with you. It essentially is just taking a website and modifying its URL to be compl...

Posted by: DKL3