Re: Robots.txt Abuses Defeating the Archive s Intent

I totally agree that this is a problem when domains expire abd needs to be addressed before more and more of the archive is lost. I do not know if the archive deletes the pre robots.txt data or just. ..

Posted by: K9ine