Robots.txt Abuses Defeating the Archive s Intent

We were considering contributing and promoting the Archive on educational sites we're developing, but as time goes on we are finding the robots.txt policy, though well intended, has way too often, bee...

Posted by: Phoenix Sandman