robots.txt

In the FAQ there is written that a a exclusion written in a robots.txt file is applied also to the older versions. ("Internet Archive, does respect robots.txt instructions, and even does so retro...

Posted by: helohe