Re: Why does the wayback machine pay attention to robots.txt

I think it would make sense to show data from squatted domains, even if the current owner forbids it via robots.txt. Anyway, robots.txt was meant to prevent crawlers from visiting a site, but we're ta...

Posted by: PiRSquared17