Re: Honoring present instead of past robots.txt is illogical

I agree. Currently if an old site is down (which may frequently be the case, since we're trying to access old data), it disallows us from it since it can't access its current robots.txt file. This k...

Posted by: rogerdpack