Re: Robots.txt problem solving suggestion

Bits of agree and disagree. Some copyright owners don't realize that robots.txt or the archive exists and they should not be penalized heavily for their ignorance. Who can say they know all the RFCs p...

Posted by: David McQuillan