Re: robots.txt only gives temporary removal of pages?

Is there a specific spider name that the Archive uses, there must be. One of our sites has databases that are not unique to the site that a few other sites use. Some of these databases are big as 1...

Posted by: YakYak