server logs robots.txt file

Almost two months ago robots.txt file was placed on the server allowing any robot to crawl our website. After checking the server logs it shows the following: The web archive (IA Archiver) June Hits:...

Posted by: pista73.com