There are numerous reasons as to why or when you should control the access of the web robots or web crawlers to your site. As much as you want Googlebot to come to you site, you don't want the spam bots to come and collect private information from your site. Not to mention that when a robot crawls … [Read more...] about How to Control Access of the Web Crawlers or Web Robots to Your Site