sk8r Posted December 29, 2010 Share Posted December 29, 2010 What is the most definitive way to find out if and IP address crawling my site should be blocked or not? I go to a "WhoIs" site, type in the IP address, I get this company......but now what??? How do I decide if it is a good bot or a bad bot? How do I know if it is a data collections center for a shopping comparison site, or a malicious bot stealing bandwidth? Has someone posted a "How to detect good and bad bots for Dummies" set of instructions somewhere? Link to comment Share on other sites More sharing options...
Guest Posted December 29, 2010 Share Posted December 29, 2010 Gill, Try one of these: Bad Bad Blocker Protect your site via htaccess SID Killer Denial of Service Controller Chris Link to comment Share on other sites More sharing options...
sk8r Posted December 29, 2010 Author Share Posted December 29, 2010 Thanks Chris. I knew someone out there could get me started in the right direction! Link to comment Share on other sites More sharing options...
Procommerce Posted January 3, 2011 Share Posted January 3, 2011 While you find out what will be your final decision, i suggest this simple but effective modification to the robots.txt file: User-agent: Google Disallow: User-agent: * Disallow: / Some of my clients also opted for blocking entire countries via htaccess... Hope this helps Providing Ecommerce & CRM Solutions since 1995 Vote my post up if you found it usefull Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.