Guest Posted May 24, 2004 Share Posted May 24, 2004 Hi all! I need your help! It seems that Googlebot and MSNbot are both getting caught in my site. They are downloading over 300MB a day and its costing me heaps of money! Can anyone help me figur out why they are getting stuck. I can't for the life of me figure it out. Please help!!!! In the mean time i have added a robots.txt to disallow goglebot and msnbot til the end of the month. Thanks Edwin! Link to comment Share on other sites More sharing options...
agiftcodotcom Posted May 24, 2004 Share Posted May 24, 2004 Do you have (in admin panel) configuration --> sessions --> prevent spider sessions set to true? Contributions I used : Updated 06-13-04 23:42 ---------------- Vote on My Graphis Poll Link to comment Share on other sites More sharing options...
Guest Posted May 24, 2004 Share Posted May 24, 2004 yeah i do... and i cant see any sessions being created by the spiders. Link to comment Share on other sites More sharing options...
agiftcodotcom Posted May 24, 2004 Share Posted May 24, 2004 Do a search and try using a search phrase of 'spiders getting stuck' or 'google getting stuck' or... you get the point, I think I've seen someone ask about this before but I don't honestly recall the answer at the moment. Contributions I used : Updated 06-13-04 23:42 ---------------- Vote on My Graphis Poll Link to comment Share on other sites More sharing options...
Guest Posted May 24, 2004 Share Posted May 24, 2004 Already did that but can find any relevant posts or a solution to this problem. Please can some one help! P.S. Thanks agiftcodotcom for your help so far. Link to comment Share on other sites More sharing options...
Guest Posted May 24, 2004 Share Posted May 24, 2004 Any one? Link to comment Share on other sites More sharing options...
Guest Posted May 24, 2004 Share Posted May 24, 2004 Im having a problem with my website and not sure if its to do with this too. Basically my website the last 2/3 days has been slow or I get an error "Cannot find server" this is when logging onto my main page or admin. When I do get on I see in my user tracking that an msn bot is searching through all my pages. It has no session but has been online for 23hours and is going through every page. msnbot64130.search.msn.com, IP: 65.54.164.130. Could this be the problem why my website is not loading?? If so how do I stop them from visiting or is there another way to fix it?? I have asked my host (doteasy) who are normally very good but have had no resonse. Many Thanks for any help. Link to comment Share on other sites More sharing options...
Guest Posted May 24, 2004 Share Posted May 24, 2004 It may be possible... sounds like the server is overloading.... Link to comment Share on other sites More sharing options...
scotte Posted May 25, 2004 Share Posted May 25, 2004 Ok your answer! Go to google and search for a script or hotscripts) that is php and can ban IP's. Ban the IP of the bot and he will go Link to comment Share on other sites More sharing options...
Drazan Posted May 25, 2004 Share Posted May 25, 2004 I know this is cross posting on sites, but I'm not going to copy all that info over here. It's a lot of info. On a side note I'm part of Gallery's dev people...and that is were I posted tons of info along with another moderator on that site. What you want to do is create a "robots.txt" this keeps most of the web bots from accessing pages & folders that they don't need to be accessing and allowing them access to your pages that you want to be indexed for ranking on search engines. This thread covers .htaccess Banning IP, Bad referer, Hotlinking, along with the Robot.txt http://gallery.menalto.com/index.php?name=...iewtopic&t=8160 =) Jade (I'm Jade Dragon on that site) Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.