richo3880 Posted July 28, 2011 Share Posted July 28, 2011 i just noticed that my oscommerce website has extremely high traffic, to a limit where it passes the bandwidth limit, it is now on 77GB. the traffic has been going up for the past 3 months. inside aw-stats it says 10GB are traffic viewed but the other 60some GB are not viewed traffic, and my FTP usage is just 58MB out of the deal. how to track what is going on here ? it seem like a security issue here... any advice is greatly appreciated. thank you Link to comment Share on other sites More sharing options...
richo3880 Posted July 29, 2011 Author Share Posted July 29, 2011 these were robots and spiders taking up all the bandwith. i did adjust the robot.txt file yesterday and in noticed less bandwidth usage since yesterday, hopefully this stay the same which means this was the issue here... Link to comment Share on other sites More sharing options...
Guest Posted July 29, 2011 Share Posted July 29, 2011 Richard, If you get a hosting account with unlimited bandwidth usage, you won't have to worry about keeping track of it. There are some that start as low as $4 a month. Chris Link to comment Share on other sites More sharing options...
Jack_mcs Posted July 29, 2011 Share Posted July 29, 2011 All hosting companies, that I've checked, that offer unlimited resources have strict guidelines in their TOS that limit the usage, so unlimited is usually not unlimited. But even if it was, extreme usage shouldn't be ignored because if the bandwidth is that high, it means the server is busy serving the data and that means the server may be busy when a real customer comes along. I suggest you update the spiders file first. If the usage is still high, then you should look in the server logs to find who is doing all of the accessing and ban their IP's. Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
richo3880 Posted July 29, 2011 Author Share Posted July 29, 2011 some times i get error 509 Bandwidth Limit Exceeded and the site stop working. so i had to track my bandwidth, since i'm not the one who purchased the plan. and tracking previous months from up to 3 years ago for the same site, the bandwidth usage was the highest around 13GB. but to jump to 75GB within the past 2 months is strange. and on awstats, i read the highest bandwidth usage for this month is for an "Unknown robot (identified by 'spider')" so i did update my robot.txt file for block "spider" hopefully this make it work. unless there is other file that i'm not aware of that i should adjust as well. thanks for the advice guys, you rock. Link to comment Share on other sites More sharing options...
Taipo Posted July 29, 2011 Share Posted July 29, 2011 In the stats are you finding that there is one or a few files that are being continuously requested or is it more of a general spidering by search servers trolling their way through all the files? - Stop Oscommerce hacks dead in their tracks with osC_Sec (see discussion here)- Another discussion about infected files ::here::- A discussion on file permissions ::here::- Site hacked? Should you upgrade or not, some thoughts ::here::- Fix the admin login bypass exploit here- Pareto Security: New security addon I am developing, a remake of osC_Sec in PHP 5 with a number of fixes- BTC:1LHiMXedmtyq4wcYLedk9i9gkk8A8Hk7qX Link to comment Share on other sites More sharing options...
richo3880 Posted July 29, 2011 Author Share Posted July 29, 2011 i have no idea... the pages request look normal as comparison to previous months. it's general crawling/spidering i guess. my stats said and i quote "Unknown robot (identified by 'spider')" under one line and one set of numbers. other lines are for google, yahoo... and other search engines here is an screen shot from what i read Link to comment Share on other sites More sharing options...
Dennisra Posted July 29, 2011 Share Posted July 29, 2011 Alter your robots.txt if you don't like it. Link to comment Share on other sites More sharing options...
Debs Posted July 29, 2011 Share Posted July 29, 2011 It wouldn't be the Yandex bot would it? A misbehaved, hyper-aggressive Russian spider that ignores rules. If left unchecked it can eat up to 5GB a day on some sites... There are a few ways to block it... here's one that should work in your root htaccess file: ########## start block SetEnvIfNoCase User-Agent "^Yandex*" bad_bot <Limit GET POST> Order Allow,Deny Allow from all Deny from env=bad_bot </Limit> ########## end block Link to comment Share on other sites More sharing options...
richo3880 Posted July 29, 2011 Author Share Posted July 29, 2011 Interesting Debs, we had previous encounters from Russia and eastern European ip's before. so it is not far fetched. ill check into this, and see what will happen. btw, i monitored my bandwidth all day today, and i noticed the difference that it hasn't exceeded 500MB all day yesterday after i changed the robot.txt file to block all spider agents. thanks for the insights. Link to comment Share on other sites More sharing options...
Jack_mcs Posted July 30, 2011 Share Posted July 30, 2011 Blocking all spiders will pretty much ruin your site as far as getting traffic from web searches. Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
richo3880 Posted July 30, 2011 Author Share Posted July 30, 2011 so how can i block specific ones? all i have on hand is "spider" ad unknown. Link to comment Share on other sites More sharing options...
Jack_mcs Posted July 30, 2011 Share Posted July 30, 2011 so how can i block specific ones? all i have on hand is "spider" ad unknown. As I already mentioned, first update the spiders file. Then locate the ip that is causing the problem in the logs. Look up the domain infor of the ip using one of the services on the web to get the ip range they use and block the whole range. Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
richo3880 Posted July 30, 2011 Author Share Posted July 30, 2011 i'll try this, and will let you know how it goes, thank you. Link to comment Share on other sites More sharing options...
richo3880 Posted August 2, 2011 Author Share Posted August 2, 2011 how can i find the bots and spiders ip address? Link to comment Share on other sites More sharing options...
Jack_mcs Posted August 2, 2011 Share Posted August 2, 2011 In your server logs. Explain to your host what you want to do and ask how you can see the logs. It varies with the server but they should be able to provide instructions, or the log. Support Links: For Hire: Contact me for anything you need help with for your shop: upgrading, hosting, repairs, code written, etc. Get the latest versions of my addons Recommended SEO Addons Link to comment Share on other sites More sharing options...
Taipo Posted August 2, 2011 Share Posted August 2, 2011 An analysis of your logs is the first place to look as mentioned by others. From there you should be able to see where the excessive bandwidth is being consumed. From there you can make a plan of attack but until then it is best to leave your options open as to how to fix this issue rather than zeroing in on what is at this stage still an assumption. - Stop Oscommerce hacks dead in their tracks with osC_Sec (see discussion here)- Another discussion about infected files ::here::- A discussion on file permissions ::here::- Site hacked? Should you upgrade or not, some thoughts ::here::- Fix the admin login bypass exploit here- Pareto Security: New security addon I am developing, a remake of osC_Sec in PHP 5 with a number of fixes- BTC:1LHiMXedmtyq4wcYLedk9i9gkk8A8Hk7qX Link to comment Share on other sites More sharing options...
richo3880 Posted August 3, 2011 Author Share Posted August 3, 2011 the robot.txt worked just fine. thanks all for your help and advice. Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.