Jump to content
mhsuffolk

Facebook Spider overloading site

Recommended Posts

My site is being flooded by Facebook spiders with IPs starting 69.171. I only sell my own products so this intrusion is unnecessary, Google etc list me high in their first pages.
Can I stop this as it exceeds the maximum number of mysql connections allowed by my host. But I need to still allow Facebook access from my posts advertising my products. Is this possible, do others have the same issue?

Capture-1.JPG.4eb25bbc64138edfb0a790cf0388d12e.JPG


Live Phoenix 1.0.5.0 on PHP 7.3

Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix 1.0.7.6) here.

Share this post


Link to post
Share on other sites

Is it possible to deny access to all pages (except product_info.php) for all IPs in the range 69.171.*.*

I'm not a htaccess expert, I expect it is possible but I do not know how.

 


Help shape the future of Phoenix; join the Phoenix Club

Share this post


Link to post
Share on other sites
3 hours ago, mhsuffolk said:

My site is being flooded by Facebook spiders with IPs starting 69.171. I only sell my own products so this intrusion is unnecessary, Google etc list me high in their first pages.
Can I stop this as it exceeds the maximum number of mysql connections allowed by my host. But I need to still allow Facebook access from my posts advertising my products. Is this possible, do others have the same issue?

I think you will need to block there range of IP's: deny from 69.171.224.0/19

As far as I know, those IP's are just for their bots so normal connections from FB shouldn't be affected. But check it yourself after blocking them and you can't connect, block each IP separately. It is possible to block a number of IP's from just the product page as @burt mentioned but the FB bot doesn't know what your product pages are so it will search the whole site and would still cause a traffic problem.

Edited by Jack_mcs

Share this post


Link to post
Share on other sites
1 hour ago, Jack_mcs said:

I think you will need to block there range of IP's: deny from 69.171.224.0/19

I assume it would make a difference...the IP in the screen shot is 69.171.251.

Dan

Share this post


Link to post
Share on other sites

No Facebook do not pay attention to any bot or crawler restrictions.

According to Facebook.....😂

" We don't have a crawler. We have a scraper that scrapes meta data on pages that have like buttons/are shared on FB. "

Got a problem then contact them 😊 I'm sure zuckerberg will be happy to explain it all..

"Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the web page or the embed tag of a video. Our system retrieves this information only after a user provides us with a link. You may have found this page because a Facebook user sent a link from your website to other Facebook users. If you have any questions or concerns about any links or content sent by one of our users, please contact us at legal@facebook.com."


 

Share this post


Link to post
Share on other sites

Thank you for all your replies.

I am done with Facebook spiders so have banned 69.171.251.0/24  and 69.171.224.0/24 . That has stopped them so far!

I am still getting click throughs from my Facebook posts so it is a win win for me.

Hey Facebook, just wind your neck in and stop being so intrusive.


Live Phoenix 1.0.5.0 on PHP 7.3

Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix 1.0.7.6) here.

Share this post


Link to post
Share on other sites

This is a bit off-topic...I'm sorry!...but I have a crawler from Huawei Clouds that is bombing my website with requests.

It originates in Hong Kong (Asia) - and all of the IP addresses begin with 159.138 but then the trailing numbers are all over the place

         
00:51:56 0 Guest 159.138.154.251      
00:17:49 0 Guest 159.138.158.249      
00:17:06 0 Guest 159.138.149.172      
00:38:12 0 Guest 159.138.150.182

If I put a block on 159.138.

Just like I typed - and that's it...

It kicks them out, but...

Am I potentially blocking other users from outside of Asia, too? 

- Andrea

Share this post


Link to post
Share on other sites
14 minutes ago, puggybelle said:

Am I potentially blocking other users from outside of Asia, too?

tracing the ip leads back to, may be they are just geting back at DT 😊...

image.png.43c8a32901e47392e2964cd8787431e6.png

Ip address details

104.109.101.63

New York City, New York, United States

 

Share this post


Link to post
Share on other sites
42 minutes ago, puggybelle said:

This is a bit off-topic...I'm sorry!...but I have a crawler from Huawei Clouds that is bombing my website with requests.

It originates in Hong Kong (Asia) - and all of the IP addresses begin with 159.138 but then the trailing numbers are all over the place

         
00:51:56 0 Guest 159.138.154.251      
00:17:49 0 Guest 159.138.158.249      
00:17:06 0 Guest 159.138.149.172      
00:38:12 0 Guest 159.138.150.182

If I put a block on 159.138.

Just like I typed - and that's it...

It kicks them out, but...

Am I potentially blocking other users from outside of Asia, too? 

- Andrea

Looks like all IPs in that range are located in Singapore, so no.

https://cleantalk.org/blacklists/as136907/159.138.0.0/16

Share this post


Link to post
Share on other sites
16 hours ago, puggybelle said:

If I put a block on 159.138.

Just like I typed - and that's it...

It kicks them out, but...

Am I potentially blocking other users from outside of Asia, too? 

Just typing in those numbers won't work at all if you are using the .htaccess file but I assume you didn't show the actual command. I always use CIDR notation so I'm not sure about just entering the numbers like that but if they are evaluating to 150.138.0.0/8, that would be millions of IP's. And since IP's are not assigned by country, it could mean you block some you don't want to block. Though the IP's are usually assigned by general area so the chances may be small.

Share this post


Link to post
Share on other sites
22 hours ago, Dan Cole said:

I assume it would make a difference...the IP in the screen shot is 69.171.251.

It appears incorrect but isn't. See the CIDR here.

Share this post


Link to post
Share on other sites
55 minutes ago, Jack_mcs said:

Just typing in those numbers won't work at all if you are using the .htaccess file

I entered that string of 159.138. in the IP Blocker page accessed via cPanel. 

1133376700_ipblocker.JPG.8c57a6eac5d10e4785f617ca7aa4cab0.JPG

I never added that to the .htaccess file itself and when I view that file, I only see full IPs that were placed there before, a few dozen or so.

That's why I wrote - I don't want to inadvertently block someone from New York or Canada or Spain when all I'm trying to do is kick out this crawler from Hong Kong.  When it shows up, it will have over 500 different IP addresses in use on my site at once, but they all begin with 159.138

Hopefully, everything will be alright.  What I entered in the IP blocker did take care of it.

I just hope I haven't blocked people from here, there and everywhere else.  I don't ship to Hong Kong, China, Japan, etc. (too expensive and no audience for what I sell there anyway) so...

Just trying to keep the site running smoothly!

- Andrea

Share this post


Link to post
Share on other sites

The Cpanel IP blocker is a very good tool. You can add an IP in a number of ways

IP address can be entered in the following formats.

  • Single IP address, E.g. 192.168.0.1
  • IP address Range, E.g. 192.168.0.1–192.168.0.100 or 192.168.0.1–100
  • CIDR format, E.g. 192.168.0.1/32
  • Using Wildcard, E.g. 192.168.*.*
  • Domain Name, google.com

 

 


 

Share this post


Link to post
Share on other sites
2 hours ago, puggybelle said:

I never added that to the .htaccess file itself and when I view that file, I only see full IPs that were placed there before, a few dozen or so.

That tool in cpanel does the block by adding the command to he roots .htaccess file so it's there, though it doesn't matter if you see it or not as long as it works.

 

2 hours ago, puggybelle said:

I just hope I haven't blocked people from here, there and everywhere else.  I don't ship to Hong Kong, China, Japan, etc. (too expensive and no audience for what I sell there anyway) so

The IP's are usually assigned in a geographically localized area so you should be OK. There should not be any crossover between the US and Asia but it's possible, though probably not very likely, with eastern Europe and Asia.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×