Jump to content

Archived

This topic is now archived and is closed to further replies.

GLWalker

IP Banishment

Recommended Posts

valid bots do not 'disobay' the robots.txt file. it could be some way you have it setup or could be it is not a bot at all.

Yes I know the valid bots will honor the robots.txt. Thats why I want to destroy the ones that drift from what the file tells them.


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

I had a "trap" set for bad bots on a website I used to run.

 

It's really simple.

 

All you do is put a folder in your robots.txt that isn't referenced by any code on your site.

 

Like:

 

Disallow: /secret/

Then in this "secret" folder, put an index.php that records the visitors IP address to a file. Any visitor to this folder has to be a "bad bot" since the only place it can be found is in your robots.txt file.

 

Then you have a small snippet of code at the top of your regular index.php that first opens this file of banned IP's and looks thru them when a visitor comes to your site.

 

If the visitors IP address matches any found in this banned IP file, they get directed to a page (of your choosing).

 

That's how it was setup.

 

I did one alteration to it. I coded it to send me an email when an IP address got banned (the email had the IP address in it) so I could check it out.

 

You might want to alter this a bit (if possible) to have it fit within an osC store.

 

The site I had it on was a small, non-commercial, low traffic site.

 

If you want, I can post the code I used.

 

And, just to set the record straight, I didn't author the code nor "invent" the method. I found it somewhere on the net five or six years ago.

:)


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

OK.

 

Make a folder on your site to trap the bad bots. For demonstration purposes, we'll call this folder "secret".

 

Then disallow this folder from good robots in your robots.txt file:

 

Disallow: /secret/

I'd put it at the top of the "disallow list". It was my experience that the bad bots hit the "disallowed" things first thing as soon as they crawl your site.

 

In this /secret folder, make an index.php file and put the following code in it:

 

<?php
$ip = "$REMOTE_ADDR"."\n";

// the next line is the path to the file that stores the banned IP addresses - alter as required
$sandtrap = file('/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt');
sort($sandtrap);
reset($sandtrap);

$found = 0;
foreach( $sandtrap as $blockip ) {
 $tester = strcmp($blockip,$ip);
 if ( $tester == 0 ) {
$found = 1;
break;
 }
 if ( $tester > 0 ) {
array_push($sandtrap,$ip);
sort($sandtrap);
reset($sandtrap);
break;
 }
}

if ( $found == 0 ) {
// path to your banned IP file - alter as required
 $fp = fopen("/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt","w");
 if ( $fp != 0 ) {
foreach( $sandtrap as $blockip ) {
  fputs($fp,"$blockip");
}
fclose($fp);
 }
}
$ua = $HTTP_SERVER_VARS['HTTP_USER_AGENT'];
$ip = $REMOTE_ADDR;
$todaysdate = date("m/d/Y h:i:s a",time()); 
// the next line mails you when an IP gets banned - alter to suit
mail("you@youremail.com", "IP Banned $todaysdate", "$ip ($ua) has been banned.\n\n","From: Admin@yoursite.com");
// the next line is the page to redirect them to after they get banned - alter to suit
header ("Location: http://www.yoursite.com/blocked.php");
?>
<HTML>
<HEAD>
 <meta name="robots" content="noindex,nofollow">
 <TITLE>Gotcha!</TITLE>
 <META NAME="generator" CONTENT="NoteTab Light 4.86c">
 <META NAME="author" CONTENT="germ">
 <META NAME="description" CONTENT="">
 <META NAME="keywords" CONTENT="">
</HEAD>
<BODY>
<center>
<h1>Gotcha!</h1>
</center>
</BODY>
</HTML>

At the top of your sites main index.php, incorporate this code:

 

<?php
// path to your banned IP file - alter as required
$sandtrap = file('/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt');
$ua = $HTTP_SERVER_VARS['HTTP_USER_AGENT'];
$ip = "$REMOTE_ADDR"."\n";

$punish = 0;

// this sets up code to ban them if they don't supply a user agent
if ( $ua == "" ) {
 $punish = 2;
}

// I found this bot using several IP's, so I'm banning him by name
if (stristr($ua,"Fetch API Request")!="") {
$punish = 1;
}

foreach( $sandtrap as $blockip ) {
 $tester = strcmp($blockip,$ip);
 if ( $tester < 0 ) {
continue;
 }
 if ( $tester == 0 ) {
$punish = 1;
break;
 }
 if ( $tester > 0 ) {
break;
 }
}

if ( $punish != 0 ) {
// the next line is the page to redirect them to - alter to suit
 header ("Location: http://www.yoursite.com/blocked.php");
 exit;
}
?>

In the time I had this trap set, this is the list of banned IP's I had:

 

128.194.135.80

193.255.207.253

202.108.249.184

202.108.249.185

202.108.249.186

202.108.249.187

202.108.249.188

202.108.249.189

202.108.250.242

202.109.97.248

205.162.205.66

205.162.205.71

205.162.205.72

210.192.122.68

211.152.14.95

211.152.14.98

211.157.8.41

211.157.8.46

212.98.46.104

213.190.195.100

213.190.195.105

220.240.211.195

24.129.175.231

24.53.80.56

61.145.129.73

61.88.171.6

62.65.142.213

63.148.99.229

63.148.99.232

63.148.99.234

63.148.99.239

63.148.99.240

63.148.99.244

63.148.99.250

63.148.99.253

63.238.200.181

64.207.46.75

65.68.201.190

65.75.135.40

68.108.120.32

68.166.233.128

68.52.209.17

68.52.72.202

69.28.197.242

70.84.132.74

70.84.196.98

80.129.110.3

81.216.194.197

Of course you should backup your main index.php before making any edits to it.

:thumbsup:


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

I found the link to the original site I saw this at:

 

linux.oldcrank.com: Defending Against Email Harvesters, Leechers, and Web Beacons

 

Looking at it now, I did make some mod's to make it work faster.

 

My code sorts the IP's alphabetically to make it work quicker to know when to stop looking when adding new IP's or checking for blocked IP's.

 

One thing I don't remember.

 

If the script "chokes" when you fire your site up after installing it, you might have to make the banned IP file. Don't put anything into it, just create a blank file for it to find.

 

And you might have to give the banned IP file "777" permissions. That will just depend on your server config I guess.

 

In fact, I think you should create a blank banned IP file right off the bat.


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

Man oh Man! Im stoked about this. It works F%^&@N Awesome!

Heres what I did:

I made a folder called /welcome/

 

I made the index.php and placed inside it with my server/site info/email address modified.

 

Then I made a page called blocked.php with a nice message, and placed it in my shops root folder.

 

Then I took the block of code that was to go into my sites index.php and made it into a new file called welcome.php, and saved it to the includes/modules/ directory

then in application top a few lines down I placed a require(DIR_WS_MODULES . 'welcome.php');

 

Then I made a bannedisps.txt file and placed it in the includes directory.

 

It works flawlesly. It emailed me everytime I went to the welcome/ directory, and blocked every page I tried to go to in the shop.

 

I wonder if there is a way to get it to write the IP Adress to the bannedisps.txt file automatically?

 

Thanks Germ!


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

Hey there .. caught this post and thought i`d post some input.

 

I made a custom admin page for banning ip's. Can add and remove ip's ... which are written to the FS_BLOCKED_IPS_FILE congiration table value.

I just have it called blocked.txt

 

Anyway here is the code which actually writes the ip to the text file

 

 

require('includes/application_top.php');

 

// Get updated vars from form

if ($_POST['ip']):

$ip = $_POST['ip'];

$rows = count($ip);

$del_ip = $_POST['del_ip'];

$date_blocked = $_POST['date_blocked'];

 

// update blocked.txt

if (!$fp_out = fopen(DIR_FS_CATALOG . FS_BLOCKED_IPS_FILE, 'w')):

$msg_stack = "Cannot open file : " . DIR_FS_CATALOG . FS_BLOCKED_IPS_FILE;

exit;

endif;

$blocked_ip = array();

for ($x=0; $x<$rows; $x++) {

if ($ip[$x] == '' || $del_ip[$x] == 'yes') :

$blocked_ip[$x] = '';

else:

$blocked_ip[$x] = $ip[$x] . ',' . $date_blocked[$x] . "\n";

fwrite($fp_out, $blocked_ip[$x]);

endif;

}

fclose($fp_out);

 

$messageStack->add_session(TEXT_BLOCKED_IPS_FILE_UPDATED, 'success');

endif;

 

clearstatcache();

$longnow = date("Y-n-d H:i:s", mktime());

$shortnow = date("Y-n-d", mktime());

 

 

-------------------------------------

 

Now this is almost the exact variables you guys have been using and I`m sure would not be hard to modify to work with this forum topic.

 

Ideally I have been trying to get time to do it by using a banned_ip database table ... insted of using a .txt file. This way it would be more uniform with other oscommerce admin pages and osc code.

 

I will try to follow up on this if I get some time. Hope the code I posted gives you some ideas ... I think it would just have to be added somewhere in germ's original code within this section after ..... if ( $punish != 0 ) {

 

if ( $punish != 0 ) {

<<< ip banning code here >>>

break;

}

Share this post


Link to post
Share on other sites
I wonder if there is a way to get it to write the IP Adress to the bannedisps.txt file automatically?

 

Thanks Germ!

:thumbsup:

 

Glad it works for you G.L.!

 

What do you mean by "write the IP Adress to the bannedisps.txt file automatically"?

:unsure:


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites
:thumbsup:

 

Glad it works for you G.L.!

 

What do you mean by "write the IP Adress to the bannedisps.txt file automatically"?

:unsure:

I just saw your other post about creating a blank bannedips file.

What I did last night was create the bannedisps.txt file and set it to 777 to see if when I went into the forbiden folder the file would be wrote to with my IP. It didnt write to it though.

Is it supposed too?


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

It can't ban you if it doesn't write the IP to the file.

 

So the answer is, "Yes", it's supposed to write IP's to the file.

 

Something's amiss...

:huh:


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites
It can't ban you if it doesn't write the IP to the file.

 

So the answer is, "Yes", it's supposed to write IP's to the file.

 

Something's amiss...

:huh:

I'll double check things and post my results.


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

Ok heres what I came up with. I changed a small block of code in the index.php file for the /secret/ folder. So it now looks like this:

<?php
$ip = "$REMOTE_ADDR"."\n";

// the next line is the path to the file that stores the banned IP addresses - alter as required
$sandtrap = file('/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt');
sort($sandtrap);
reset($sandtrap);

$found = 0;
foreach( $sandtrap as $blockip ) {
 $tester = strcmp($blockip,$ip);
 if ( $tester == 0 ) {
$found = 1;
break;
 }
 if ( $tester > 0 ) {
array_push($sandtrap,$ip);
sort($sandtrap);
reset($sandtrap);
break;
 }
}

if ( $found == 0 ) {
  $ip = "$REMOTE_ADDR\n";
// path to your banned IP file - alter as required
 $fp = fopen("/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt","w");
$fp = fopen($banip, "a");
 $write = fputs($fp, $ip);
fclose($fp); 
}
$ua = $HTTP_SERVER_VARS['HTTP_USER_AGENT'];
$ip = $REMOTE_ADDR;
$todaysdate = date("m/d/Y h:i:s a",time()); 
// the next line mails you when an IP gets banned - alter to suit
mail("you@youremail.com", "IP Banned $todaysdate", "$ip ($ua) has been banned.\n\n","From: Admin@yoursite.com");
// the next line is the page to redirect them to after they get banned - alter to suit
//header ("Location: http://www.yoursite.com/blocked.php");
?>
<HTML>
<HEAD>
 <meta name="robots" content="noindex,nofollow">
 <TITLE>Gotcha!</TITLE>
 <META NAME="generator" CONTENT="NoteTab Light 4.86c">
 <META NAME="author" CONTENT="germ">
 <META NAME="description" CONTENT="">
 <META NAME="keywords" CONTENT="">
</HEAD>
<BODY>
<center>
<h1>Gotcha!</h1>
</center>
</BODY>
</HTML>

 

It works like a champ. Now how do I get to my site? :'(

 

So would it be safe to use on the images directory too? Maybe the tmp and cache and includes? I just wonder if the site itself would get blocked if I placeit in the wrong directory?


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites
Now how do I get to my site? :'(

That's the index.php that goes in the /secret folder.

 

Be sure you modify the lines that need modified.

 

Then back in a previous post, there is code you need to apply to the index.php in your root folder.

 

There are lines in it that need modified too.

 

You don't "use it on a directory".

 

You put the /secret folder in your robots.txt file as "disallowed".

 

You put the special index.php in the /secret folder.

 

Then you add the code to your main index.php file.


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites
That's the index.php that goes in the /secret folder.

I was kidding there. Its all running as it should. Thanks a ton for your help.

 

I placed the code that is supposed to go in my main sites index.php file into a new file in the catalog/includes/modules/folder, and then placed a call to it in aplication_top.php . Now every page on the site will redirect to the banned.php file.

This is really nice to have. Before our server upgrade to php5, I had a few sites get hit with a nasty bit of script that added a ton of php and .htaccess files to the images folder and any other that had full permissions. It looks like with php5 I'm able to set those folders to a lower permission and still have them written too from within the site. Ive added a bit of code to the .htaccess file to block known hackers/ spammers/ harvesters, but just want more security on different levels.

Do you think it would interfere with any datafeeds that go out from the site, IE Froogle, if I placed the secret/index.php into the images folder as well?


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

jim, this is some cool stuff.

 

the only thing i'd like to see added is that once a 'bad bot' is detected, you send it a bunch of garbage. it's obviously looking for something, so sending it gigabytes of random characters and badly formatted xml and stuff would give it something to chew on. or am i just being mean? :)

 

okay, maybe not a good idea as it'd start using up your bandwidth and server resources. but it'd be fun to try. <evil laugh>

Share this post


Link to post
Share on other sites

G.L.,

 

I think only one place on your site is enough. Any more and you'll end up banning legit (but curious) people, IMHO.

 

Put your "secret" folder at the top of your "disallow" list in your robots.tx file. The bad bots will hit that first, and be shutdowm immediately. There are other ways to keep "prying eyes" out of other folders without banning them. If you want suggestions on that, just ask.

 

Dave,

 

You naughty boy!

 

I like the way you think!!!!

:thumbsup:


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

So I changed the index.php in the secret folder again,

this time I removed the bottom HTML and added code to generate thousands of bogus links, dont know how well they will tangle up the bot though, becuase they appear to redirect to the sites root, but then they get redirected to the banned.php page. Do bots move back?

 

<?php
$ip = "$REMOTE_ADDR"."\n";

// the next line is the path to the file that stores the banned IP addresses - alter as required
$sandtrap = file('/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt');
sort($sandtrap);
reset($sandtrap);

$found = 0;
foreach( $sandtrap as $blockip ) {
 $tester = strcmp($blockip,$ip);
 if ( $tester == 0 ) {
$found = 1;
break;
 }
 if ( $tester > 0 ) {
array_push($sandtrap,$ip);
sort($sandtrap);
reset($sandtrap);
break;
 }
}

if ( $found == 0 ) {
  $ip = "$REMOTE_ADDR\n";
// path to your banned IP file - alter as required
 $fp = fopen("/home/httpd/vhosts/yoursite.com/httpdocs/bannedips.txt","w");
$fp = fopen($banip, "a");
 $write = fputs($fp, $ip);
fclose($fp); 
}
$ua = $HTTP_SERVER_VARS['HTTP_USER_AGENT'];
$ip = $REMOTE_ADDR;
$todaysdate = date("m/d/Y h:i:s a",time()); 
// the next line mails you when an IP gets banned - alter to suit
mail("you@youremail.com", "IP Banned $todaysdate", "$ip ($ua) has been banned.\n\n","From: Admin@yoursite.com");
// the next line is the page to redirect them to after they get banned - alter to suit
//header ("Location: http://www.yoursite.com/blocked.php");
$page = ''; 
for ( $i = 0; $i < 5000; $i++ ) 
{ 
$page .= new_link(); 
} 
function new_link() 
{ 
$link = ''; 
$letters_array = array('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 
's', 't', 'u', 'v', 'w', 'x', 'y', 'z'); 
for ( $i = 0; $i < 17; $i++ ) 
{ 
$link .= ( $i!== 10 )? $letters_array[ mt_rand( 0, 25) ] : ''; 
} 
$link .= '.com'; 
$link = '<a href="http:/">' . $link . "</a>\n"; 
return $link; 
} 
$page .= "Gotcha!"; 
echo $page; 
?>


Follow the community build:

BS3 to osCommerce Responsive from the Get Go!

Check out the new construction:

Admin Gone to Total BS!

Share this post


Link to post
Share on other sites

I don't think they "move back".

 

I have had them try to get "in" several times in succession, though....

 

If you don't mind using the bandwith (as Dave pointed out), bogus links (or I used to have a PERL script that fed them bogus email addy's) are a nifty idea!

;)


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

I've been playing with this this morning - I can't get it to work.

 

It seems the problem is with the code writing the ip to the txt file. The file in the secret directory does not appear to write the ip for me. If I place my IP in the txt file manually and then go to the page with the checking in it, it redirects to the blocked page. So that seems to confirm I have the path to the txt file right.

 

I've tried setting the txt file to 777 permissions - no difference.

 

Bit puzzled! Any ideas?

Share this post


Link to post
Share on other sites

Fixed it myself.

 

I changed this in the personal/index.php:

 

if ( $tester > 0 )

 

to:

 

if ( $tester != 0 )

 

and now it writes the ip (the strcom function can give a positive or negative value if the strings to compare aren't equal)

Share this post


Link to post
Share on other sites

You misunderstand the logic:

 

$found = 0;
foreach( $sandtrap as $blockip ) {
 $tester = strcmp($blockip,$ip);
 if ( $tester == 0 ) {
$found = 1;
break;
 }
 if ( $tester > 0 ) {
array_push($sandtrap,$ip);
sort($sandtrap);
reset($sandtrap);
break;
 }
}

The IP's are sorted alphabetically.

 

You're comparing the stored IP's to the IP of whomever just fell into the trap.

 

So, in the loop, if the compare = 0, the IP address was found, you set the flag (the flag keeps from adding duplicate IP's) and break from the loop.

 

As soon as the compare > 0 (the stored IP address is greater than the one you're checking), you know the IP address wasn't found so you add it and break from the loop.

 

You really don't give a hoot when the compare < 0.

 

The idea there is to speed things up. As long as they're sorted alphabetically, it works and you don't have to do the compare on every banned IP address.

 

Why it wasn't workingfor you, I haven't a clue.

 

I copied the code directly from a web page I had it on where it worked flawlessly for several years.


If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

 

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."

- Me -

 

"Headers already sent" - The definitive help

 

"Cannot redeclare ..." - How to find/fix it

 

SSL Implementation Help

 

Like this post? "Like" it again over there >

Share this post


Link to post
Share on other sites

My bannedip.txt file has some example ip addresses in it. However, my ip is higher than those.

 

Does your code work in such a situation? I don't think it does as the ip will never be pushed into the array. Thats why my bodge sort of fixed it (at the expense of speed). Or does '$tester > 0' report true when it compares it against the last empty line in the file?

 

I've added 999.999.999.999 at the end of the ip file and your original code now blocks me :)

 

(of course, I could be totally wrong, I'm still learning php)

Share this post


Link to post
Share on other sites

×