Jump to content

Archived

This topic is now archived and is closed to further replies.

gaspower

Stop Msn.bot indexing Site

Recommended Posts

Hello,

 

we have had our site up and running for many years, version prior to MS2.31. Just recently we are having a msn.bot indexing our site and it is absolutely crippling our site. The site almost comes to a complete holt, and we have to wait until the indexing is over. Is there anyway to prevent this from msn.bot? We have other search engines on the site, and no issues.

 

Thanks JR

Share this post


Link to post
Share on other sites

I've noticed they seem to be increasing their searches too. But if you stop them altogether, you would prevent your site from being listed by them, which is not good. You can add the following to your robots file and it should help. Change them as you wish, of course.

Crawl-delay: 2
Request-rate: 1/2
Visit-time: 0300-0900

Share this post


Link to post
Share on other sites

You should have a user agent statement. see http://www.robotstxt.org/ for examples and testing. The permissions don't matter but shouldn't be higher than 644.

Share this post


Link to post
Share on other sites

Hello,

 

Got it,

 

User-agent: msnbot

Crawl-delay: 20

Request-rate: 1/2

Visit-time: 0300-0900

 

Thanks JR

Share this post


Link to post
Share on other sites

×