Jump to content
  • Checkout
  • Login
  • Get in touch

osCommerce

The e-commerce.

Our store has less than 700 products. Why have spiders accessed it 10,000 times in two days?


rs2k

Recommended Posts

Our newest store went live about two days ago. We are using an XML sitemap and are not currently blocking anything in robots.txt. Could there be a duplicate content problem I'm not aware of? Could Google just be going over the pages more than once? Google has had between two and five spiders in this store most of the day. Yahoo and MSN have been coming and going all day.

 

I'm not complaining, but I don't understand it at all. One unnamed spider accessed all 700 products in about 5 minutes after reading the products site map. Google bot alone has nearly 8,000 hits in two days. What could have caused all of this spider activity so quickly. None of our other stores ever came close to this kind of spider activity so soon after the store opened. We have no back links and this is a brand new URL.

 

The only major difference with this store and our other stores is the meta content is a little better and there is an XML site map with no errors that we pinged to the major search engines.

 

Google had the home page indexed in two hours. I didn't even have time to fill in all of the header tags. (I pinged the site map a bit too early).

 

 

I can only hope we get a fraction of customer movement like this.

Link to comment
Share on other sites

  • 1 month later...
Our newest store went live about two days ago. We are using an XML sitemap and are not currently blocking anything in robots.txt. Could there be a duplicate content problem I'm not aware of? Could Google just be going over the pages more than once? Google has had between two and five spiders in this store most of the day. Yahoo and MSN have been coming and going all day.

 

I'm not complaining, but I don't understand it at all. One unnamed spider accessed all 700 products in about 5 minutes after reading the products site map. Google bot alone has nearly 8,000 hits in two days. What could have caused all of this spider activity so quickly. None of our other stores ever came close to this kind of spider activity so soon after the store opened. We have no back links and this is a brand new URL.

 

The only major difference with this store and our other stores is the meta content is a little better and there is an XML site map with no errors that we pinged to the major search engines.

 

Google had the home page indexed in two hours. I didn't even have time to fill in all of the header tags. (I pinged the site map a bit too early).

 

 

I can only hope we get a fraction of customer movement like this.

 

 

Wow indexed in two hours whats the secret?

Getting better with mods but no programmer am I.

Link to comment
Share on other sites

Archived

This topic is now archived and is closed to further replies.

×
×
  • Create New...