I observe the actions of bots (Google, Yahoo v.v) on my web (with "Who online" contribution). I see that, my site know Google bot is a bot but with yahoo, she only know he is a real customer. So, when Yahoo bot view 1 link (maybe a categorie or a product ..etc.) , after that, he will reach the link /cookie_usage.php and be stoped here.
How can i do to solve this problem ? I think if i can solve the /cookie_usage.php problem, yahoo bot can view my site continuously. That mean, my site will be show on the search engine easyer.
Latest News: (loading..)
0
2 replies to this topic
#3
Posted 02 December 2006, 02:22
I believe I am giving you the correct info, I am sure if I am not someone will be along to correct me.
You need to update your spider.txt file. I believe there is a contrib with an up to date list.
The file is in /includes/spiders.txt
You should also set (in your admin) prevent spider sessions to true.
If you dont want your cookie usage (or any other page) to be indexed you should disallow it in your robot.txt file. (Disallow: /catalog/cookie_usage.php)
Hope that helps.
You need to update your spider.txt file. I believe there is a contrib with an up to date list.
The file is in /includes/spiders.txt
You should also set (in your admin) prevent spider sessions to true.
If you dont want your cookie usage (or any other page) to be indexed you should disallow it in your robot.txt file. (Disallow: /catalog/cookie_usage.php)
Hope that helps.
