Hiding some products fro Search Engines

[Ruthey]

Hi all, this may seem like a strange request but there is a genuine reason!!

 

I need to make sure that some of the products on our site cannot be found on any search engines.

 

I have found a solution to prevent the products showing in the main store but still be accessed and added to a cart if the direct URL is given but in addition to this I need to make sure the pages won't be found by Google et al and listed in searches.

 

There's not a vast number of products this is required for so its possible to manually edit each product page after creation, if that's the way oscommerce works.

 

Does anyone have any ideas or managed to do this successfully in the past?

 

Thanks in advance

 

Ruth

[burt]

Most search engines obey the noindex tag; http://www.mattcutts.com/blog/handling-noindex-meta-tags/

 

So what you need to do is have a system whereby you can assign a noindex tag to individual products.

 

I would do this by adding in an extra "status" button. This would assign a 1 or a 0 to each product.

If the product has a 1, then write this code on the product_info page;

 

<meta name="robots" content="noindex" />

 

And if it has a 0 status, then do nothing as it's a normal product that you do want in the SE's

[Ruthey]

That makes sense, I will give it a go and let you know how we get on, thanks very much :)

[Jack_mcs]

For google, the no index tag means they won't rank the page. They may or may not index it. To do this, you would have to prevent the link from being shown at all if the visitor is a spider. Also keep in mind that hiding something from the search engines that can be seen by visitors and is against their rules. It is unlikely they would notice something like this but if they do, you risk getting your site banned.

[Hotclutch]

If you are creating new products, then you can add these to robots.txt under a disallow directive. This will prevent SEs from crawling the products, and hence adding them to the index.

 

For existing products, you will probably have to use meta = noindex, as suggested above.

[burt]

For google, the no index tag means they won't rank the page. They may or may not index it. To do this, you would have to prevent the link from being shown at all if the visitor is a spider.

 

Incorrect. Please read http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710 which specifically states;

 

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

 

Also keep in mind that hiding something from the search engines that can be seen by visitors and is against their rules. It is unlikely they would notice something like this but if they do, you risk getting your site banned.

 

Look at the question being posed. IF what you say is true, COMMON SENSE would suggest that Google would not ever obey the noindex rule.

[Jack_mcs]

Incorrect. Please read http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710 which specifically states;

 

To entirely prevent a page's contents from being listed in the Google web index even if other sites link to it, use a noindex meta tag. As long as Googlebot fetches the page, it will see the noindex meta tag and prevent that page from showing up in the web index.

 

Yes, that is correct but you are skipping over the "As long as Googlebot fetches the page" part. If google sees the no index tag on a page, they will not list it, or will remove it from their listing if it is already present. But if they don't crawl the page and they find a link to it, they may still list it. This has long been common knowledge by those familiar with SEO and Matt Cutts has disccussed it many times. That's why I said the link had to be hidden from the search engines.

[Debs]

Adding an extra "status" button may be difficult if you are not familiar with oscommerce.

 

A simple workaround for a product (or handful of) would be something like this added to your product_info.php page (before the closing tag):

 

Change products_id==249 to your product id you are targeting. Just view your product and the id will be in the url. If you have many products to target, place them in an array.

 

<?php $noindex='';

if($products_id==249) $noindex='on';

?>

 

<?php

if ($noindex == 'on') {

echo '<meta name="robots" content="noindex" />';

}

?>

[Debs]

this before the closing head tag... it appears to be stripping some of my post </head>

[burt]

Let's say that some of these products ARE already showing in Google. They take my advice to to do noindex.

FINE. Any already indexed page does not need links to it in order for the bot to return

 

Let's say that these products are NOT already showing in Google. The poster therefore wants to ensure that they never get listed. They take my advice to noindex.

FINE. These pages are not indexed, the first time the bot appears it see the noindex and "goes away".

 

My advice stands in both cases. Do the noindex.

 

 

Let's say that the poster adds a new product, with noindex. Googlebot takes it time to visit, but in the meantime sees a link to the page. In this scenario and only this scenario does the post of Jack make a little sense. Even so, you do not know that it is YOUR link that created the listing. It could be a link on facebook or where-ever - the point is "you cannot control who links to your pages".

 

In short;

 

- add the noindex

- don't remove links to the noindex page

 

 

If Google is listing a noindex page, then you know that two things have happened;

 

1. Google knows about your page

2. Googlebot has not visited the page

 

You can be sure that in very short order the page will be removed as per noindex.

 

If, for some reason, a noindex page is stuck as a listing, go to your webmaster tools and remove it.

[Jack_mcs]

Let's say that the poster adds a new product, with noindex. Googlebot takes it time to visit, but in the meantime sees a link to the page. In this scenario and only this scenario does the post of Jack make a little sense. Even so, you do not know that it is YOUR link that created the listing. It could be a link on facebook or where-ever - the point is "you cannot control who links to your pages".

You're still missing the point. Google doesn't have to visit a page to list it. Just having a link on the site can get it listed. The no index tag would not make one bit of difference in that case. If the link has never appeared on the web (outside of the shop) then a 301 needs to be issued for it to be sure it is removed. If the link isn't on the web, anywhere, then it has to be hidden from the search engines on the site. A no index tag simply won't accomplish what the op asked for.

[Hotclutch]

....then a 301 needs to be issued for it to be sure it is removed....

 

It looks like the pages the original poster is "trying to hide" is returning HTTP 200, and not 404 or 302, in which case a 301 redirect would probably be needed. The noindex should work if robots are seeing HTTP 200.

[Hotclutch]

...Google doesn't have to visit a page to list it. Just having a link on the site can get it listed...

 

Hi Jack,

 

I have not seen this yet. This would imply a listing without content ? as Google has not visited (crawled) the page yet.

[Jack_mcs]

Hi Jack,

 

I have not seen this yet. This would imply a listing without content ? as Google has not visited (crawled) the page yet.

Yes, that's exactly what they do, though probably not for every link. See what

. This is from about three years ago but it has been this way longer than that.