Jump to content

Archived

This topic is now archived and is closed to further replies.

sackling

googlebot 302 redirect & cookie_usage.php

Recommended Posts

I am having a problem with spiders it seems.

 

When I test out a url with "fetch as googlebot" I get the following error:

 

HTTP/1.1 302 Moved Temporarily
Date: Sun, 06 Feb 2011 20:32:09 GMT
Server: Apache
X-Powered-By: PHP/5.2.15
Location: http://www.mysite.com/cookie_usage.php
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

 

I am not sure why it is getting redirected to cookie_usage.php but I have a feeling it is somehow related to the fact that the spider cannot create a session. I have spider sessions disabled but I am pretty sure they are supposed to be. How can i fix this?

Share this post


Link to post
Share on other sites

Robert,

 

You have your cookies set to true in your admin/configuration/sessions. And you should. However the script you used to test your site would not accept a cookie to the session ended.

 

If you want to test your site like that, try disabling 'Force Cookie Usage' before running that script.

 

 

Chris


:|: Was this post helpful ? Click the LIKE THIS button :|:

 

See my Profile to learn more about add ons, templates, support plans and custom coding (click here)

Share this post


Link to post
Share on other sites

I just did a fetch as googlebot from webmaster tools and it works fine.

 

 

HTTP/1.1 200 OK
Date: Mon, 07 Feb 2011 01:24:23 GMT
Server: Apache mod_fcgid/2.3.5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.15
Set-Cookie: cookie_test=please_accept_for_session; expires=Wed, 09-Mar-2011 01:24:24 GMT; path=/; domain=.mysite.com
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

Share this post


Link to post
Share on other sites

Robert,

 

You have your cookies set to true in your admin/configuration/sessions. And you should. However the script you used to test your site would not accept a cookie to the session ended.

 

If you want to test your site like that, try disabling 'Force Cookie Usage' before running that script.

 

 

Chris

 

I didn't have force cookies set to true which I believe is the correct way to have it. I tried changing it to true and i still got the same 302 message from teh googlebot

Share this post


Link to post
Share on other sites

I just did a fetch as googlebot from webmaster tools and it works fine.

 

 

HTTP/1.1 200 OK
Date: Mon, 07 Feb 2011 01:24:23 GMT
Server: Apache mod_fcgid/2.3.5 mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
X-Powered-By: PHP/5.2.15
Set-Cookie: cookie_test=please_accept_for_session; expires=Wed, 09-Mar-2011 01:24:24 GMT; path=/; domain=.mysite.com
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html

 

 

But I am guessing you did it on your site.. Obviously something is wrong with my setup.

Share this post


Link to post
Share on other sites

I just decided to have a look at my product_info.php and noticed this little tidbit in the code:

 

 // redirect the customer to a friendly cookie-must-be-enabled page if cookies are disabled (or the session has not started)
 if ($session_started == false) {
   tep_redirect(tep_href_link(FILENAME_COOKIE_USAGE));
 }

 

Now.. This makes perfect sense.. since the spider is not allowed to have a session all the product info pages will redirect to cookie_usage.php

 

So removing that fixed the googlebot issue.

 

Now I don't know how important that code is to keep for customer purposes.

Share this post


Link to post
Share on other sites

That code should be very important as you may have a customer who would not have cookies enabled and would get frustrated trying to add a product to the cart.

 

I suggest you put the code back.

 

Instead you should open up /includes/spiders.txt and ensure that google and/or googlebot exists there.

Share this post


Link to post
Share on other sites

That code should be very important as you may have a customer who would not have cookies enabled and would get frustrated trying to add a product to the cart.

 

I suggest you put the code back.

 

Instead you should open up /includes/spiders.txt and ensure that google and/or googlebot exists there.

 

Here is the spider.txt I am using from the contribution which provides one. I don't see google listed there per se but it may be under a different name? how would having it in there not redirect it to the cookies_usage page though since spiders arent allowed to have sessions and the code explicitly says if there is no session redirect to that page?:

 

crawl
slurp
spider
ebot
obot
abot
dbot
hbot
kbot
lbot
mbot
nbot
pbot
rbot
sbot
tbot
ubot
vbot
ybot
zbot
bot.
bot/
_bot
.bot
/bot
-bot
:bot
(bot
accoona
adressendeutschland
appie
architext
asterias
atlocal
atomz
augurfind
bannana_bot
baypup
bdfetch
biglotron
blaiz
blo.
blog
boitho
booch
butterfly
ccubee
cfetch
charlotte
comagent
combine
csci
curl
dataparksearch
daumoa
depspid
digger
ditto
dmoz
docomo
dtaagent
ebingbong
ejupiter
falcon
findlinks
gazz
genieknows
goforit
gralon
grub
gulliver
harvest
helix
heritrix
holmes
homer
htdig
ia_archiver
ichiro
iconsurf
iltrovatore
indexer
ingrid
ivia
jakarta
jetbot
kit_fireball
knowledge
kretrieve
lachesis
larbin
libwww
lwp
mantraagent
mapoftheinternet
mediapartners
mercator
metacarta
microsoft url control
minirank
miva
mj12
mnogo
moget/
multitext
muscatferret
myweb
najdi
nameprotect
ncsa beta
netmechanic
netresearchserver
nokia6682/
npbot
noyona
nutch
objectssearch
omni
onetszukaj
openintelligencedata
osis-project
pagebull
page_verifier
panscient
pear.
pogodak
poirot
pompos
poppelsdorf
psycheclone
publisher
python
rambler
salty
sbider
scooter
scoutjet
scrubby
seeker
seek.
shopwiki
sidewinder
silk
smartwit
sna-
snappy
sohu
sphere
sphider
spinner
spyder
steeler/
sygol
szukacz
tarantula
t-h-u-n-d-e-r-s-t-o-n-e
/teoma
theophrastus
tutorgig
twiceler
twisted
updated
vagabondo
volcano
voyager/
voyager-hc
w3c_validator
walker
wauuu
wavefire
webmonkey
websitepulse
wget
worldlight
worm
wwwster
xenu
xirq
yandex
yanga
yeti
yodao
zao/
zippp
zyborg
....
! spiders.txt Contribution version 2010-04-17  - Please read readme before editing

Share this post


Link to post
Share on other sites

×