Jump to content
Latest News: (loading..)
RAC

.htaccess redirect and robots.txt confilct

Recommended Posts

Hi all.

I discovered that my new shop website is not working and it seems I have an .htaccess and robots.txt conflict that I am unable to sort out.

First I noticed that Google Sitemap XML was not able to access my robots.txt, after some help from Jack_mce and my host technical support, it was found that the ,htaccess redirect was causing the problem.

Obviously I need both of these files for my site to work properly, can anyone suggest a fix for this .htaccess file.

My shop is in the public_html folder along with the robots.txt and .htaccess.

The .htaccess file is

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.retro-antiques-curios.co.uk/$1 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]
RewriteCond %{REQUEST_URI} !^/RAC_Shop [NC]
RewriteRule ^(.*)$ /RAC_Shop/$1

 

My robots.txt file is

# Robots.txt

 

User-agent: *

Disallow: /includes/

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_newsletters.php

Disallow: /account_notifications.php

Disallow: /account_password.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_process.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /logoff.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /redirect.php

Disallow: /shopping_cart.php

Disallow: /user.css

sitemap: https://www.retro-antiques-curios.co.uk/RAC_Shop/sitemapindex.xml

 

Thanks in advance Shaun

Running BS Edge.

Share this post


Link to post
Share on other sites

I'm not totally sure I understand your site structure but assuming the site is actually in the subdirectory /RAC_Shop/ I can see a couple of issues

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]

redirects any file or directory name that doesn't exist sending it as a parameter to /index.php

I guess this is url rewriting but if your shop is in a subdirectory then the rewrite should point to the subdirectory:

RewriteRule ^(.*)$ /RAC_Shop/index.php?/$1 [L]

The other issue is with the last redirect - when anyone looks for robots.txt they get redirected to the subdirectory so you need an extra rule to exclude the file:

RewriteCond %{REQUEST_URI} !^/robots.txt$ [NC]

put before the last rewrite rule


For a new install or if your store isn't mobile-friendly, get the community-supported responsive osCommerce (2.3.4.1 CE) here: https://github.com/gburton/Responsive-osCommerce/archive/2341-Frozen.zip

Working on generalising bespoke solutions for Quickbooks integration, Easify integration and pay4later (DEKO) integration at 2.3.x

Share this post


Link to post
Share on other sites

Hello John

Thank you for your code, that worked perfectly, you are a star, I can now call my robots.txt file.

However in my Google webmaster account when I attempt the robots.txt tester I get an error Fetch Failed!

Could this be anything to do with my original problem, or will I need to start a new thread about Google Webmaster?

Kind regards Shaun

 

Share this post


Link to post
Share on other sites

My guess is it's related.

So when you go to Crawl / robots.txt Tester you probably see the results of a previous fetch attempt. If you click on See live robots.txt does that work ok?

If so, try clicking submit at the bottom so it attempts another fetch.


For a new install or if your store isn't mobile-friendly, get the community-supported responsive osCommerce (2.3.4.1 CE) here: https://github.com/gburton/Responsive-osCommerce/archive/2341-Frozen.zip

Working on generalising bespoke solutions for Quickbooks integration, Easify integration and pay4later (DEKO) integration at 2.3.x

Share this post


Link to post
Share on other sites

Again your instructions worked perfectly, thank you John, the robots.txt tester is now working.

Hopefully this matter is now resolved.

I will get back if I discover any more errors or problems.

Shaun

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×