.htaccess redirect and robots.txt confilct

[RAC 6]

Hi all.

I discovered that my new shop website is not working and it seems I have an .htaccess and robots.txt conflict that I am unable to sort out.

First I noticed that Google Sitemap XML was not able to access my robots.txt, after some help from Jack_mce and my host technical support, it was found that the ,htaccess redirect was causing the problem.

Obviously I need both of these files for my site to work properly, can anyone suggest a fix for this .htaccess file.

My shop is in the public_html folder along with the robots.txt and .htaccess.

The .htaccess file is

RewriteEngine On
RewriteCond %{HTTPS} off [OR]
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ https://www.retro-antiques-curios.co.uk/$1 [R=301,L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]
RewriteCond %{REQUEST_URI} !^/RAC_Shop [NC]
RewriteRule ^(.*)$ /RAC_Shop/$1

 

My robots.txt file is

# Robots.txt

 

User-agent: *

Disallow: /includes/

Disallow: /account.php

Disallow: /account_edit.php

Disallow: /account_history.php

Disallow: /account_history_info.php

Disallow: /account_newsletters.php

Disallow: /account_notifications.php

Disallow: /account_password.php

Disallow: /address_book.php

Disallow: /address_book_process.php

Disallow: /checkout_confirmation.php

Disallow: /checkout_payment.php

Disallow: /checkout_payment_address.php

Disallow: /checkout_process.php

Disallow: /checkout_shipping.php

Disallow: /checkout_shipping_address.php

Disallow: /checkout_success.php

Disallow: /cookie_usage.php

Disallow: /create_account.php

Disallow: /create_account_process.php

Disallow: /create_account_success.php

Disallow: /login.php

Disallow: /logoff.php

Disallow: /password_forgotten.php

Disallow: /popup_image.php

Disallow: /redirect.php

Disallow: /shopping_cart.php

Disallow: /user.css

sitemap: https://www.retro-antiques-curios.co.uk/RAC_Shop/sitemapindex.xml

 

Thanks in advance Shaun

Running BS Edge.

[♥BrockleyJohn 312]

I'm not totally sure I understand your site structure but assuming the site is actually in the subdirectory /RAC_Shop/ I can see a couple of issues

RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php?/$1 [L]

redirects any file or directory name that doesn't exist sending it as a parameter to /index.php

I guess this is url rewriting but if your shop is in a subdirectory then the rewrite should point to the subdirectory:

RewriteRule ^(.*)$ /RAC_Shop/index.php?/$1 [L]

The other issue is with the last redirect - when anyone looks for robots.txt they get redirected to the subdirectory so you need an extra rule to exclude the file:

RewriteCond %{REQUEST_URI} !^/robots.txt$ [NC]

put before the last rewrite rule

[RAC 6]

Hello John

Thank you for your code, that worked perfectly, you are a star, I can now call my robots.txt file.

However in my Google webmaster account when I attempt the robots.txt tester I get an error Fetch Failed!

Could this be anything to do with my original problem, or will I need to start a new thread about Google Webmaster?

Kind regards Shaun

 

[♥BrockleyJohn 312]

My guess is it's related.

So when you go to Crawl / robots.txt Tester you probably see the results of a previous fetch attempt. If you click on See live robots.txt does that work ok?

If so, try clicking submit at the bottom so it attempts another fetch.

[RAC 6]

Again your instructions worked perfectly, thank you John, the robots.txt tester is now working.

Hopefully this matter is now resolved.

I will get back if I discover any more errors or problems.

Shaun