Jump to content



Photo
- - - - -

Meta Robot NoIndex Header Tag Module Proposal for v2.3.3


  • Please log in to reply
33 replies to this topic

#21   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 29 July 2012 - 17:22

I'm with the other Jim.

I just think the unparalleled coding talent that comprises the osC project would be better spent if directed at items that can't be solved with a simple text file. (See, I can suck up to the boss as well as anyone else in this thread).


I can certainly understand the feeling.

When one is aware of robots.txt, aware of the syntax of robots.txt ( or you'll kill the site potentially ) then robots.txt is ofc the way to go.

BUT and it's a big but for me .. the vast majority of shop owners do not know .. therefore an "already installed" version of this looks to me a big plus.

Ask the average site owner to actually create syntax of any form and it virtually always goes wrong.

I do think though it would have been far more valuable if e.g you could: -

Select a root file and choose: -

Noindex
Noindex, follow ( if querystring has these keys )
canonical ( params to remove from tep_href_link() ( tep_get_all_get_params() ) )

Edited by FWR Media, 29 July 2012 - 17:28.


#22   germ

germ
  • Members
  • 13,921 posts
  • Real Name:Jim
  • Gender:Male
  • Location:USA (GMT-6)

Posted 29 July 2012 - 17:56

Ask the average site owner to actually create syntax of any form and it virtually always goes wrong.


I agree wholeheartedly.

You wouldn't believe how many I've seen rename the admin then "disallow" it in the robots.txt file...
/huh.png' class='bbc_emoticon' alt=':huh:' />

Doing it correctly would then entail module control, although having any of the aforementioned pages indexed by search engines isn't the "end of the world" for sites as far as SEO purposes are concerned I wouldn't think.
If I suggest you edit any file(s) make a backup first - I'm not perfect and neither are you.

"Given enough impetus a parallelogramatically shaped projectile can egress a circular orifice."
- Me -

"Headers already sent" - The definitive help

"Cannot redeclare ..." - How to find/fix it

SSL Implementation Help

Like this post? "Like" it again over there >

#23   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 29 July 2012 - 18:18

Doing it correctly would then entail module control


Well yes, but that is exactly what Harald has done. There is no added benefit of excluding in robots.txt against excluding via noindex.

I think people are fixated on it as it is in webmaster tools.

Edited by FWR Media, 29 July 2012 - 18:18.


#24   Mark Evans

Mark Evans

    Code Monkey Rockstar

  • Core Team
  • 2,521 posts
  • Real Name:Mark
  • Gender:Male
  • Location:Behind you :-P

Posted 29 July 2012 - 22:13

Harald, please do not invoke jQuery for this. jQuery is buggy and from v2.0 and beyond there will be no support for legacy browser like IE<10.


jQuery is already used in the front-end of osCommerce, integration into the backend was only a matter of time.

Personally I think the many benefits of jQuery far outweigh the few negatives and the no-support for older browsers is a good thing, the whole world would be better off not supporting IE7,IE8 and IE9
Mark Evans
osCommerce Monkey & Lead Guitarist for "Sparky + the Monkeys" (Album on sale in all good record shops)

---------------------------------------
Software is like sex: It's better when it's free. (Linus Torvalds)

#25   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 29 July 2012 - 22:53

Personally I think the many benefits of jQuery far outweigh the few negatives and the no-support for older browsers is a good thing, the whole world would be better off not supporting IE7,IE8 and IE9


Seconded! the world wants standards compliance.

Edited by FWR Media, 29 July 2012 - 22:56.


#26   kymation

kymation

    Code Monkey

  • Community Sponsor
  • 8,293 posts
  • Real Name:Jim Keebaugh
  • Gender:Male
  • Location:Aberdeen WA USA

Posted 29 July 2012 - 23:07

A stock osCommerce 2.3x admin already uses jQuery, so this isn't adding anything. Also, while I agree the world would be better off without Internet Explorer, jQuery is not removing support for it. The 1.x branch will continue to be updated and will still support the old versions of IE. Those who choose not to support old IE can use the 2.x branch.

As for bugs, show me any (non-trivial) software package anywhere that has no bugs. I've never seen such an animal.

Regards
Jim

My Addons

Banners Box Download Support
Categories Accordion Box Download Support
Closest Shipper 2.2x Support
Document Manager 2.2x Support
Generic Box Download Support
Get 1 Free 2.2x Support
Modular Front Page Download Support
Modular SEO Header Tags Download Support
MVS 2.2x Support
PDF Datasheet Download Support
Price in Cart Only/MAPP Download Support
Price Updater 2.2x
Products Specifications 2.3.x Development Version Support Bugs/Suggestions
Request a Review Download Support
Shopping List Download Support New!
Specials Image Overlay Download Support
Superfish Categories Box Download Support
Theme Switcher 2.3+ Support  Updated


#27   Juto

Juto
  • Members
  • 369 posts
  • Real Name:Sara
  • Gender:Female

Posted 30 July 2012 - 05:20

Yes, that's true. jQuery do have benefits, I realize that. But, is it needed for ticking a files in a list?
Once the MS realize it's left behind by jQuery I hope they feel the pressure to adhere to standards.

As for my experimental site, once I have replaced fancybox, I don't need jQuery at all.
Unless it's required for some "super" -application. By the way have you looked into http://rightjs.org/
The author claims it has passed jslint. I haven't tested that lib., it looks interesting though.

As for a totaly bugfree site, well Jim, you are quite right. My experimental animal, is nearly there.
But I don't think I ever will find the very last bug... It's a vision, a dream I keep alive, til' the end.
I do realize that it's impossible, since php etc., have bugs in itself. Once I was involved, for some
years, writing fortran code for security analys for nuclear power plants, which planted a quality
requirement in the back of my head. So, I try to fix the tinest flaw, even if it's just a textual one.

All I have done couldn't be if it weren't for this excellent community and osC's remarkable developers.
So, a big thank you to all of you.

Sara

#28   burt

burt

    Vanquisher of Demons

  • Community Team
  • 9,841 posts
  • Real Name:G Burton
  • Gender:Male
  • Location:UK/DEV/on

Posted 30 July 2012 - 07:42

@Harald Ponce de Leon

I did something quite similar a while back and ended up using this:

$files_array = array_map('basename', glob(DIR_FS_CATALOG . '*.php'));

which I think does pretty much the same job as this:


$file_extension = substr($PHP_SELF, strrpos($PHP_SELF, '.'));


	$files_array = array();
if ($dir = @[member='dir'](DIR_FS_CATALOG)) {
   while ($file = $dir->read()) {
	 if (!is_dir(DIR_FS_CATALOG . $file)) {
	   if (substr($file, strrpos($file, '.')) == $file_extension) {
			$files_array[] = $file;
		  }
		}
	  }
	  sort($files_array);
	  $dir->close();
	}

Not sure which would be preferable? If yours is preferable, I'll change my script!




Gary

Edited by burt, 30 July 2012 - 07:43.

IF YOU MAKE A POST REQUESTING HELP...please state the exact version of osCommerce that you are using. THANKS
 
Big Bang Templates for 2.3 osCommerce - 2.3.1 > 2.3.4 - Buy One, Get One Free
 
--
Making your osCommerce better, one module at a time - get in touch.

#29   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 31 July 2012 - 07:08

@Harald Ponce de Leon
Not sure which would be preferable? If yours is preferable, I'll change my script!


glob is PHP_VERSION >= 4.3.0
array_map PHP_VERSION >= 4.0.6

Otherwise osCommerce could have a simple wrapper like: -

  function tep_directory_iterator( array $args = array() ) {
    if (!isset($args['path'],$args['pattern'],$args['flags'])) {
	  trigger_error(__FUNCTION__ . ': Missing params, function requires the keys path/pattern/flags', E_USER_NOTICE);
	  return array();
    }
    return array_map('basename', glob($args['path'] . $args['pattern'], $args['flags']));  
  }


#30 ONLINE   Harald Ponce de Leon

Harald Ponce de Leon

    Healthy Giraffe

  • Core Team
  • 4,857 posts
  • Real Name:Harald Ponce de Leon
  • Gender:Male
  • Location:Solingen, Germany

Posted 31 July 2012 - 09:11

@burt @FWR Media

That optimization can be considered for v2.4 when the minimum PHP version requirement increases.
Harald Ponce de Leon

#31   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 31 July 2012 - 10:35

@Harald Ponce de Leon

Well if the version allows SPL I have a directory iterator if interested, it is quite verbose but offers quite a lot.

Uses: -
DirectoryIterator, FilterIterator. ArrayIterator, ArrayObject

The basic functionality is: -

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG )->iterate() as $key => $directory ) {
  // Returns all directories except ., .., .svn, CVS;
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG )->setType( 'file' )
   ->setFilters( array( 'index.php' ) )
   ->iterate() as $key => $directory ) {
  // returns all files except for index.php
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG )->setType( 'file' )
   ->setFilterMode( 'allow_extn' )
   ->setFilters( array( '.php' ) )
   ->iterate() as $key => $directory ) {
  // Returns all files that have a .php extension
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG )->setType( 'file' )
  ->setFilterMode( 'allow' )
  ->setFilters( array( 'index.php', 'contact_us.php', 'account.php' ) )
  ->iterate() as $key => $directory ) {
  // returns only index.php, contact_us.php, account.php
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG . DIR_WS_IMAGES )->setType( 'file' )
  ->setFilterMode( 'allow_extn' )
  ->setFilters( array( '.png' ) )
  ->iterate() as $key => $directory ) {
  // Returns all files with .png extn
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG . DIR_WS_IMAGES )->setType( 'file' )
   ->setFilters( array( '.gif' ) )
   ->setFilterMode( 'allow_extn' )
   ->iterate() as $key => $directory ) {
  // Returns all .gif images
  }

  $iterator = new FilteredIterator;
  foreach ( $iterator->setPath( DIR_FS_CATALOG . DIR_WS_IMAGES )->setType( 'file' )
   ->setFilters( array( '.gif', '.htaccess' ) )
   ->setFilterMode( 'filter_extn' )
   ->iterate() as $key => $directory ) {
  // Returns all files EXCEPT .gif and .htaccess
  }


#32   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 31 July 2012 - 11:43

Forgot to mention PHP 5.4 even shorter

  foreach ( ( new FilteredIterator )->setPath( DIR_FS_CATALOG )->iterate() as $directory ) {
    // Returns all directories except ., .., .svn, CVS;
  }


#33   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 01 August 2012 - 11:46

I hadn't thought of arraymap and glob for a while ( too busy getting confused with IteratorItoratorIteratorIt........... s )

@burt so thanks for bring it up.

Tested against my system arraymap with glob is significantly faster ( 4 X ) on simple file extension searches like *.php but significantly slow on directory searches GLOB_ONLYDIR.

E.g. an iteration searching for directories in osCommerce root ( tests are ran 100 times ): -

FilteredDirectoryIterator ran for: 0.40297293663025 seconds

array_map/glob ran for: 0.41827583312988 seconds

Yet with the extension search ( *.php ) in root it is a different story: -

FilteredDirectoryIterator ran for: 0.43397998809814 seconds

array_map/glob ran for: 0.12356305122375 seconds

Anyone got any data on directory iterations?

For now I have changed my system to use arraymap/glob only where it is a simple allow full filename/allow extension search and where the search is only one item and where it is a file search.

Edited by FWR Media, 01 August 2012 - 11:47.


#34   FWR Media

FWR Media
  • Members
  • 6,839 posts
  • Real Name:Robert Fisher
  • Gender:Male
  • Location:Stowmarket - Suffolk - UK

Posted 01 August 2012 - 13:03

This is better all tests are 100 cycles: -

Iteration finding all directories in osCommerce root: -

FilteredDirectoryIterator: 0.23796987533569

array_map/glob: 0.42517614364624

FilteredDirectoryIterator is now a lot faster.

Iteration ( not using arraymap/glob ) finding all .php files in osCommerce root: -

FilteredDirectoryIterator: 0.24290108680725

array_map: 0.12460207939148

arraymap/glob is a lot faster ( but not as bad as before /smile.png' class='bbc_emoticon' alt=':)' /> ).

Iteration ( using arraymap/glob ) finding all .php files in osCommerce root: -

FilteredDirectoryIterator: 0.12593603134155

array_map: 0.12427401542664

Tiny difference due to class overhead as expected.

Hmmm I'm happy with that.