Jump to content
ArtcoInc

Search add-on Reviews

Recommended Posts

@burtThanks again Gary.....I'll have a look at the keywords system.  

BTW, my fuzzy search is working it is just that the metaphone function doesn't produce the results I was expecting.  

Dan

 

Share this post


Link to post
Share on other sites

@Dan Cole

 

On 6/7/2017 at 8:09 AM, Dan Cole said:

@ArtcoInc Malcolm....do you still have these search modules set up somewhere, where you can easily turn them on and off?    I just had an interesting conversation with a customer about our search function which didn't product any results when they didn't use a space in the appropriate spot....for example if you're looking for "Marine Flake" fish food on my site and entered "marineflake" it doesn't give you any results.  I know it's looking for the full term as a single word and can't find it since it doesn't exist and I get that but it's not very smart.   Just for the heck of it I tried it in google...the search engine that everyone is being educated to use and likely expect all searches to work the same and as you might expect google doesn't have an issue with it.  "marineflake" will give you the expected results.  So I'm left wondering if any of the available osC search modules might work in the same way as google when users don't add any spaces.  Just wondering if you or anyone else reading this would happen to know?

Dan

@burt already answered you about his Fuzzy module. There is another way ...

You can add Google Search to your site. You'd have to create an account with Google, add the bit of code to your site, and wait for them to crawl your site (or, submit a site map to Google to speed things up).

I started to play with this, but didn't get too deep into it, so I don't know how smart it's search capability is, nor if it would be able to handle your "marineflake" example.

Beyond these, I don't think that any of the osC search modules would be smart enough to handle your example. As far as I can tell, they all just search for the specified word (or words).

Malcolm


Get the latest Responsive osCommerce CE (community edition) here .

Share this post


Link to post
Share on other sites

One other observation ...

I find that I can get so involved when working on our stores that we 'know' how thing work, and may assume that our customers do too. For example ...

When I was testing the Twitter Type-Ahead search function, I asked someone to try it and tell me what they thought. I watched them...

1) they clicked on the search box

2) they looked down at the keyboard as they typed their search term, and pressed enter

(as they were typing, the search box had a number of suggestions, with the first one highlighted. When the user pressed Enter, it selected the highlighted suggestion)

3) they looked up at the screen, and wondered why search returned something other than what they wanted

Just an observation .... :cool:

Malcolm


Get the latest Responsive osCommerce CE (community edition) here .

Share this post


Link to post
Share on other sites

@ArtcoInc Thanks for the input Malcolm.   I haven't decided what to do yet.  I had a look at the fuzzy search module mentioned by Gary but it's indexing didn't help any. That go me to thinking about how we did this in the old days which was to create our own index.  I plan to look at that but just haven't gotten to it yet.  Back in the day there were several programs available on the web that you could use to build the index. You would then adjust you search routine to use it for any queries.  Google is certainly an option too.

Dan

Share this post


Link to post
Share on other sites

@Dan Cole

Have you looked at this one? I started to mess with it, but life got in the way.

https://apps.oscommerce.com/caQMT

It seems that the people who developed Sphider stopped working on it. There is a spin-off, but (if I remember correctly) it is a commercial product.

That said, I was able to integrate the above add-on into the admin side, and was able to generate the index through the back end. I ran into snags on the front end (beyond my coding skills), though.

HTH

Malcolm


Get the latest Responsive osCommerce CE (community edition) here .

Share this post


Link to post
Share on other sites

@ArtcoInc Thanks again Malcolm.  I wasn't aware of that add on so I'll be sure to have a look at it.  I see it was uploaded by @pbor1234 who I think still frequents these forums so maybe we can get an update on where it ended up.    The challenge will likely be in finding suitable software to create the index...I saw mention of Sphider in my searches as well as what I'm thinking is the other one you were referring to -- Lucene but I don't know anything about either.

Dan

Share this post


Link to post
Share on other sites

@ArtcoInc Here is the support thread for the contribution....it looks like the development died along with Sphider.  I'm not sure what the current status of Lucene is.

Dan

Edited by Dan Cole

Share this post


Link to post
Share on other sites

@Dan Cole I just installed Fuzzy Search on a test site...

Admin Side:

Made a new product called "Marine Flake"
Installed the module for advanced_search_result

Shop Side:

I searched for Marine Flake:  
1 normal search result - expected

I searched for marineflake:  
0 results - expected
1 fuzzy result - "Marine Flake"

I searched for neflake:
0 results - expected
1 fuzzy result - "Marine Flake"

I searched for "rine flake":
1 result - expected

This is with Fuzzy Factor set to 4.
 


This is a signature that appears on all my posts.  
IF YOU MAKE A POST REQUESTING HELP...please state the exact version
of osCommerce that you are using. THANKS

 
Get the latest Responsive osCommerce CE (community edition) here

Share this post


Link to post
Share on other sites

@Dan Cole

As I mentioned, I was able to get the back end of Sphider to work. It basically goes through your site and indexes every word, and stores it in several database tables. Here's a screen shot from the admin side:

5948012f51184_ScreenCapture-06-19-17.thumb.JPG.25c821f81e04a34e3733baa5a9fa8b85.JPG

For example, you can see that the word 'kevlar' is the most used word on my site. :rolleyes:

Several points:

1) I'm sure you *could* add words to the database, but they are spread out across multiple tables, with cross indexes, etc. I think it would take a bit of work to figure out how everything is stored and sorted.

2) This only indexes real words, and your 'marineflake' example probably would not find a match.

3) As I also said, I only got the back end working. I'm sure we *could* resolve the issues on the front end, but I'm not sure it would be worth the effort. As you noted, support for this has all but died. And, even if you/we were to revive it, I don't know if this search function is any better than other options.

Finally, @burt has shown that his Fuzzy search does pretty much what you want. So, unless Google search installed on our stores offers better search capabilities, I think that @burt's Fuzzy is the current winner. :biggrin:

Wow, this 'Search Review' sure has taken on a life of it's own. :tongue: Who'd have thought? :ph34r:

Malcolm

 


Get the latest Responsive osCommerce CE (community edition) here .

Share this post


Link to post
Share on other sites
6 hours ago, burt said:

@Dan Cole I just installed Fuzzy Search on a test site...

Admin Side:

Made a new product called "Marine Flake"
Installed the module for advanced_search_result

Shop Side:

I searched for Marine Flake:  
1 normal search result - expected

I searched for marineflake:  
0 results - expected
1 fuzzy result - "Marine Flake"

I searched for neflake:
0 results - expected
1 fuzzy result - "Marine Flake"

I searched for "rine flake":
1 result - expected

This is with Fuzzy Factor set to 4.
 

@burt That is interesting Gary....now you have me wondering what the heck I did....I'll revisit it.

Dan

Share this post


Link to post
Share on other sites
48 minutes ago, ArtcoInc said:

Finally, @burt has shown that his Fuzzy search does pretty much what you want. So, unless Google search installed on our stores offers better search capabilities, I think that @burt's Fuzzy is the current winner. :biggrin:

Wow, this 'Search Review' sure has taken on a life of it's own. :tongue: Who'd have thought? :ph34r:

Malcolm

@ArtcoInc Thanks Malcolm....I'll have to revisit this again....I'm now not sure if I tried the example using the fuzzy search or if I was having other issues with it.   I'll post back when I give it another go. 

Dan

Share this post


Link to post
Share on other sites
1 hour ago, Dan Cole said:

@burt That is interesting Gary....now you have me wondering what the heck I did....I'll revisit it.

Dan

@burt Gary I tried the fuzzy search again and it still failed to find marineflake on my site....in an effort to see what was going on I echoed the query to the screen and it shows the metaphone as '%MRNF%' without a space so I can see why it isn't being found.  I haven't yet checked my database to see if "MRNF" was indexed but I'm off to check that now.

Here is the actual output.

select distinct p.products_id, p.products_image, pd.products_name from products p inner join products_description pd on (p.products_id = pd.products_id) where p.products_status = '1' and pd.language_id = '1' and pd.products_metaphone like '%MRNF%' order by pd.products_metaphone limit 4

Now I'm curious what the metaphone for your query was?

Dan

 

Share this post


Link to post
Share on other sites

@burt Now I'm really puzzled....the product metaphone is....

OMKNMRNFLKW0KRLKK

So I'm thinking it should have found MRNF using MYSQL LIKE but doesn't for some reason.  I'm going to add a product with just the name Marine Flake and see what happens.

Dan

Share this post


Link to post
Share on other sites

I tried adding a product with just the name Marine Flake and then regenerated the metaphones.  No change.  I then tried running the query against the database directly and it worked! It just won't work from the search bar...I'm out of ideas for what else I can do to trouble shoot it so I'll let it percolate for the time being.   Hopefully an idea or two will surface.

Dan

Share this post


Link to post
Share on other sites

As usual, my bad.   I had added some breaks in the code to test it previously when it wasn't getting the results I expected.  I left those breaks in place so the search wasn't getting past the query.   Removing the breaks got me back to where I was.  I then spend a bit more time with it and realized it was working but showing what seemed to be odd results.  The results where limited to 4 and didn't seem to bare any relationship to the search query....I increased the limit to 20 and guess what showed up....just way down the list....yes, Marine Flakes both the original product as well as the new one I added. 

In summary it looks like the fuzzy search does work, it is just that there is no way to ensure that it will identify the most relevant results if you have a number of products containing the metaphone being searched for.  The more products you have the less relevant the results will be. 

I'm now wondering if there is a way to sort or rework the results so the more relevant ones will appear higher in the list? I'll play around with the fuzzy factor a bit to see if that helps.

Dan

Share this post


Link to post
Share on other sites

It's difficult when you have a number of items that potentially match the Metaphone.

The (in my opinion) logical route to go down would be to order the results by the "start position" of the search query.

So let us imagine you have 3 products:

Super Duper Fantastic Blue Widget Marine Flake
Super Marine Flake
Marine Flake

And then someone searched for "marine flake"

- use the normal search function as this is a match for all three products

And then someone searched for "marineflake"

- matches nothing using the normal search, but return results as follows in the Fuzzy;

Marine Flake aka MRNF (fuzzy starts at position 1)
Super Marine Flake aka SPRMRNF (fuzzy starts at position 4)
Super Duper Fantastic Blue Widget Marine Flake SPRDPRFNTSTCBLWDGTMRNF (fuzzy starts position 19)

At least you now have some order...

That would be the way to do it I think.


This is a signature that appears on all my posts.  
IF YOU MAKE A POST REQUESTING HELP...please state the exact version
of osCommerce that you are using. THANKS

 
Get the latest Responsive osCommerce CE (community edition) here

Share this post


Link to post
Share on other sites

ps playing with the Fuzzy Factor probably will help a bit but not massively so.  All it does is change the %aaaa% bit below from 4 characters to whatever you choose: 

and pd.products_metaphone like '%MRNF%' order by

 

The higher the number, the less likely to match in the metaphone string.

 

This Fuzzy system is potentially one of the game changer addons, but it's something that is a bit of a blunt instrument (ie, helps a massive amount for an off the shelf piece of code that takes 1 minute to set up, but is not bespoke enough to be 100% perfect at all times).

 


This is a signature that appears on all my posts.  
IF YOU MAKE A POST REQUESTING HELP...please state the exact version
of osCommerce that you are using. THANKS

 
Get the latest Responsive osCommerce CE (community edition) here

Share this post


Link to post
Share on other sites
5 hours ago, burt said:

It's difficult when you have a number of items that potentially match the Metaphone.

The (in my opinion) logical route to go down would be to order the results by the "start position" of the search query.

So let us imagine you have 3 products:

Super Duper Fantastic Blue Widget Marine Flake
Super Marine Flake
Marine Flake

And then someone searched for "marine flake"

- use the normal search function as this is a match for all three products

And then someone searched for "marineflake"

- matches nothing using the normal search, but return results as follows in the Fuzzy;

Marine Flake aka MRNF (fuzzy starts at position 1)
Super Marine Flake aka SPRMRNF (fuzzy starts at position 4)
Super Duper Fantastic Blue Widget Marine Flake SPRDPRFNTSTCBLWDGTMRNF (fuzzy starts position 19)

At least you now have some order...

That would be the way to do it I think.

@burt I like this Gary....I'm not sure how to order it in that way but I agree that makes sense to me too.  Proof will be in the pudding.

Quote

This Fuzzy system is potentially one of the game changer addons, but it's something that is a bit of a blunt instrument (ie, helps a massive amount for an off the shelf piece of code that takes 1 minute to set up, but is not bespoke enough to be 100% perfect at all times).

I think, as we are discovering here, it should be a excellent supplement to the regular search routine especially if we can order it appropriately. 

Dan

Edited by Dan Cole

Share this post


Link to post
Share on other sites

It's gotta be possible to do it.  I haven't not thought about it other than my thoughts above.


This is a signature that appears on all my posts.  
IF YOU MAKE A POST REQUESTING HELP...please state the exact version
of osCommerce that you are using. THANKS

 
Get the latest Responsive osCommerce CE (community edition) here

Share this post


Link to post
Share on other sites

@bruyndoncx  We need help from our database wizard.   Gary has developed an awesome fuzzy search module which can likely be improved even more with some refinements to the order of the results being displayed.   His idea is described in this post.   Thinking a bit about this I'm wondering if it would be possible to weight the individual results so that a sort order is assigned to them based on the position of the metaphone being searched for in the pd.products_metaphone field.  That's a mouth full.

The current query looks like this...

select distinct p.products_id, p.products_image, pd.products_name from products p inner join products_description pd on (p.products_id = pd.products_id) where p.products_status = '1' and pd.language_id = '1' and pd.products_metaphone like '%MRNF%' order by pd.products_metaphone limit 4

I'm thinking something like if metaphone being searched for is in position 1 sort order = 1 or put another way megaphone sort order = starting metaphone position.  Once the sort orders are determined then order by metaphone sort order.

Is something like this possible in mysql?

Dan

Share this post


Link to post
Share on other sites

in mysql 5.6 you have the locate function with this as example, if you cannot order by immediately in the sql, you can wrap it in an extra sql clause and order in the outer clause

SELECT pub_name,LOCATE('at',pub_name)

FROM publisher

WHERE locate('at',pub_name)>0;


KEEP CALM AND CARRY ON

I do not use the responsive bootstrap version since i coded my responsive version earlier, but i have bought every 28d of code package to support burts effort and keep this forum alive (albeit more like on life support).

So if you are still here ? What are you waiting for ?!

 

Find the most frequent unique errors to fix:

grep "PHP" php_error_log.txt | sed "s/^.* PHP/PHP/g" |grep "line" |sort | uniq -c | sort -r > counterrors.txt

Share this post


Link to post
Share on other sites

@bruyndoncx Thanks Carine.  I think that will work!  I'm thinking something like should work nicely.

SELECT LOCATE("MRNF", pd.products_metaphone) AS MatchPosition FROM  pd.products_description ORDER BY MatchPosition desc

Now I just need to see what version of MYSQL I'm running on.

Dan

Share this post


Link to post
Share on other sites

Seems I'm running on an earlier version of MySQL so I figured I was out of luck then I spotted this on w3shools.  

Quote
Works in: MySQL 5.7, MySQL 5.6, MySQL 5.5, MySQL 5.1, MySQL 5.0, MySQL 4.1, MySQL 4.0, MySQL 3.23

Happily it is back on my todo list.

Dan

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×