Dan Cole Posted June 7, 2017 Share Posted June 7, 2017 @burtThanks again Gary.....I'll have a look at the keywords system. BTW, my fuzzy search is working it is just that the metaphone function doesn't produce the results I was expecting. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
ArtcoInc Posted June 18, 2017 Author Share Posted June 18, 2017 @Dan Cole On 6/7/2017 at 8:09 AM, Dan Cole said: @ArtcoInc Malcolm....do you still have these search modules set up somewhere, where you can easily turn them on and off? I just had an interesting conversation with a customer about our search function which didn't product any results when they didn't use a space in the appropriate spot....for example if you're looking for "Marine Flake" fish food on my site and entered "marineflake" it doesn't give you any results. I know it's looking for the full term as a single word and can't find it since it doesn't exist and I get that but it's not very smart. Just for the heck of it I tried it in google...the search engine that everyone is being educated to use and likely expect all searches to work the same and as you might expect google doesn't have an issue with it. "marineflake" will give you the expected results. So I'm left wondering if any of the available osC search modules might work in the same way as google when users don't add any spaces. Just wondering if you or anyone else reading this would happen to know? Dan @burt already answered you about his Fuzzy module. There is another way ... You can add Google Search to your site. You'd have to create an account with Google, add the bit of code to your site, and wait for them to crawl your site (or, submit a site map to Google to speed things up). I started to play with this, but didn't get too deep into it, so I don't know how smart it's search capability is, nor if it would be able to handle your "marineflake" example. Beyond these, I don't think that any of the osC search modules would be smart enough to handle your example. As far as I can tell, they all just search for the specified word (or words). Malcolm Quote Link to comment Share on other sites More sharing options...
ArtcoInc Posted June 18, 2017 Author Share Posted June 18, 2017 One other observation ... I find that I can get so involved when working on our stores that we 'know' how thing work, and may assume that our customers do too. For example ... When I was testing the Twitter Type-Ahead search function, I asked someone to try it and tell me what they thought. I watched them... 1) they clicked on the search box 2) they looked down at the keyboard as they typed their search term, and pressed enter (as they were typing, the search box had a number of suggestions, with the first one highlighted. When the user pressed Enter, it selected the highlighted suggestion) 3) they looked up at the screen, and wondered why search returned something other than what they wanted Just an observation .... Malcolm Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 @ArtcoInc Thanks for the input Malcolm. I haven't decided what to do yet. I had a look at the fuzzy search module mentioned by Gary but it's indexing didn't help any. That go me to thinking about how we did this in the old days which was to create our own index. I plan to look at that but just haven't gotten to it yet. Back in the day there were several programs available on the web that you could use to build the index. You would then adjust you search routine to use it for any queries. Google is certainly an option too. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
ArtcoInc Posted June 19, 2017 Author Share Posted June 19, 2017 @Dan Cole Have you looked at this one? I started to mess with it, but life got in the way. https://apps.oscommerce.com/caQMT It seems that the people who developed Sphider stopped working on it. There is a spin-off, but (if I remember correctly) it is a commercial product. That said, I was able to integrate the above add-on into the admin side, and was able to generate the index through the back end. I ran into snags on the front end (beyond my coding skills), though. HTH Malcolm Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 @ArtcoInc Thanks again Malcolm. I wasn't aware of that add on so I'll be sure to have a look at it. I see it was uploaded by @pbor1234 who I think still frequents these forums so maybe we can get an update on where it ended up. The challenge will likely be in finding suitable software to create the index...I saw mention of Sphider in my searches as well as what I'm thinking is the other one you were referring to -- Lucene but I don't know anything about either. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 (edited) @ArtcoInc Here is the support thread for the contribution....it looks like the development died along with Sphider. I'm not sure what the current status of Lucene is. Dan Edited June 19, 2017 by Dan Cole Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
burt Posted June 19, 2017 Share Posted June 19, 2017 @Dan Cole I just installed Fuzzy Search on a test site... Admin Side: Made a new product called "Marine Flake" Installed the module for advanced_search_result Shop Side: I searched for Marine Flake: 1 normal search result - expected I searched for marineflake: 0 results - expected 1 fuzzy result - "Marine Flake" I searched for neflake: 0 results - expected 1 fuzzy result - "Marine Flake" I searched for "rine flake": 1 result - expected This is with Fuzzy Factor set to 4. Quote Link to comment Share on other sites More sharing options...
ArtcoInc Posted June 19, 2017 Author Share Posted June 19, 2017 @Dan Cole As I mentioned, I was able to get the back end of Sphider to work. It basically goes through your site and indexes every word, and stores it in several database tables. Here's a screen shot from the admin side: For example, you can see that the word 'kevlar' is the most used word on my site. Several points: 1) I'm sure you *could* add words to the database, but they are spread out across multiple tables, with cross indexes, etc. I think it would take a bit of work to figure out how everything is stored and sorted. 2) This only indexes real words, and your 'marineflake' example probably would not find a match. 3) As I also said, I only got the back end working. I'm sure we *could* resolve the issues on the front end, but I'm not sure it would be worth the effort. As you noted, support for this has all but died. And, even if you/we were to revive it, I don't know if this search function is any better than other options. Finally, @burt has shown that his Fuzzy search does pretty much what you want. So, unless Google search installed on our stores offers better search capabilities, I think that @burt's Fuzzy is the current winner. Wow, this 'Search Review' sure has taken on a life of it's own. Who'd have thought? Malcolm Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 6 hours ago, burt said: @Dan Cole I just installed Fuzzy Search on a test site... Admin Side: Made a new product called "Marine Flake" Installed the module for advanced_search_result Shop Side: I searched for Marine Flake: 1 normal search result - expected I searched for marineflake: 0 results - expected 1 fuzzy result - "Marine Flake" I searched for neflake: 0 results - expected 1 fuzzy result - "Marine Flake" I searched for "rine flake": 1 result - expected This is with Fuzzy Factor set to 4. @burt That is interesting Gary....now you have me wondering what the heck I did....I'll revisit it. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 48 minutes ago, ArtcoInc said: Finally, @burt has shown that his Fuzzy search does pretty much what you want. So, unless Google search installed on our stores offers better search capabilities, I think that @burt's Fuzzy is the current winner. Wow, this 'Search Review' sure has taken on a life of it's own. Who'd have thought? Malcolm @ArtcoInc Thanks Malcolm....I'll have to revisit this again....I'm now not sure if I tried the example using the fuzzy search or if I was having other issues with it. I'll post back when I give it another go. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
ArtcoInc Posted June 19, 2017 Author Share Posted June 19, 2017 (I'm sure @burt will pull this if it is inappropriate ... ) For anyone else wanting to try @burt's Fuzzy search http://clubosc.com/29daysofcode/16.php Malcolm Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 1 hour ago, Dan Cole said: @burt That is interesting Gary....now you have me wondering what the heck I did....I'll revisit it. Dan @burt Gary I tried the fuzzy search again and it still failed to find marineflake on my site....in an effort to see what was going on I echoed the query to the screen and it shows the metaphone as '%MRNF%' without a space so I can see why it isn't being found. I haven't yet checked my database to see if "MRNF" was indexed but I'm off to check that now. Here is the actual output. select distinct p.products_id, p.products_image, pd.products_name from products p inner join products_description pd on (p.products_id = pd.products_id) where p.products_status = '1' and pd.language_id = '1' and pd.products_metaphone like '%MRNF%' order by pd.products_metaphone limit 4 Now I'm curious what the metaphone for your query was? Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 @burt Now I'm really puzzled....the product metaphone is.... OMKNMRNFLKW0KRLKK So I'm thinking it should have found MRNF using MYSQL LIKE but doesn't for some reason. I'm going to add a product with just the name Marine Flake and see what happens. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 19, 2017 Share Posted June 19, 2017 I tried adding a product with just the name Marine Flake and then regenerated the metaphones. No change. I then tried running the query against the database directly and it worked! It just won't work from the search bar...I'm out of ideas for what else I can do to trouble shoot it so I'll let it percolate for the time being. Hopefully an idea or two will surface. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 20, 2017 Share Posted June 20, 2017 As usual, my bad. I had added some breaks in the code to test it previously when it wasn't getting the results I expected. I left those breaks in place so the search wasn't getting past the query. Removing the breaks got me back to where I was. I then spend a bit more time with it and realized it was working but showing what seemed to be odd results. The results where limited to 4 and didn't seem to bare any relationship to the search query....I increased the limit to 20 and guess what showed up....just way down the list....yes, Marine Flakes both the original product as well as the new one I added. In summary it looks like the fuzzy search does work, it is just that there is no way to ensure that it will identify the most relevant results if you have a number of products containing the metaphone being searched for. The more products you have the less relevant the results will be. I'm now wondering if there is a way to sort or rework the results so the more relevant ones will appear higher in the list? I'll play around with the fuzzy factor a bit to see if that helps. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
burt Posted June 20, 2017 Share Posted June 20, 2017 It's difficult when you have a number of items that potentially match the Metaphone. The (in my opinion) logical route to go down would be to order the results by the "start position" of the search query. So let us imagine you have 3 products: Super Duper Fantastic Blue Widget Marine Flake Super Marine Flake Marine Flake And then someone searched for "marine flake" - use the normal search function as this is a match for all three products And then someone searched for "marineflake" - matches nothing using the normal search, but return results as follows in the Fuzzy; Marine Flake aka MRNF (fuzzy starts at position 1) Super Marine Flake aka SPRMRNF (fuzzy starts at position 4) Super Duper Fantastic Blue Widget Marine Flake SPRDPRFNTSTCBLWDGTMRNF (fuzzy starts position 19) At least you now have some order... That would be the way to do it I think. Quote Link to comment Share on other sites More sharing options...
burt Posted June 20, 2017 Share Posted June 20, 2017 ps playing with the Fuzzy Factor probably will help a bit but not massively so. All it does is change the %aaaa% bit below from 4 characters to whatever you choose: and pd.products_metaphone like '%MRNF%' order by The higher the number, the less likely to match in the metaphone string. This Fuzzy system is potentially one of the game changer addons, but it's something that is a bit of a blunt instrument (ie, helps a massive amount for an off the shelf piece of code that takes 1 minute to set up, but is not bespoke enough to be 100% perfect at all times). Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 20, 2017 Share Posted June 20, 2017 (edited) 5 hours ago, burt said: It's difficult when you have a number of items that potentially match the Metaphone. The (in my opinion) logical route to go down would be to order the results by the "start position" of the search query. So let us imagine you have 3 products: Super Duper Fantastic Blue Widget Marine Flake Super Marine Flake Marine Flake And then someone searched for "marine flake" - use the normal search function as this is a match for all three products And then someone searched for "marineflake" - matches nothing using the normal search, but return results as follows in the Fuzzy; Marine Flake aka MRNF (fuzzy starts at position 1) Super Marine Flake aka SPRMRNF (fuzzy starts at position 4) Super Duper Fantastic Blue Widget Marine Flake SPRDPRFNTSTCBLWDGTMRNF (fuzzy starts position 19) At least you now have some order... That would be the way to do it I think. @burt I like this Gary....I'm not sure how to order it in that way but I agree that makes sense to me too. Proof will be in the pudding. Quote This Fuzzy system is potentially one of the game changer addons, but it's something that is a bit of a blunt instrument (ie, helps a massive amount for an off the shelf piece of code that takes 1 minute to set up, but is not bespoke enough to be 100% perfect at all times). I think, as we are discovering here, it should be a excellent supplement to the regular search routine especially if we can order it appropriately. Dan Edited June 20, 2017 by Dan Cole Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
burt Posted June 22, 2017 Share Posted June 22, 2017 It's gotta be possible to do it. I haven't not thought about it other than my thoughts above. Quote Link to comment Share on other sites More sharing options...
Dan Cole Posted June 23, 2017 Share Posted June 23, 2017 @bruyndoncx We need help from our database wizard. Gary has developed an awesome fuzzy search module which can likely be improved even more with some refinements to the order of the results being displayed. His idea is described in this post. Thinking a bit about this I'm wondering if it would be possible to weight the individual results so that a sort order is assigned to them based on the position of the metaphone being searched for in the pd.products_metaphone field. That's a mouth full. The current query looks like this... select distinct p.products_id, p.products_image, pd.products_name from products p inner join products_description pd on (p.products_id = pd.products_id) where p.products_status = '1' and pd.language_id = '1' and pd.products_metaphone like '%MRNF%' order by pd.products_metaphone limit 4 I'm thinking something like if metaphone being searched for is in position 1 sort order = 1 or put another way megaphone sort order = starting metaphone position. Once the sort orders are determined then order by metaphone sort order. Is something like this possible in mysql? Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
♥bruyndoncx Posted June 26, 2017 Share Posted June 26, 2017 in mysql 5.6 you have the locate function with this as example, if you cannot order by immediately in the sql, you can wrap it in an extra sql clause and order in the outer clause SELECT pub_name,LOCATE('at',pub_name) FROM publisher WHERE locate('at',pub_name)>0; Quote KEEP CALM AND CARRY ON I do not use the responsive bootstrap version since i coded my responsive version earlier, but i have bought every 28d of code package to support burts effort and keep this forum alive (albeit more like on life support). So if you are still here ? What are you waiting for ?! Find the most frequent unique errors to fix: grep "PHP" php_error_log.txt | sed "s/^.* PHP/PHP/g" |grep "line" |sort | uniq -c | sort -r > counterrors.txt Link to comment Share on other sites More sharing options...
Dan Cole Posted June 26, 2017 Share Posted June 26, 2017 @bruyndoncx Thanks Carine. I think that will work! I'm thinking something like should work nicely. SELECT LOCATE("MRNF", pd.products_metaphone) AS MatchPosition FROM pd.products_description ORDER BY MatchPosition desc Now I just need to see what version of MYSQL I'm running on. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
Dan Cole Posted June 27, 2017 Share Posted June 27, 2017 Seems I'm running on an earlier version of MySQL so I figured I was out of luck then I spotted this on w3shools. Quote Works in: MySQL 5.7, MySQL 5.6, MySQL 5.5, MySQL 5.1, MySQL 5.0, MySQL 4.1, MySQL 4.0, MySQL 3.23 Happily it is back on my todo list. Dan Quote Need help? See this thread and provide the information requested. Is your version of osC up to date? You'll find the latest osC community version (CE Phoenix) here. Link to comment Share on other sites More sharing options...
ArtcoInc Posted September 12, 2017 Author Share Posted September 12, 2017 @burt Am I correct in that your fuzzy search function *only* searches the product name? Malcolm Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.