Problem in Shop search with special letters like ä ö ü

[mcmannehan]

There is a problem with the special letters like ä ö ü and more in the serach of osC 2.3.4. If some product name have special letter, than the search result

did'n find the product.

The mystery thing is, if you serach in the admin, products with special letters the search is working. The database is UTF8.

Anyone know some resolution for that, because i did try a lot filter stuff but it's not working.

[beerbee]

Hi @mcmannehan,

I had this problem because I have Security Pro installed https://apps.oscommerce.com/o19Sn&security-pro-2-0-r7

which kicked out any umlauts in the search. If that is the case you could change in includes/modules/fwr_media_security_pro.php both instances of  this:

$cleansed = preg_replace ( "/[^\s{}a-z0-9_\.\-]/i", "", urldecode ( $string ) );

to that:

$cleansed = preg_replace ( "/[^\s{}äÄöÖüÜßa-z0-9_\.\-]/i", "", urldecode ( $string ) );

And save as UTF8 without BOM.

Best regards

Christoph

[mcmannehan]

@beerbee

Thank for the answer. No i don't have installed SecurityPro. The special letters don't get kick aut. I see them in the $_GET['search']. and it's only in the Shop search. At the admin the search is working with the special letters. It's not only ä ü ö ß it's all special letters also á à or é è and so on and so on.

In the Shop search the search have a loop for every letter, in the admin don't have.

Shop Search:

if (isset($search_keywords) && (sizeof($search_keywords) > 0)) {
    $where_str .= " and (";
    for ($i=0, $n=sizeof($search_keywords); $i<$n; $i++ ) {
      switch ($search_keywords[$i]) {
        case '(':
        case ')':
        case 'and':
        case 'or':
          $where_str .= " " . $search_keywords[$i] . " ";
          break;
        default:
          $keyword = tep_db_prepare_input($search_keywords[$i]);
          $where_str .= "(";
          if ( (defined('MODULE_HEADER_TAGS_PRODUCT_META_KEYWORDS_STATUS')) && (MODULE_HEADER_TAGS_PRODUCT_META_KEYWORDS_STATUS != 'Meta') ) {
            $where_str .= "pd.products_seo_keywords LIKE '%" . tep_db_input($keyword) . "%' OR ";
          }
          $where_str .= "pd.products_name like '%" . tep_db_input($keyword) . "%' or p.products_model like '%" . tep_db_input($keyword) . "%' or m.manufacturers_name like '%" . tep_db_input($keyword) . "%'";
          if (isset($_GET['search_in_description']) && ($_GET['search_in_description'] == '1')) $where_str .= " or pd.products_description like '%" . tep_db_input($keyword) . "%'";
          $where_str .= ')';
          break;
      }
    }
    $where_str .= " )";
  }

Admin Search

if (isset($_GET['search'])) {
      $products_query = tep_db_query("select p.products_id, pd.products_name, p.products_quantity, p.products_image, p.products_price, p.products_date_added, p.products_last_modified, p.products_date_available, p.products_status, p2c.categories_id from " . TABLE_PRODUCTS . " p, " . TABLE_PRODUCTS_DESCRIPTION . " pd, " . TABLE_PRODUCTS_TO_CATEGORIES . " p2c where p.products_id = pd.products_id and pd.language_id = '" . (int)$languages_id . "' and p.products_id = p2c.products_id and pd.products_name like '%" . tep_db_input($search) . "%' order by pd.products_name");
    } else {
      $products_query = tep_db_query("select p.products_id, pd.products_name, p.products_quantity, p.products_image, p.products_price, p.products_date_added, p.products_last_modified, p.products_date_available, p.products_status from " . TABLE_PRODUCTS . " p, " . TABLE_PRODUCTS_DESCRIPTION . " pd, " . TABLE_PRODUCTS_TO_CATEGORIES . " p2c where p.products_id = pd.products_id and pd.language_id = '" . (int)$languages_id . "' and p.products_id = p2c.products_id and p2c.categories_id = '" . (int)$current_category_id . "' order by pd.products_name");
    }

I think i will complett reprogramming the Shop search, because this don't work with some server.

[MrPhil]

Are you sure that you have compatible encodings throughout your store, from user input to database? I can't see anything in the code (including utility routines) that would exclude non-ASCII characters, so it's most likely a character encoding mismatch (e.g., e-acute as Latin-1 input xE9 not matching database UTF-8 xC3A9). Be sure to check your database fields (in phpMyAdmin browse) to confirm that they are actually in UTF-8, if that's what you have them declared as (rather than the default Latin-1).

Regarding Security Pro,

$cleansed = preg_replace ( "/[^\s{}a-z0-9_\.\-]/i", "", urldecode ( $string ) );

that makes no sense (and gives no security) to exclude all but a subset of ASCII characters. This prevents any accented or non-Latin character from working. It would be better to check for any forbidden characters (if there are any), or just replace with

$cleansed = urldecode($string);

if there are no forbidden characters.

[beerbee]

All php files should be also saved in UTF8.

Best regards

Christoph

[MrPhil]

35 minutes ago, beerbee said:

All php files should be also saved in UTF8.

If they were edited to insert UTF-8 (non-ASCII) characters. Note that it is undesirable to include any non-ASCII characters or text in PHP files, except for "language" files, where the encoding (UTF-8, etc.) is clearly specified. Sometimes this can't be helped, but try to avoid doing this if you can.

[tgely]

All questions start again and again...

hundreds of topics started in this theme. Php doesnt transpose utf-8 characters to mysql if multibyte internal encoding is not enabled or assured..

The main problem comes from:

tep_parse_search_string()

which can not serve utf-8 arrays.

A solution example like this:
 

////
// Parse search string into indivual objects
  function tep_parse_search_string($search_str = '', &$objects) {
    $search_str = trim( mb_detect_encoding((strtolower($search_str)), 'UTF-8', true) ? strtolower($search_str) : mb_strtolower($search_str) );