Author Topic: [DONE] [Suggestion] case sensitive quick search with non english letters  (Read 5210 times)

Offline Bartojan

  • Colonel
  • ****
  • Posts: 136
    • View Profile
Quick search (key Q in production, research…) works fine unless you are using non english letters, because non english letters are case sensitive.
When you search 'ipky' or 'iPkY' the game finds 'Šipky'.
When you search 'šip' it can not find anything.
Search 'Šip' finds 'Šipky'.

Window 10 (64bit)
OpenXcom Extended+ 3.9c (v2017-10-10)
« Last Edit: July 17, 2018, 02:49:19 pm by Meridian »

Offline Kammerer

  • Colonel
  • ****
  • Posts: 141
    • View Profile

Offline Bartojan

  • Colonel
  • ****
  • Posts: 136
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #2 on: November 24, 2017, 02:35:35 pm »
Oh. I don't like this 'feature'.

Offline tkzv

  • Commander
  • *****
  • Posts: 583
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #3 on: November 25, 2017, 12:54:43 am »
It's not a bug, it's a feature: https://openxcom.org/forum/index.php/topic,4520.msg65913.html#msg65913
By any chance, do you know which commit added the quick search? The fix may be something trivial like an example from a Unicode textbook.

I looked through the history of Meridian's commits for OXCE3.5+ branch since May, but titles don't mention this search.

Offline Kammerer

  • Colonel
  • ****
  • Posts: 141
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #4 on: November 25, 2017, 08:03:31 am »
By any chance, do you know which commit added the quick search?

Sure, here they are (from June 3 through June 4): https://github.com/MeridianOXC/OpenXcom/commits/oxce2.9-plus-proto

Offline tkzv

  • Commander
  • *****
  • Posts: 583
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #5 on: November 25, 2017, 05:30:13 pm »
Sure, here they are (from June 3 through June 4): https://github.com/MeridianOXC/OpenXcom/commits/oxce2.9-plus-proto
Thanks, I looked at a wrong branch.

Looks like most of this code is present in 3.5, except for the ability to switch the option. I wonder why.

It seems to use towupper(wchar_t) to transform the search string and each processed name to upperccase. This towupper manual gives an example, which correctly transforms š to Š if the locale is changed to en_US.utf8.

To make the search work with most non-English letters one can add a single std::setlocale(LC_ALL, "en_US.utf8"); at a strategic place. But that risks breaking something else. An alternative is adding it in all 23 places where towupper is called, followed by setlocale(LC_ALL, "C");

I'm going try it later today.

If a single lowercase letter corresponds to a sequence of uppercase letters (e.g. German "ß" to "SS") it is NOT converted. Similarly, typing "ss" you will never find "ß". This is doable, but would require much more work with string normalization.

Interestingly, when locale is "fr_FR.utf8" conversion does not happen. I don't know if it's system-specific or not. — Yes, it is, it is not set up properly in my system.

Update: Looks like replacing towupper(c) with toupper(c, std:locale("")) does the trick, if your system locale is UTF-8. There're 3 more complex iterators where this won't work, though. Does anybody know, is OpenXcom written in C++11? Or an earlier version? — Did it the old way — fewer lines to add.

Update 2: If your system locale is 7-bit ASCII, the conversion won't work for non-ASCII. No idea how it would work for system language on an 8-bit locale.

Update 3: Submitted to GitHub: https://github.com/tkzv/OpenXcom/commit/31594f7c7e45bf0c1e0868975ef6da67b6827650 Thanks to xaizek and fsb4000 at linux.org.ru.

If you compile from my branch, I recommend not to enable "Control civilians in battle" — it's buggy.

The search works fine, although it seems to give false positives in the inventory screen, especially for 1-letter queries. For example, it managed to find "ц" in "Heavy Plasma" :) Should do something about that.

Update 4: The bug isn't mine! The unpatched version managed to find "w" in "Bone Club" and over a hundred other items. Ergo — the patch is as workable as the rest of OXCE+.
« Last Edit: November 26, 2017, 02:55:53 pm by tkzv »

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3350
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #6 on: November 26, 2017, 02:32:36 pm »
OpenXcom require C++03
OpenXcomExtended and OpenXcomExtended+ require C++11 to compile
This is because I used some new functionality from next standard to implements something. Meridian keep his code compatible with C++03 to allow backport his functionality to vanilla OXC, but I do not know if he still plan to do it.

Offline tkzv

  • Commander
  • *****
  • Posts: 583
    • View Profile
Re: Minor bug: case sensitive quick search with non english letters
« Reply #7 on: November 26, 2017, 02:54:28 pm »
OpenXcom require C++03
OpenXcomExtended and OpenXcomExtended+ require C++11 to compile
This is because I used some new functionality from next standard to implements something. Meridian keep his code compatible with C++03 to allow backport his functionality to vanilla OXC, but I do not know if he still plan to do it.
It was possible to use lambdas, but in the end I decided to keep it easier to read and create a 1-line function (13 with headers, comments and whitespace). So, C++11 features were not needed. Another guy suggested something deprecated in 11, but I avoided that too.