Converting OXCE to rapidyaml

[Yankes]

My idea was find what is exactly is in `<vector>` when we compile android.
This would answer what go wrong with this.
This could even give us fix for "upstream" to support this specific library in rapidyaml itself.

[Delian]

In one of the c4core issues they get the same error, and they mention that _LIBCPP_ABI_NAMESPACE was only introduced with Clang 8.0.0-rc1.

You're using... NDK 18?
NDK r17 uses GCC
NDK r18 uses Clang 7.0.2
NDK r19 uses Clang 8.0.2
NDK r20 uses Clang 8.0.7
NDK r21 uses Clang 9.0.3

So, only NDK r18 is the issue because they don't use GCC anymore, and the _LIBCPP_ABI_NAMESPACE isn't introduced yet in Clang. How unlucky is that lol

[Meridian]

yes, using Clang 7.0.2 on NDK r18b

NDK r19 wasn't compatible when we tried some years ago... project compiled, but the game just crashed.

I can try NDK r20, r21, maybe it's still relatively easy to upgrade...

[Meridian]

I was able to use r19c, r20b and r21e.
(r22 and higher have some other structure and didn't work.)

All 3 versions I tried compiled without problems (and without any change in rapidyaml code).

However, none of them worked.
When I tried to run them on my phone (2-year old Samsung), all I got was a black screen


I'm pretty sure it's not a problem of those newer NDKs, or at least it is not caused only by them, because other people already compiled for example with r23b: https://github.com/MeridianOXC/openxcom-android/pull/3

...but until somebody gives me an idiot-proof guideline how to compile a working version with a newer SDK/NDK, I'm stuck with r18b, which is the only version that works for me.

"Custom library fix" is indeed annoying, but I'm willing to live with this one until there's a better solution.

[Delian]

Might avoid the custom library fix if you just manually defined _LIBCPP_ABI_NAMESPACE in one of our files in the android branch.
But yeah, sorry, I don't know much about building and testing android apps. If r22+ has a different structure, then how were they able to compile with r23b? Also, have you tried running their build on your phone?

[Meridian]

But yeah, sorry, I don't know much about building and testing android apps. If r22+ has a different structure, then how were they able to compile with r23b? Also, have you tried running their build on your phone?

They used a different SDK and modified the build files too, and also used a newer build plugin.
(might have even built it in Android Studio directly, instead of console, not sure)
I don't have their build result (APK file), so I cannot test it.

Eventually, I'll ask some colleague(s) from work to help me upgrade, but not now.

Ah, sorry, didn't read documentation carefully enough. As for making a suggestion, I can't, because I'm not a modder and it's not a QoL suggestion.

Congratulations, you've been added to the whitelist: https://openxcom.org/forum/index.php?topic=11631.msg166226#msg166226
Enjoy the new benefits (in moderation pls).

[Delian]

Maybe, after this, we'll be able to afford 1ms to always have "discovered" and "autoSales" vectors sorted in the sav files. It's been super annoying testing round trips due to randomization of those two sections on every save.

[Meridian]

If a yaml line contains any (trailing) tabs, they need to be removed
"  - a: 1         " is bad
"  - a: 1         #comment" is bad
"  - a: 1" is good
"  - a: 1 #comment" is good

I don't seem to get errors like this. (VS build)
Has something changed? (I use your last commit 78eab29)
Which file/line in the xcom1 ruleset could I use as an example to add a trailing tab?

[Delian]

Hmm. Oh, it only applies to flow-style lines. Not "a: 1" but "{a: 1}" or "[1, 2]"
You can try changing options.cfg, first mapping from
  - active: true
    id: piratez
to
 - {active: true, id: piratez} tab tab

Also, to mapping or sequence start:
mods: tab tab
  - active: true
This will also crash. However, I don't think this occurred in any of the mods.

[Meridian]

XCF, FMP and FMPE have now been updated to be compatible, we can use them for testing

[Delian]

I retested them, and they seem to be working fine.

In the mean time, besides the increase in saving and loading speed, my goal has also been to make the current debug build startup faster than the yaml-cpp release build startup.
130,011 ms for yaml-cpp debug build startup.
7,856 ms for yaml-cpp release build startup.
16,077 ms (-113,934) for rapidyaml debug build startup.
It's fast, but still 2x slower than the yaml-cpp release build. However, if we consider other extra optimizations that aren't related to rapidyaml...
14,533 ms (-1,544) by avoiding use of  stringstreams
12,408 ms (-2,125) with smarter sorting in ModScript constructor
9,661 ms (-2,747) by using script cache to avoid re-compilation of of built-in scripts.
8,667 ms (-994) with smarter sorting in Mod::sortLists
7,887 ms (-780) by setting Linker > General > Enable Incremental Linking to "No (/INCREMENTAL:NO)"
Only 30ms left to go! But can't find anything major anymore... let's see...
7,764 ms (-123) by optimizing SoldierNamePool::load (it's still not perfect tho, because it keeps loading one the same .nam files over and over)

Woo, I did it! And I didn't even have to bring out the big guns.

About the Linker thing, I dunno why we had that turned on. It doesn't make building any faster, only slows down performance by 10% and puffs up the .exe file by 30%. If I make a small modification to a .cpp file, then usually only this .obj file will have to be rebuilt. Any larger change will require the whole project to be rebuilt, with or without incremental building. Please tell me if you can demonstrate a type of code change where incremental building would reduce build time.

I haven't put the script caching code anywhere yet because I'm still testing it.

Oh yeah, another thing I wanted to mention. Game decompresses and stores all the sounds in RAM, bloating it from 390MB to 760MB. This happens even with Options::lazyLoadResources enabled. Is there any reason we need to keep all the sounds in RAM all the time?

[Delian]

XcomUtil_Statstrings/XcomUtil_Statstrings.rul:8:19 ERROR: Could not deserialize value to type <int>!

Hmm... fixed.

[Buscher]

While trying to get ROSIGMA to work with rapidyaml I ran into a problem with skills creating some kind of stack smashing. A small test mod next to pictures + log is attached.

I am building a debug build for Linux using this branch https://github.com/Yankes/OpenXcom/commits/rapidyam/
which currently is on this commit https://github.com/Yankes/OpenXcom/commit/d77c769ff147a7d0069f51d4baf54936b4295373

(ignore the Brutal stuff in the log in the pictures, I have exchanged the standard/common folder)

[Delian]

Besides the trailing whitespaces and incorrect float values, ROSIGMA had the following unusual errors:
- incorrectly defined missionBountyItem (expected string, found a sequence). But, this was also not properly reported by our api. Fixed, so that it detects when trying to deserialize a non-scalar into string or integral type.

- incorrectly defined craft marker (expected int, found a map). But, this was also not properly reported by out api. Also fixed.

Stack smashing! It turns out that hard casting an enum* to int* is not a good idea hehe. Because some of our enums are defined as char and unsigned char. Fixed.

[Buscher]

You probably are testing with the ROSIGMA version on mod.io. Meridian was already kind enough to create a pull request to fix the major part of it. The Repo is here

Also I can confirm that the stack smashing error is gone.