Author Topic: [FIXED] Crashes related to ruleset file load order  (Read 1487 times)

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #15 on: December 16, 2024, 10:01:20 pm »
I have found only two usages outside of this file, and they are probably mistakes.

So cruel to call them mistakes! They work fine! (as long as file loading is z-a)

And here's an example of some problems resulting from different locales: https://openxcom.org/forum/index.php?topic=5146.msg83361#msg83361

...

...

No, that problem wasn't due to different locales. It was due to duplicate folder names between common/SoldierName and Celebrate_Diversity/SoldierName.
The problem was that common/SoldierName has "argentina.nam", but Celebrate_Diversity does not (it has "argentinian.nam"). So after merging the two virtual dirs, this extra file causes the offset.
Note that, even though we were already using std::sort() for rul files back then, that doesn't apply to soldier names. For soldier names we explicitly call a locale-aware sort.
« Last Edit: December 16, 2024, 10:30:42 pm by Delian »

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9197
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #16 on: December 17, 2024, 09:47:56 am »
No, that problem wasn't due to different locales.

Yes, it WAS also due to different locales!
Don't make me an idiot.

Both duplicate folder and locale issue needed to be fixed.
Fixing just one still lead to offset issues.
Different problems, same result for the player... dysfunctional mod.
« Last Edit: December 17, 2024, 09:49:55 am by Meridian »

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #17 on: December 17, 2024, 02:53:19 pm »
Ok, but back then soldier name files were not yet being sorted locale-aware. Stoddard only made namepool sorting locale-aware 5 years ago.

Yes, I agree that there are some problems resulting from sorting in different locales. I am with you on that.

For demonstration:
Let's say that a modder defines "Flag25" to be Resources/Flags/Kenya.png.
How did the modder determine that Flag25 is Kenya? He looked at the SoldierNames (or whatever) folder and saw that the 26th file was Kenyan.nam
So nationality with index 25 is Kenya. Soldier with _nationality = 25 will get this flag.
Soldier also gets a name from names[_nationality]. So the name pool with index 25 will be used for the name. names vector was generated after locale-aware sorting files.
As you can guess, this will fail in other locales, because there's no guarantee that Kenyan.nam will be the 26th file after applying a locale-aware sort.
I don't know why Stoddard made name pool sorting locale-aware. It seems like a really bad idea to do that :D
To being with, anything (_nationality included) referring to the index position of a file in a folder is too fragile and is simply bound to fail one way or another. Add or remove a file and it fails.

So, uhh, what were we talking about again? Ruleset load order... yeah... so there's an issue that a lot of mods rely on the order the rul files are loaded in, while modders don't have any ability to specify what the order is. If you wish, I can try implementing something that would give modders some control over this order.

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9197
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #18 on: December 28, 2024, 05:19:48 pm »
Since 2014, I haven't seen or heard even a single false positive from this check. All reports were correctly found issues.

But you are technically correct, we can move it to afterLoad.

Check was moved: https://github.com/MeridianOXC/OpenXcom/commit/f3bec17944951f542c607a7b18564f60e0106d30

--

As for your proposed bugfix for sorting: https://github.com/Delian0/OpenXcom/commit/fb548b79a59b2731077d8286475396e9123560b0

It changed sorting not only for zipped mods, but also for folder mods, see attached example.
For example "xyz.rul" and "xyz/ppp.rul" are sorted differently before and after the change.
Is that intentional?
If yes, I don't mind, as I said we don't guarantee any particular file load order (and won't guarantee it in the future either).
(Just mentioning it will be a breaking change.)

All I am interested in is that on the same environment, under the same conditions, both folder and zipped mods would have the same file load order... which seems to be the case after the change.

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3394
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #19 on: December 28, 2024, 05:43:30 pm »
I think files should be first, before all folders. I think I could create some function that make better sorting like that.

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #20 on: December 28, 2024, 06:59:09 pm »
Logically, '/' has a higher ascii code than '.', so in z-a sorting it will appear first. The intention behind this simple bugfix was to make sorting consistent. But as you show, applying the same sorting after the VFS loading is done will produce different results, which I wasn't aware of - that wasn't my intention.

I'm not sure what type of sorting would be the best. If it was up to me, I'd make it...

Breadth-first means all files in the current folder before moving into any subfolders.

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3394
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #21 on: December 28, 2024, 09:14:33 pm »
Right now I working on helper class that split `std::string_view` based on `/` and compare number of parts and then string parts on the fly.

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #22 on: December 29, 2024, 08:03:09 pm »
Before, with code
Code: [Select]
std::vector<FileMap::FileRecord> sortedRulesetFiles = rulesetFiles;
std::sort(sortedRulesetFiles.begin(), sortedRulesetFiles.end(),
[](const FileMap::FileRecord& a, const FileMap::FileRecord& b)
{
return a.fullpath > b.fullpath;
});
result:
Code: [Select]
/rosigma/Ruleset/vars_ROSIGMA.rul
/rosigma/Ruleset/ufos.rul
/rosigma/Ruleset/ufopedia40k.rul
/rosigma/Ruleset/ufoTrajectories.rul
/rosigma/Ruleset/terrain_40k.rul
/rosigma/Ruleset/tags_ROSIGMA.rul
/rosigma/Ruleset/startingConditions_40k.rul
/rosigma/Ruleset/startingBase40k.rul
/rosigma/Ruleset/soldierBonuses.rul
/rosigma/Ruleset/scripts_40k.rul
/rosigma/Ruleset/scripts/scripts_turret_reload.rul
/rosigma/Ruleset/scripts/scripts_tenacious_enemies.rul
...
/rosigma/Ruleset/scripts/scripts_abilities_miracles.rul
/rosigma/Ruleset/scripts/scripts_abilities_commands.rul
/rosigma/Ruleset/scripts/scripts_UFO_visual_confirmed.rul
/rosigma/Ruleset/scripts/scripts_TU_accuracy_penalty.rul
/rosigma/Ruleset/scripts/script_dig_in.rul
/rosigma/Ruleset/scripts/Scripts_remove_flight.rul
/rosigma/Ruleset/scripts/IGMA_scripts_change_color_ammo.rul
/rosigma/Ruleset/scripts/IGMA_scripts_buff.rul
/rosigma/Ruleset/research_ROSIGMA.rul
/rosigma/Ruleset/regions.rul
/rosigma/Ruleset/palette.rul
/rosigma/Ruleset/musics.rul
/rosigma/Ruleset/missionsScripts.rul
/rosigma/Ruleset/mapScripts.rul
/rosigma/Ruleset/itemCategories.rul
/rosigma/Ruleset/invs.rul
/rosigma/Ruleset/interfaces.rul
/rosigma/Ruleset/globe40k.rul
/rosigma/Ruleset/facilities_ROSIGMA.rul
/rosigma/Ruleset/extraSprites_gibs_of_gory.rul
/rosigma/Ruleset/extraSprites_HANDOB.rul
/rosigma/Ruleset/extraSprites.rul
/rosigma/Ruleset/extraSounds.rul
/rosigma/Ruleset/events.rul
/rosigma/Ruleset/eventScripts.rul
/rosigma/Ruleset/eldar.rul
/rosigma/Ruleset/commendations.rul
/rosigma/Ruleset/arcScripts_ROSIGMA.rul
/rosigma/Ruleset/alienRaces_40k.rul
/rosigma/Ruleset/alienMissions_40k.rul
/rosigma/Ruleset/alienItemLevels.rul
/rosigma/Ruleset/alienDeployments_40k.rul
/rosigma/Ruleset/advancedTurrets.rul
/rosigma/Ruleset/Voices_40k.rul
/rosigma/Ruleset/SM/ufopaedia_SM.rul
/rosigma/Ruleset/SM/startingBase_SM.rul
/rosigma/Ruleset/SM/soldiers_SM.rul
/rosigma/Ruleset/SM/soldierTransformations.rul
/rosigma/Ruleset/SM/scripts_SM.rul
/rosigma/Ruleset/SM/manufacture_SM.rul
/rosigma/Ruleset/SM/manufacture_CSM.rul
/rosigma/Ruleset/SM/items_scripts_SM.rul
/rosigma/Ruleset/SM/items_SM.rul
/rosigma/Ruleset/SM/events_SM.rul
/rosigma/Ruleset/SM/crafts_SM.rul
/rosigma/Ruleset/SM/armors_SM.rul
/rosigma/Ruleset/SM/PRIMARIS/ufopaedia_PRIMARIS.rul
/rosigma/Ruleset/SM/PRIMARIS/manufacture_PRIMARIS.rul
/rosigma/Ruleset/SM/PRIMARIS/armors_PRIMARIS.rul
/rosigma/Ruleset/SM/Deathwatch/armors_voices_DW.rul
/rosigma/Ruleset/SM/Deathwatch/armors_layers_DW.rul
/rosigma/Ruleset/SM/Deathwatch/Deathwatch.rul
/rosigma/Ruleset/OTHER/manufacture_killpoints.rul
/rosigma/Ruleset/OTHER/corpses_killpoints.rul
/rosigma/Ruleset/OTHER/corpses_civ.rul
/rosigma/Ruleset/OTHER/captures_civ.rul
/rosigma/Ruleset/OTHER/alienDeployments_killpoints.rul
/rosigma/Ruleset/OTHER/alertSounds_disable.rul
/rosigma/Ruleset/IG/weapons_IG.rul
/rosigma/Ruleset/IG/units_IG.rul
/rosigma/Ruleset/IG/ufos_IG.rul
/rosigma/Ruleset/IG/ufopaedia_IG.rul
/rosigma/Ruleset/IG/training_mission_IG.rul
/rosigma/Ruleset/IG/terrains_IG.rul
/rosigma/Ruleset/IG/strategies/missionScripts_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/manufacture_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/items_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/events_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/eventScripts_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/crafts_strategies_IG.rul
/rosigma/Ruleset/IG/startingBase_IG.rul
/rosigma/Ruleset/IG/soldiers_command_IG.rul
/rosigma/Ruleset/IG/soldiers_IG.rul
/rosigma/Ruleset/IG/soldierTransformation_IG.rul
/rosigma/Ruleset/IG/skills_IG.rul
/rosigma/Ruleset/IG/missions_IG.rul
/rosigma/Ruleset/IG/manufacture_IG.rul
/rosigma/Ruleset/IG/gibs_of_gory_IG.rul
/rosigma/Ruleset/IG/events_IG.rul
/rosigma/Ruleset/IG/eventScripts_IG.rul
/rosigma/Ruleset/IG/crafts_IG.rul
/rosigma/Ruleset/IG/corpses_IG.rul
/rosigma/Ruleset/IG/armors_voices_IG.rul
/rosigma/Ruleset/IG/armors_scripts_IG.rul
/rosigma/Ruleset/IG/armors_layers_IG.rul
/rosigma/Ruleset/IG/armors_IG.rul
/rosigma/Ruleset/IG/alienDeployments_IG.rul
/rosigma/Ruleset/ENEMY/zombification_weapons.rul
/rosigma/Ruleset/ENEMY/weapons_daemons.rul
/rosigma/Ruleset/ENEMY/weapons_chaos.rul
/rosigma/Ruleset/ENEMY/ufopaedia_chaos.rul
/rosigma/Ruleset/ENEMY/items_spawner.rul
/rosigma/Ruleset/ENEMY/corpses_chaos.rul
/rosigma/Ruleset/ENEMY/captures_chaos.rul
/rosigma/Ruleset/ENEMY/armors_daemon.rul
/rosigma/Ruleset/ENEMY/armors_cultist.rul
/rosigma/Ruleset/ENEMY/armors_chaos.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/weapons_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/corpses_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/captures_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/armors_tzeentch.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/corpses_traitor_guard.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/captures_traitor_guard.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/armors_traitor_guard.rul
/rosigma/Ruleset/ENEMY/SLAANESH/weapons_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/corpses_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/captures_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/armors_slaanesh.rul
/rosigma/Ruleset/ENEMY/ORK/weapons_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/units_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/ufopaedia_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/corpses_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/captures_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/armors_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/armor_scripts_ORK.rul
/rosigma/Ruleset/ENEMY/NURGLE/units_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/corpses_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/captures_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/armors_nurgle.rul
/rosigma/Ruleset/ENEMY/NECRON/units_necron.rul
/rosigma/Ruleset/ENEMY/NECRON/score_necrons.rul
/rosigma/Ruleset/ENEMY/NECRON/gibs_of_gory_necrons.rul
/rosigma/Ruleset/ENEMY/KHORNE/corpses_khorne.rul
/rosigma/Ruleset/ENEMY/KHORNE/captures_khorne.rul
/rosigma/Ruleset/ENEMY/KHORNE/armors_khorne.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/corpses_hereticsisters.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/captures_hereticsisters.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/armors_hereticsisters.rul
/rosigma/Ruleset/ENEMY/GENE/weapons_gene.rul
/rosigma/Ruleset/ENEMY/GENE/units_gene.rul
/rosigma/Ruleset/ENEMY/GENE/corpses_gsc.rul
/rosigma/Ruleset/ENEMY/GENE/captures_gsc.rul
/rosigma/Ruleset/ENEMY/GENE/armors_gene.rul
/rosigma/Ruleset/ENEMY/DESERTERS/corpses_deserters.rul
/rosigma/Ruleset/ENEMY/DESERTERS/captures_deserters.rul
/rosigma/Ruleset/Difficulty/extraSprites_DifficultyMod_Bakgrounds.rul
/rosigma/Ruleset/CHAMBER/weapons_INQ.rul
/rosigma/Ruleset/CHAMBER/ufopaedia_INQ.rul
/rosigma/Ruleset/CHAMBER/startingBase_INQ.rul
/rosigma/Ruleset/CHAMBER/soldiers_INQ.rul
/rosigma/Ruleset/CHAMBER/manufacture_INQ.rul
/rosigma/Ruleset/CHAMBER/events_INQ.rul
/rosigma/Ruleset/CHAMBER/crafts_INQ.rul
/rosigma/Ruleset/CHAMBER/armors_INQ.rul
/rosigma/Ruleset/CHAMBER/INQUISITION_CORPSES.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/soldierTransformations_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/crafts_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_scripts_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_layers_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armor_sprites.rul
/rosigma/Ruleset/CHAMBER/DEATHWATCH/ufopaedia_DW.rul
/rosigma/Ruleset/BALANCE/special_ammo.rul
/rosigma/Ruleset/BALANCE/special.rul
/rosigma/Ruleset/BALANCE/plasma_ammo.rul
/rosigma/Ruleset/BALANCE/plasma.rul
/rosigma/Ruleset/BALANCE/melta_ammo.rul
/rosigma/Ruleset/BALANCE/melta.rul
/rosigma/Ruleset/BALANCE/melee.rul
/rosigma/Ruleset/BALANCE/las_ammo.rul
/rosigma/Ruleset/BALANCE/las.rul
/rosigma/Ruleset/BALANCE/incendiaries_ammo.rul
/rosigma/Ruleset/BALANCE/incendiaries.rul
/rosigma/Ruleset/BALANCE/grav_ammo.rul
/rosigma/Ruleset/BALANCE/grav.rul
/rosigma/Ruleset/BALANCE/explosives.rul
/rosigma/Ruleset/BALANCE/ballistics_arbites.rul
/rosigma/Ruleset/BALANCE/ballistics_ammo.rul
/rosigma/Ruleset/BALANCE/ballistics_allfactions.rul
/rosigma/Ruleset/BALANCE/ballistics_adeptas.rul
/rosigma/Ruleset/BALANCE/ballistics_SM.rul
/rosigma/Ruleset/BALANCE/ballistics_IG.rul
/rosigma/Ruleset/ARBITES/ufopaedia_arbites.rul
/rosigma/Ruleset/ARBITES/startingBase_arbites.rul
/rosigma/Ruleset/ARBITES/soldiers_arbites.rul
/rosigma/Ruleset/ARBITES/soldierTransformation_arbites.rul
/rosigma/Ruleset/ARBITES/manufacture_arbites.rul
/rosigma/Ruleset/ARBITES/items_arbites.rul
/rosigma/Ruleset/ARBITES/crafts_arbites.rul
/rosigma/Ruleset/ARBITES/armors_voices_arbites.rul
/rosigma/Ruleset/ARBITES/armors_scripts_arbites.rul
/rosigma/Ruleset/ARBITES/armors_arbites.rul
/rosigma/Ruleset/ALLFACTIONS/weapons_turrets.rul
...
/rosigma/Ruleset/ALLFACTIONS/armors.rul
/rosigma/Ruleset/ADEPTAS/weapons_adeptas.rul
...
/rosigma/Ruleset/ADEPTAS/adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/manufacture_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/items_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/events_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/eventScripts_strategies_adeptas.rul
/rosigma/Ruleset/40k.rul


after, with code
Code: [Select]
std::string bufA, bufB;
std::vector<FileMap::FileRecord> sortedRulesetFiles = rulesetFiles;
std::sort(sortedRulesetFiles.begin(), sortedRulesetFiles.end(),
[&](const FileMap::FileRecord& a, const FileMap::FileRecord& b)
{
size_t fileNamePosA = a.fullpath.find_last_of("/");
size_t fileNamePosB = b.fullpath.find_last_of("/");
std::string_view dirpathA(a.fullpath.data(), fileNamePosA);
std::string_view dirpathB(b.fullpath.data(), fileNamePosB);
if (dirpathA == dirpathB)
{
// files are in the same folder -> compare filenames
std::string_view filenameA = a.fullpath;
std::string_view filenameB = b.fullpath;
filenameA.remove_prefix(fileNamePosA + 1);
filenameB.remove_prefix(fileNamePosB + 1);
bufA.assign(filenameA);
bufB.assign(filenameB);
}
else
{
// files are in different folders -> compare folders
bufA.assign(dirpathA);
bufB.assign(dirpathB);
}
Unicode::lowerCase(bufA);
Unicode::lowerCase(bufB);
return bufA < bufB;
});
result:
Code: [Select]
/rosigma/Ruleset/40k.rul
/rosigma/Ruleset/advancedTurrets.rul
/rosigma/Ruleset/alienDeployments_40k.rul
/rosigma/Ruleset/alienItemLevels.rul
/rosigma/Ruleset/alienMissions_40k.rul
/rosigma/Ruleset/alienRaces_40k.rul
/rosigma/Ruleset/arcScripts_ROSIGMA.rul
/rosigma/Ruleset/commendations.rul
/rosigma/Ruleset/eldar.rul
/rosigma/Ruleset/events.rul
/rosigma/Ruleset/eventScripts.rul
/rosigma/Ruleset/extraSounds.rul
/rosigma/Ruleset/extraSprites.rul
/rosigma/Ruleset/extraSprites_gibs_of_gory.rul
/rosigma/Ruleset/extraSprites_HANDOB.rul
/rosigma/Ruleset/facilities_ROSIGMA.rul
/rosigma/Ruleset/globe40k.rul
/rosigma/Ruleset/interfaces.rul
/rosigma/Ruleset/invs.rul
/rosigma/Ruleset/itemCategories.rul
/rosigma/Ruleset/mapScripts.rul
/rosigma/Ruleset/missionsScripts.rul
/rosigma/Ruleset/musics.rul
/rosigma/Ruleset/palette.rul
/rosigma/Ruleset/regions.rul
/rosigma/Ruleset/research_ROSIGMA.rul
/rosigma/Ruleset/scripts_40k.rul
/rosigma/Ruleset/soldierBonuses.rul
/rosigma/Ruleset/startingBase40k.rul
/rosigma/Ruleset/startingConditions_40k.rul
/rosigma/Ruleset/tags_ROSIGMA.rul
/rosigma/Ruleset/terrain_40k.rul
/rosigma/Ruleset/ufopedia40k.rul
/rosigma/Ruleset/ufos.rul
/rosigma/Ruleset/ufoTrajectories.rul
/rosigma/Ruleset/vars_ROSIGMA.rul
/rosigma/Ruleset/Voices_40k.rul
/rosigma/Ruleset/ADEPTAS/adeptas.rul
...
/rosigma/Ruleset/ADEPTAS/weapons_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/events_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/eventScripts_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/items_strategies_adeptas.rul
/rosigma/Ruleset/ADEPTAS/STRATEGIES/manufacture_strategies_adeptas.rul
/rosigma/Ruleset/ALLFACTIONS/armors.rul
...
/rosigma/Ruleset/ALLFACTIONS/weapons_turrets.rul
/rosigma/Ruleset/ARBITES/armors_arbites.rul
/rosigma/Ruleset/ARBITES/armors_scripts_arbites.rul
/rosigma/Ruleset/ARBITES/armors_voices_arbites.rul
/rosigma/Ruleset/ARBITES/crafts_arbites.rul
/rosigma/Ruleset/ARBITES/items_arbites.rul
/rosigma/Ruleset/ARBITES/manufacture_arbites.rul
/rosigma/Ruleset/ARBITES/soldiers_arbites.rul
/rosigma/Ruleset/ARBITES/soldierTransformation_arbites.rul
/rosigma/Ruleset/ARBITES/startingBase_arbites.rul
/rosigma/Ruleset/ARBITES/ufopaedia_arbites.rul
/rosigma/Ruleset/BALANCE/ballistics_adeptas.rul
/rosigma/Ruleset/BALANCE/ballistics_allfactions.rul
/rosigma/Ruleset/BALANCE/ballistics_ammo.rul
/rosigma/Ruleset/BALANCE/ballistics_arbites.rul
/rosigma/Ruleset/BALANCE/ballistics_IG.rul
/rosigma/Ruleset/BALANCE/ballistics_SM.rul
/rosigma/Ruleset/BALANCE/explosives.rul
/rosigma/Ruleset/BALANCE/grav.rul
/rosigma/Ruleset/BALANCE/grav_ammo.rul
/rosigma/Ruleset/BALANCE/incendiaries.rul
/rosigma/Ruleset/BALANCE/incendiaries_ammo.rul
/rosigma/Ruleset/BALANCE/las.rul
/rosigma/Ruleset/BALANCE/las_ammo.rul
/rosigma/Ruleset/BALANCE/melee.rul
/rosigma/Ruleset/BALANCE/melta.rul
/rosigma/Ruleset/BALANCE/melta_ammo.rul
/rosigma/Ruleset/BALANCE/plasma.rul
/rosigma/Ruleset/BALANCE/plasma_ammo.rul
/rosigma/Ruleset/BALANCE/special.rul
/rosigma/Ruleset/BALANCE/special_ammo.rul
/rosigma/Ruleset/CHAMBER/armors_INQ.rul
/rosigma/Ruleset/CHAMBER/crafts_INQ.rul
/rosigma/Ruleset/CHAMBER/events_INQ.rul
/rosigma/Ruleset/CHAMBER/INQUISITION_CORPSES.rul
/rosigma/Ruleset/CHAMBER/manufacture_INQ.rul
/rosigma/Ruleset/CHAMBER/soldiers_INQ.rul
/rosigma/Ruleset/CHAMBER/startingBase_INQ.rul
/rosigma/Ruleset/CHAMBER/ufopaedia_INQ.rul
/rosigma/Ruleset/CHAMBER/weapons_INQ.rul
/rosigma/Ruleset/CHAMBER/DEATHWATCH/ufopaedia_DW.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armor_sprites.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_layers_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/armors_scripts_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/crafts_GK.rul
/rosigma/Ruleset/CHAMBER/GREY KNIGHTS/soldierTransformations_GK.rul
/rosigma/Ruleset/Difficulty/extraSprites_DifficultyMod_Bakgrounds.rul
/rosigma/Ruleset/ENEMY/armors_chaos.rul
/rosigma/Ruleset/ENEMY/armors_cultist.rul
/rosigma/Ruleset/ENEMY/armors_daemon.rul
/rosigma/Ruleset/ENEMY/captures_chaos.rul
/rosigma/Ruleset/ENEMY/corpses_chaos.rul
/rosigma/Ruleset/ENEMY/items_spawner.rul
/rosigma/Ruleset/ENEMY/ufopaedia_chaos.rul
/rosigma/Ruleset/ENEMY/weapons_chaos.rul
/rosigma/Ruleset/ENEMY/weapons_daemons.rul
/rosigma/Ruleset/ENEMY/zombification_weapons.rul
/rosigma/Ruleset/ENEMY/DESERTERS/captures_deserters.rul
/rosigma/Ruleset/ENEMY/DESERTERS/corpses_deserters.rul
/rosigma/Ruleset/ENEMY/GENE/armors_gene.rul
/rosigma/Ruleset/ENEMY/GENE/captures_gsc.rul
/rosigma/Ruleset/ENEMY/GENE/corpses_gsc.rul
/rosigma/Ruleset/ENEMY/GENE/units_gene.rul
/rosigma/Ruleset/ENEMY/GENE/weapons_gene.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/armors_hereticsisters.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/captures_hereticsisters.rul
/rosigma/Ruleset/ENEMY/HERETICSISTER/corpses_hereticsisters.rul
/rosigma/Ruleset/ENEMY/KHORNE/armors_khorne.rul
/rosigma/Ruleset/ENEMY/KHORNE/captures_khorne.rul
/rosigma/Ruleset/ENEMY/KHORNE/corpses_khorne.rul
/rosigma/Ruleset/ENEMY/NECRON/gibs_of_gory_necrons.rul
/rosigma/Ruleset/ENEMY/NECRON/score_necrons.rul
/rosigma/Ruleset/ENEMY/NECRON/units_necron.rul
/rosigma/Ruleset/ENEMY/NURGLE/armors_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/captures_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/corpses_nurgle.rul
/rosigma/Ruleset/ENEMY/NURGLE/units_nurgle.rul
/rosigma/Ruleset/ENEMY/ORK/armor_scripts_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/armors_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/captures_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/corpses_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/ufopaedia_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/units_ORK.rul
/rosigma/Ruleset/ENEMY/ORK/weapons_ORK.rul
/rosigma/Ruleset/ENEMY/SLAANESH/armors_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/captures_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/corpses_slaanesh.rul
/rosigma/Ruleset/ENEMY/SLAANESH/weapons_slaanesh.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/armors_traitor_guard.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/captures_traitor_guard.rul
/rosigma/Ruleset/ENEMY/TRAITOR_GUARD/corpses_traitor_guard.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/armors_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/captures_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/corpses_tzeentch.rul
/rosigma/Ruleset/ENEMY/TZEENTCH/weapons_tzeentch.rul
/rosigma/Ruleset/IG/alienDeployments_IG.rul
/rosigma/Ruleset/IG/armors_IG.rul
/rosigma/Ruleset/IG/armors_layers_IG.rul
/rosigma/Ruleset/IG/armors_scripts_IG.rul
/rosigma/Ruleset/IG/armors_voices_IG.rul
/rosigma/Ruleset/IG/corpses_IG.rul
/rosigma/Ruleset/IG/crafts_IG.rul
/rosigma/Ruleset/IG/events_IG.rul
/rosigma/Ruleset/IG/eventScripts_IG.rul
/rosigma/Ruleset/IG/gibs_of_gory_IG.rul
/rosigma/Ruleset/IG/manufacture_IG.rul
/rosigma/Ruleset/IG/missions_IG.rul
/rosigma/Ruleset/IG/skills_IG.rul
/rosigma/Ruleset/IG/soldiers_command_IG.rul
/rosigma/Ruleset/IG/soldiers_IG.rul
/rosigma/Ruleset/IG/soldierTransformation_IG.rul
/rosigma/Ruleset/IG/startingBase_IG.rul
/rosigma/Ruleset/IG/terrains_IG.rul
/rosigma/Ruleset/IG/training_mission_IG.rul
/rosigma/Ruleset/IG/ufopaedia_IG.rul
/rosigma/Ruleset/IG/ufos_IG.rul
/rosigma/Ruleset/IG/units_IG.rul
/rosigma/Ruleset/IG/weapons_IG.rul
/rosigma/Ruleset/IG/strategies/crafts_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/events_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/eventScripts_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/items_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/manufacture_strategies_IG.rul
/rosigma/Ruleset/IG/strategies/missionScripts_strategies_IG.rul
/rosigma/Ruleset/OTHER/alertSounds_disable.rul
/rosigma/Ruleset/OTHER/alienDeployments_killpoints.rul
/rosigma/Ruleset/OTHER/captures_civ.rul
/rosigma/Ruleset/OTHER/corpses_civ.rul
/rosigma/Ruleset/OTHER/corpses_killpoints.rul
/rosigma/Ruleset/OTHER/manufacture_killpoints.rul
/rosigma/Ruleset/scripts/IGMA_scripts_buff.rul
/rosigma/Ruleset/scripts/IGMA_scripts_change_color_ammo.rul
/rosigma/Ruleset/scripts/script_dig_in.rul
/rosigma/Ruleset/scripts/scripts_abilities_commands.rul
/rosigma/Ruleset/scripts/scripts_abilities_miracles.rul
...
/rosigma/Ruleset/scripts/scripts_tenacious_enemies.rul
/rosigma/Ruleset/scripts/scripts_TU_accuracy_penalty.rul
/rosigma/Ruleset/scripts/scripts_turret_reload.rul
/rosigma/Ruleset/scripts/scripts_UFO_visual_confirmed.rul
/rosigma/Ruleset/SM/armors_SM.rul
/rosigma/Ruleset/SM/crafts_SM.rul
/rosigma/Ruleset/SM/events_SM.rul
/rosigma/Ruleset/SM/items_scripts_SM.rul
/rosigma/Ruleset/SM/items_SM.rul
/rosigma/Ruleset/SM/manufacture_CSM.rul
/rosigma/Ruleset/SM/manufacture_SM.rul
/rosigma/Ruleset/SM/scripts_SM.rul
/rosigma/Ruleset/SM/soldiers_SM.rul
/rosigma/Ruleset/SM/soldierTransformations.rul
/rosigma/Ruleset/SM/startingBase_SM.rul
/rosigma/Ruleset/SM/ufopaedia_SM.rul
/rosigma/Ruleset/SM/Deathwatch/armors_layers_DW.rul
/rosigma/Ruleset/SM/Deathwatch/armors_voices_DW.rul
/rosigma/Ruleset/SM/Deathwatch/Deathwatch.rul
/rosigma/Ruleset/SM/PRIMARIS/armors_PRIMARIS.rul
/rosigma/Ruleset/SM/PRIMARIS/manufacture_PRIMARIS.rul
/rosigma/Ruleset/SM/PRIMARIS/ufopaedia_PRIMARIS.rul
« Last Edit: December 29, 2024, 08:25:47 pm by Delian »

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3394
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #23 on: December 29, 2024, 08:48:53 pm »
Do we need use two `Unicode::lowerCase` for every comparison?

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #24 on: December 29, 2024, 09:04:24 pm »
I could technically convert the whole vector into lowercase first, and then do comparisons. But I'm too lazy. This sorting took 3ms for 40k+rosigma+submods in debug mode. If you wanna make a faster version that saves these 3ms, be my guest :P

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3394
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #25 on: December 30, 2024, 01:13:33 am »
My question was too if its needed at all. Most cases modders work with files not zips, if we need choose one I would choose one closer to how files where handed:
z-A, case sensitive, and this make code simpler and faster for this 3ms too (`lowerCase` was most of this time or rest of sorting was?)

Offline Delian

  • Commander
  • *****
  • Posts: 580
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #26 on: December 30, 2024, 02:10:09 am »
As Meridian said, so far we didn't guarantee anything, so there's no "backwards compatibility" that needs to be upheld. As far as "needing" sorting, it only has to be consistent, meaning, it has to happen to prevent bugs that would arise due to different sorting between OS and files inside zips.

I'm ok with leaving sorting the way it's been, as you say (and how I put it in my commit), case sensitive z-a. The arguments about sort code being simpler and faster are irrelevant (lowerCase took 1ms, I optimized it well hehe).

My opinion is also that, if modders were given control over the order in which their .rul files are loaded, it would turn, what's currently a bug nest, into a useful feature.

Offline Yankes

  • Global Moderator
  • Commander
  • *****
  • Posts: 3394
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #27 on: December 30, 2024, 03:04:07 am »
> The arguments about sort code being simpler and faster are irrelevant

If we can choose arbitrary from two different sort orders then its relevant.
Same with "backwards compatibility" we will break some mods with any option we choose,
but prudent is at least consider version that will have smaller fallout.

> (lowerCase took 1ms, I optimized it well hehe)

ok, but this is only for ASCII case? with Unicode it was more ugly? Many users can have cyrillic characters in full path (like users folder name).

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9197
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #28 on: December 30, 2024, 09:43:32 am »
Difficult question indeed.

I would start with a "less fallout" case-sensitive version.

Then I would (manually) search for mods, which depend on file load order.
So far I know only of ROSIGMA, but there may be more.

Depending on the result, I'd talk to the affected mod authors and ask them to update their mods before we update OXCE... or decide to stay with the case-sensitive version, if too much trouble for either/both of us.

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9197
    • View Profile
Re: Crashes related to ruleset file load order
« Reply #29 on: December 30, 2024, 11:19:44 am »
Well, ROSIGMA is not compatible with either of these two file load orders.

I guess we go one step back to the original proposal... and see if they can be convinced to clean up in the future.

EDIT: pushed to main branch: https://github.com/MeridianOXC/OpenXcom/commit/ae6ce0fb0b0fac97a4b49840156b31e796604afe
« Last Edit: December 30, 2024, 12:08:17 pm by Meridian »