Messages

1

XPiratez / Re: XPiratez Information Compilation, Starter Guide & Weapon Spreadsheet

« on: Today at 12:08:32 pm »

Hybrid has two unique crafts: SEK-SKOUT and SEKAR. While researching Reticulan Elder is required for both, the Reticulan Elder topic currently does *not* unlock the crafts.

The problem here is that, Reticulan Elder is pretty hard to get. Yes, Life is Hybrid allows you to produce Reticulan Files early on, however, hitting Reticulan Elder with the Files will still require 12x Reticulan Files on average. This means the player needs 120x Personal Database before he can buy SEK-SKOUT.

120! I doubt the player will be able to get 120x Personal Database in the early game.
Which means, even though SEK-SKOUT is meant to be an early game Hybrid Path craft, you can't actually get it early. This seems like an oversight. Did Dioxine forget to add the "unlocks"?

2

OpenXcom Extended (OXCE) / Re: Converting OXCE to rapidyaml

« on: December 20, 2024, 10:00:55 pm »

I did a small test. It's 100ms just visit all the nodes of all the trees (no vector building). So, good luck~

3

OpenXcom Extended (OXCE) / Re: Converting OXCE to rapidyaml

« on: December 20, 2024, 04:04:31 am »

Rapidyaml doesn't use list<node>. Internally it already uses vector<NodeData> (It's array*, same thing; NodeData = 72 byte struct), and "NodeId" is just offset in this array.

The cache-locality of this array is already great. It would be very difficult to improve it. You can try, but I think constructing your vector would take more time than the time gained from theoretically improved cache-locality. Sorting... that's worse than hashing, isn't it? That's why std::map never beats std::unordered_map.

I can give you some current timing data. Let's say game startup (debug build with XPZ; Options::mute = true) takes 5000ms
Of these 5000ms, operations related to rapidyaml take:
- 1300ms to parse raw yaml into node trees
- 333ms searching for node keys
- 263ms deserializing node values
- 87+35ms building+destroying unordered_map node key indexes
- 32+13ms building+destroying YamlNodeReader.children() vectors
- 20ms building+destroying YamlNodeReader objects
- other negligible stuff

Of the 333ms of node key searching there is:
- 220ms (10ms in release build) of indexed searching
- 72ms (4ms in release build) of unindexed searching (with stored child iterator optimization)

What about game loading? In the release build, loading a 12MB sav file, all the unindexed searching together takes <2ms.

If your proposed optimization is targeting unindexed key searching, well, there's very little left to gain there. The majority of search calls (3 to 1) are already indexed. At least for game startup. For game-loading, the majority is unindexed, however that's mostly (80%) inside BattleUnitKills::load(), where the stored child iterator optimization gives it O(1).

Btw, what are the places that you say you found where there's too much iteration?

4

XPiratez / Re: Bugs & Crash Reports

« on: December 19, 2024, 08:18:50 pm »

Damn, so we can't start bounty hunts until April? And we can only reach Rank: Racketeer (RANK_03) in May? That's like the game saying: Fuck you, you're not allowed to progress no matter how well you're doing. In other words, it's bad game design.

5

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 17, 2024, 02:53:19 pm »

Ok, but back then soldier name files were not yet being sorted locale-aware. Stoddard only made namepool sorting locale-aware 5 years ago.

Yes, I agree that there are some problems resulting from sorting in different locales. I am with you on that.

For demonstration:
Let's say that a modder defines "Flag25" to be Resources/Flags/Kenya.png.
How did the modder determine that Flag25 is Kenya? He looked at the SoldierNames (or whatever) folder and saw that the 26th file was Kenyan.nam
So nationality with index 25 is Kenya. Soldier with _nationality = 25 will get this flag.
Soldier also gets a name from names[_nationality]. So the name pool with index 25 will be used for the name. names vector was generated after locale-aware sorting files.
As you can guess, this will fail in other locales, because there's no guarantee that Kenyan.nam will be the 26th file after applying a locale-aware sort.
I don't know why Stoddard made name pool sorting locale-aware. It seems like a really bad idea to do that
To being with, anything (_nationality included) referring to the index position of a file in a folder is too fragile and is simply bound to fail one way or another. Add or remove a file and it fails.

So, uhh, what were we talking about again? Ruleset load order... yeah... so there's an issue that a lot of mods rely on the order the rul files are loaded in, while modders don't have any ability to specify what the order is. If you wish, I can try implementing something that would give modders some control over this order.

6

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 16, 2024, 10:01:20 pm »

I have found only two usages outside of this file, and they are probably mistakes.

So cruel to call them mistakes! They work fine! (as long as file loading is z-a)

And here's an example of some problems resulting from different locales: https://openxcom.org/forum/index.php?topic=5146.msg83361#msg83361

...

...

No, that problem wasn't due to different locales. It was due to duplicate folder names between common/SoldierName and Celebrate_Diversity/SoldierName.
The problem was that common/SoldierName has "argentina.nam", but Celebrate_Diversity does not (it has "argentinian.nam"). So after merging the two virtual dirs, this extra file causes the offset.
Note that, even though we were already using std::sort() for rul files back then, that doesn't apply to soldier names. For soldier names we explicitly call a locale-aware sort.

7

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 16, 2024, 04:54:45 pm »

Maybe for the list of saves?
They can be sorted from newest to oldest...

Nope. While it is possible to sort save files base on the last write time, that's implemented independently. I mean, even though we're calling the getFolderContents() function when retrieving the list of sav files, and the returned tuples include the last write time, the last write time is discarded. Yes, it would make sense to pass the tuple to the getSaveInfo() function because then we wouldn't need to call getDateModified() for every sav file again. But for now, I can say that the last write time, as retrieved by the getFolderContents() function (which is the function based on which we build the VFS and determines the sort order we load .rul files in), is not used anywhere.

Sorting depends on a lot of things and is completely unreliable.

I linked the part of code where we're sorting the list of retrieved filenames. In that link, I show that we're using std::sort() to sort this list. This std::sort() call uses a lambda which compares the filenames using the string > operator. The C++ 17 standard states that the comparison is done using the compare() member function. And the compare function states, quote:
"By default (with the default std::char_traits), this function is not locale-sensitive."
Tldr; You are talking about locale-aware sorting the OS does, but we ignore that because we perform std::sort() after retrieving the list of files, which is locale-independent.

Sure, I can code the OXCE-specific file sorting for you. But, do I need to do anything besides simply calling std::sort() to achieve that?

Loading that file first would not solve it.

That's literally what's happening right now. XPZ loads Yankes_Scripts.rul first, because Y is at the end of the alphabet, it's the last file in the folder, so it's loaded first. That's why it doesn't error during the loading. Because Yankes_Scripts.rul contains all the tag name definitions. These tags names are not defined anywhere else and the shared file isn't linked anywhere, yet it works.

Tag definitions are not visible cross-file.

As far as I can tell, script loading internally uses a shared list of tag names, and this list persists until the loading is finished. Meaning, once a tag is defined, the tag's value can subsequently be set by any rule object, no matter which file it's in.

8

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 15, 2024, 03:05:59 pm »

You're comparing apples to oranges. C++ only builds the files/functions that are encountered through the include chain, but we are unconditionally parsing everything. Different problem, different solution. Think of it like specifying an entry point: main.cpp. You do that in C++, without needing #include main.cpp in every file.

That repidyaml issue indeed looks fun (it seems like it's only cygwin problem?) hehe, but it's off topic.

9

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 15, 2024, 02:19:43 pm »

I am not aware of this.

CrossPlatform.cpp#L642
Oops, I was wrong. I assumed the files were sorted based on last write time, because there's an elaborate lastWriteTime reading, conversion and storing. This last write time is then used... nowhere. Huh. Why is it being read then? Stoddaaaaard!
But yeah, sorry, the sorting is actually z to a. Unless, as Yankes mentioned, we're using a ziploader. Ziploader doesn't perform any sorting (but files in zips are usually a-z). So if a mod is inside a zip file, .ruls are (usually) loaded a-z, and if files are in a plain folder, then they're loaded z-a.
Hmm, if sorting is actually z-a, then the whole issue is a lot less problematic. And I didn't need to make this thread at all...

Within one mod, the rule value should not be defined twice.
Best practice is to define each object only once.
If you must split the definition into multiple places, for whatever reason, then at least keep them distinct (i.e. don't repeat the same attributes for the same object).

I agree. That would solve such issues.
Hmm. Modders are currently allowed to merge objects, but the order is only guaranteed in the order of mod hierarchy, from top (master) to bottom. In theory, if the modders were guaranteed or allowed to specify the load order, then the "Rule object merging" feature would also be safe between the .rul files within the mod. One could argue that the specific load order is a wanted feature based on the fact that modders already depend on it hehe. A good example is how the rulesets are defined in ROSIGMA. He's got normal object definitions, and then he's got a BALANCE folder, which overwrites his normal definitons. He's able to change and experiment with stuff in the BALANCE folder without having to worry about duplicate keys and specifics of his base definitions.
In other words, while we currently don't guarantee any load order, if we did allow modders to specify it, it would be a useful feature to the modders because it would allow them to safely layer their rule definitions in more possible ways.

Since 2014, I haven't seen or heard even a single false positive from this check. All reports were correctly found issues.

I know about it because yesterday, after I optimized VFS loading to be 3x faster...

Code: [Select]

[15-12-2024_11-50-19] [ERROR] failed to load '40k ROSIGMA Submod'; mod disabled
C:/OXCE/user/mods/rosigma/Ruleset/BALANCE/plasma.rul: Weapon STR_INQ_PLASMA has clip size 0 and no ammo defined. Please use 'clipSize: -1' for unlimited ammo, or allocate a compatibleAmmo item....this check failed for me when I tried loading ROSIGMA files in the wrong (a-z) order hehe.

Either define them in each file you need them,
or use the shared definition: https://openxcom.org/forum/index.php?topic=11102.0

Uhh. So that solution is basically an awkward way of trying to solve this same problem.
Why do I say it's awkward? Because with that solution the modders have to put extended: { tagsFile: "Ruleset/tagsFile.rul" } (boilerplate) into every .rul file. A better solution would've been for modders to set to load "Ruleset/tagsFile.rul" first, which would happen one time in one place, and then never worry about it again.

10

OXCE Bugs / Re: Crashes related to ruleset file load order

« on: December 15, 2024, 12:22:15 am »

You say that, but... I've checked, and currently most the major mods (XPZ, XCF, 40k, ROS, TWoTS) have errors if files are loaded in the wrong order. The mod authors don't even know about it. Or maybe they know and they don't know how to fix it. The fact that they don't know that this is an issue is an issue in itself lol

11

OXCE Bugs / Crashes related to ruleset file load order

« on: December 14, 2024, 11:34:40 pm »

Currently we're loading files in the order of... newest to oldest (last write time descending order).
Usually this means z to a, because when users extract mod zip files (or clone a repository), these files are created on the hard drive in alphabetical order (the order in which they're downloaded, or order in which they appear in the zip file, which is usually alphabetically), so the z file is created last, and thus it's loaded first because it is the newest file.

This is both confusing (a normal user would expect the order to be a-z), and problematic, because if someone manually edits one of the rul files, even if they simply resave it without any changes, it changes the order in which the files (and folders) are loaded.

Currently we're loading files in the order of... z-a (descending case-sensitive alphabetical order).

This is both confusing (a normal user would expect the order to be case-insensitive a-z), and problematic, because if someone renames one of the files/folders, it changes the order in which the files are loaded.
And currently the majority of mods crash (or have other issues) if files are loaded in the wrong order.

Why is loading files in the wrong order problematic?
1. Wrong rule values. If a Rule object is defined in two .rul files, then the Rule object properties in the last file processed get to overwrite the properties of the first one.
2. Crashes due to erroneous error checks. If a Rule object is defined in two .rul files, then loading the wrong one first could throw due to object temporarily being in an invalid state.
3. Failed script parsing. Tag names need to be defined before Rule objects are allowed to set values for those tags. So the ruleset with tag names needs to be loaded first.

The 2nd problem could be solved by moving the error check to the afterLoad() function.
The 3nd problem could be solved by loading all the rul files and processing their "extended:" nodes before processing any other yaml nodes.
The 1st problem however cannot be solved. Even if we forcefully sort the files in a specific order, there's no guarantee that the modder won't rename them and thus cause a crash. It would also be inconvenient to modders if we forced a certain sort order.

Basically, what we need is to allow modders to specify which rul files to load first. That would solve all 3 of the problems. Perhaps something in metadata.yml or vars.rul.

12

OXCE Support / Re: Increase script debug log limit

« on: December 12, 2024, 01:24:10 pm »

Do you know why debug logging is slow?

Because the current implementation is inefficient. Every time you write to the log, it opens the log file and closes the log file. This file opening and file closing is what kills the logging performance.

13

OXCE Support / Re: Tabs problem

« on: December 08, 2024, 07:22:00 pm »

In the Notepad++ Find window, first change the Search Mode in the bottom left to Extended
To find a tab, search for \t

14

OXCE Suggestions NEW / [QoL] Battle Action Hot Swapping

« on: December 08, 2024, 01:30:01 pm »

Proposed feature: Shift clicking on the battlescape with an attack selected will fire a snap shot at that tile.

I do not like this suggestion for the following reason:
1. You already have hotkeys to fire a snap shot: RMB-click + Q/E + 2
2. Your suggestion only applies to snap shots, so it's not custom enough for most situations.

However, I do feel there is some redundancy in procedure of the said hotkeys usage, so I'd like to propose a different suggestion based on your suggestion:

Battle Action Hot Swapping
Allows using hotkeys to swap battle action even after the action has already been selected.

Use case 1 (easier consecutive attacking):
1. Hover cursor over a target
2. Press E to select right-hand weapon
3. Press 3 to select Auto
4. Press LMB to fire
5. Press 2 to select Snap
6. Press LMB to fire

Notes:
- In step 5: If the firearm does not have the battle action defined, the battle action is cancelled (same as RMB-click)
- The solution only applies to swapping between 3 Firearm actions (BA_AUTOSHOT, BA_SNAPSHOT, BA_AIMEDSHOT) and 2 Psi-amp actions (BA_MINDCONTROL, BA_PANIC)
- It's player's own responsibility to be highly familiar with the weapon so that after swapping and firing, they have enough TU left to retreat to safety.

Use case 2 (easier accuracy comparison):
1. Hover cursor over a distant target
2. Press E to select right-hand weapon
3. Press 3 to select Auto and observe accuracy
4. Press 2 to select Snap and observe accuracy
5. Press 1 to select Aimed and observe accuracy

15

XPiratez / Re: A thread for little questions

« on: December 06, 2024, 09:31:42 pm »

You did not have enough cold resist, because it was nerfed. You think you didn't take any hp damage, you simply weren't looking carefully enough.