Author Topic: Modding master mods.  (Read 1673 times)

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Modding master mods.
« on: March 26, 2023, 11:00:10 pm »
I would like to consider the issue of creating modifications to larger mods.

For example, I would like to add some additional ammunition types to existing weapons in the XCF, as well as to add a player class, and adjust some armors.  Conceptually, the problem is relatively simple: it requires that some data nodes be inserted into the existing configuration node tree, and in some cases collections in existing paths would need to be appended to.  While these particular modifications do not require deletions, it would in general be desirable to have these capabilities available.

Now, it is entirely not clear how to implement these changes.  The primary issue is as follows.  Let's consider an approach of copying the entire .rul files from the parent mod and editing them.  If the files in the parent mod change, then new instances in the child mods need to be generated.  This at the very least requires dependency on specific parent version, as well as introducing additional maintenance and release hurdle.  Even worse, the dependency on what set of modules is being loaded would be introduced.  That is, if other mods make changes to the same files, then the changes should be made against versions in these mods.  Not only this complicates the maintenance, but also makes modding much less flexible.  In particular, it would be quite challenging to have two mods with each adding an individual weapon.

The question is, how to solve this problem.

First, I would like to understand what is the exact semantics associated with adding new ruleset definitions in child mods?  Is this semantics dependent on the names of the files?  Is it only dependent on the total contents of all YAML definitions in all of the files (and thus the file name are actually irrelevant)?

I would like to consider several approaches, and to get clarifications.

First, is patching the files from the parent mode directly.  This requires ability to depend on exact version, and introduces issues with flexibility.

A partial solution to the previous issue would be to have a single mod contain several instances that would apply based on what parent mods versions are installed.

Another approach is to enable some form of patching of definition trees.  There are some extensions to YAML that allow that.  Approaches not based on these YAML extensions could be considered.  Specifically, a set of commands along the lines of

DELETE /armors/type:STR_GRAVMODULE_ARMOR_UC/
DELETE /armors/type:STR_ALIEN_BRAIN_ARMOR/
INSERT /items/type:STR_SMALL_LAUNCHER/compatibleAmmo/ /mymod/items/type:STR_SMALL_LAUNCHER/compatibleAmmo/

could be issued in mods.

I noticed that currently, it seems that there are provisions for issuing a delete directives.  I wonder, how could insertions be made in a sound fashion?  I think, a sound approach may be to utilize an existing YAML feature of tags, and to have the definitions in mods be tagged accordingly.  That is, if currenly a default action is to REPLACE, a special tag could indicate that the action should be to MERGE instead.

I wonder, if the latter approach is implemented already?  If so, I would appreciate some documentation reference.

In terms of engine semantics, I would like to consider an approach based on interpretation of configuration within the existing parsing semantics.

So, currently, there's an ability to add list nodes like

  - delete: STR_LASER_PISTOL
  - delete: STR_LASER_RIFLE
  - delete: STR_HEAVY_LASER
  - delete: SILACOID_WEAPON
  - delete: ZOMBIE_WEAPON
  - delete: REAPER_WEAPON
  - delete: CHRYSSALID_WEAPON


I would like to have the ability to define in addition to that something like

  - type: STR_SMALL_LAUNCHER
    compatibleAmmo:
      - insert: STR_MEGA_BOMB
      - insert: STR_GIGA_BOMB

in one mod and also

  - type: STR_SMALL_LAUNCHER
    compatibleAmmo:
      - insert: STR_GIANT_BOMB
      - insert: STR_ULTRA_BOMB

in another mod.  A final result should be the following configuration:

{ { type : "STR_SMALL_LAUNCHER" ,
      compatibleAmmo: [ "STR_STUN_BOMB" , "STR_ELERIUM_BOMB" , "STR_EMP_BOMB",
      "STR_MEGA_BOMB", "STR_GIGA_BOMB", "STR_GIANT_BOMB", "STR_ULTRA_BOMB" ] },
   # ...
}








« Last Edit: March 26, 2023, 11:10:31 pm by zee_ra »

Offline Yankes

  • Commander
  • *****
  • Posts: 3207
    • View Profile
Re: Modding master mods.
« Reply #1 on: March 27, 2023, 03:45:23 am »

Offline Chuckebaby

  • Colonel
  • ****
  • Posts: 386
  • Chrysalis are people too
    • View Profile
Re: Modding master mods.
« Reply #2 on: March 27, 2023, 06:43:05 pm »
For more info what is available, I updated https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)

I've just scratched the surface of modding my own game files. This Wiki doc is very helpful. I really appreciate it. Will come in handy when I begin to mod.
Thanks.

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 8616
    • View Profile
Re: Modding master mods.
« Reply #3 on: March 28, 2023, 09:45:03 am »
I wonder, if the latter approach is implemented already?  If so, I would appreciate some documentation reference.

Yes, all of the features you are mentioning here are available.

Documentation is available in the ruleset reference on the wiki; or in this thread: https://openxcom.org/forum/index.php/topic,11199.0.html
« Last Edit: March 29, 2023, 11:26:04 am by Meridian »

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Re: Modding master mods.
« Reply #4 on: March 28, 2023, 09:54:18 pm »
Yes, all of the features you are mentioning here are available.

Documentation is available in the ruleset reference on the wiki; or in this thread: https://openxcom.org/forum/index.php/topic,11199.0.html

I think that adding semantics into the special keys rather than using config-specific meta-programming or templating is a good approach.  The refNode approach is much better than yaml patching, whether in yaml itself or any other format.

The only huge problem with making more sophisticated mods I see with this engine is actually the scripting options available.  A question is, why does it not use Lua or a similar embeddable interpreter?  Current scripting language and the scripting engine are both clunky and idiosyncratic, to the point of being useless.  Consider the issue of why would one need to delve into another programming environment, without clear benefit, when a knowledge of existing technology could have been utilized?  The Emacs Lisp was able to get away with that, and now it has a momentum.  I doubt anyone else should be able to get away with that, and certainly would not.  :o

I feel that all scripts even in larger mods like XCF could be ported to Lua with relative ease.  In terms of design, it makes sense (in the interest of avoiding complications when multiple distinct interpreters are involved) to exclusively use either Lua or legacy interpreter in each master mod and its descendants.

By the way, the debuggability would go up drastically with correct approach to scripting, and would not need to involve any changes in the C++ engine code.
« Last Edit: March 29, 2023, 11:26:33 am by Meridian »

Offline Yankes

  • Commander
  • *****
  • Posts: 3207
    • View Profile
Re: Modding master mods.
« Reply #5 on: March 28, 2023, 10:08:50 pm »
Because it grown with OXCE, have no dependencies, is lightweight and correctly map C++ critical idioms like `const` and binding function is trivial.
It have lot of limitations but this can have benefits as it lot of easier reason about game state even if you call non stop some scripts.

Beside there was already couple people who suggest changing script engine, but it was all that hey did,
and after week or so they disappeared not contributing any thing worthwhile.

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Re: Modding master mods.
« Reply #6 on: March 28, 2023, 10:46:51 pm »
Because it grown with OXCE, have no dependencies, is lightweight and correctly map C++ critical idioms like `const` and binding function is trivial.
It have lot of limitations but this can have benefits as it lot of easier reason about game state even if you call non stop some scripts.

Beside there was already couple people who suggest changing script engine, but it was all that hey did,
and after week or so they disappeared not contributing any thing worthwhile.

I could see what I could do.  The usual caveat is "time permitting", and that may play a very significant role in how much I could do in that area.  If I encounter something in XCF that really needs scripting, I think, I would dedicate some time to this project.  So far, the changes I've been making only required the work with configs and graphics.

I wonder, if there are principal objections to Lua in the OXCE?

Offline Yankes

  • Commander
  • *****
  • Posts: 3207
    • View Profile
Re: Modding master mods.
« Reply #7 on: March 28, 2023, 11:08:52 pm »
I could see what I could do.  The usual caveat is "time permitting", and that may play a very significant role in how much I could do in that area.  If I encounter something in XCF that really needs scripting, I think, I would dedicate some time to this project.  So far, the changes I've been making only required the work with configs and graphics.

I wonder, if there are principal objections to Lua in the OXCE?
New complex dependency, integration of it into whole OXCE code base, existing mods that use my script engine ("my" is another reason :> ).
As some highlanders said "I can be only one", and my script is already in, I do not need two script engines in one exe.

Beside even if you would add Lua to OXCE, you will not grain lot, it will still need to have same restriction as current script engine that make many
Lua strengths useless. This is because of 80% usages of current script hooks is to calculate some values and noting else.

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Re: Modding master mods.
« Reply #8 on: March 28, 2023, 11:32:50 pm »
New complex dependency, integration of it into whole OXCE code base, existing mods that use my script engine ("my" is another reason :> ).
As some highlanders said "I can be only one", and my script is already in, I do not need two script engines in one exe.

Thank you for sharing this perspective.  I think, a practical solution would be to fork in this case.

As noted earlier, though, from the modding perspective, especially when modding a master mod, most things are doable by the methods of config editing.

Beside even if you would add Lua to OXCE, you will not grain lot, it will still need to have same restriction as current script engine that make many
Lua strengths useless. This is because of 80% usages of current script hooks is to calculate some values and noting else.

Could you please elaborate on that.

Offline Yankes

  • Commander
  • *****
  • Posts: 3207
    • View Profile
Re: Modding master mods.
« Reply #9 on: March 29, 2023, 12:13:34 am »
Could you please elaborate on that.
Stateless calculation of some values. Can you make that some Lua script will have absolute not effects aside from return value?
And it will be not some gentleman agreement but enforced requirement.
Why this is needed? because if I remove call to some script hook or add it in another place nobody can detect it and relay on it.
How would you refactor big code base if any change of order of some function would break every mod?


Offline Finnik

  • Colonel
  • ****
  • Posts: 492
  • Finnik#0257
    • View Profile
Re: Modding master mods.
« Reply #10 on: March 29, 2023, 08:53:32 am »
After Yankes implemented loops in y-scripts, I don't see any practical reasons to change it to Lua from modding perspective. It would be a huge time investment and I don't see the goal for that but to change scripting syntax. Would be much better to invest that time to expand y-script API with more gesocape hooks.
Also, I don't see a point - if "there could be only one", then what about backwards capability? Lots of mods are using y-scripts.
About syntax - yes, it can be intimidating, but you can use to it. Also, there is an extension made by Pedro to highlight y-scripts code, witch is very helpfull.

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Re: Modding master mods.
« Reply #11 on: March 29, 2023, 10:22:20 am »
After Yankes implemented loops in y-scripts, I don't see any practical reasons to change it to Lua from modding perspective. It would be a huge time investment and I don't see the goal for that but to change scripting syntax. Would be much better to invest that time to expand y-script API with more gesocape hooks.
Also, I don't see a point - if "there could be only one", then what about backwards capability? Lots of mods are using y-scripts.
About syntax - yes, it can be intimidating, but you can use to it. Also, there is an extension made by Pedro to highlight y-scripts code, witch is very helpfull.

Concerning the language itself.  It does matter.  The current language is not satisfactory in a number of ways.  The one important conclusion that I would argue forth is that the current scripting is not suitable for sophisticated scripted logic.  Note that all the larger mods are not attempting to implement anything sophisticated with scripting.  And, as Yankees pointed out, this has not been the goal nor an intended application of the scripting engine.

However, that is largely irrelevant at this point in the grander scheme of things.  The reason for this is as follows.  All the larger mods have already made the use of scripting to produce a logic that is reasonably configurable through the config stanzas.  That is largely more than enough for modding.

Currently, the end result is adequate, and is well maintained.  It is also moddable.  This is what matters from the standpoint of enjoying the game.

I think, any significant improvement -- that includes (a significant improvement to ...) scripting -- would require ultimately nothing short of a rewrite.  I think that adding Lua, while possible, is not worth time, given that sub-modding could adequately be done without it.

I suggested Lua as a known and suitable option.  Ultimately, a future game engine should not be using such scripting engine, if only not to fall into the Clausewitz II (which is another game engine) design trap.

Offline zee_ra

  • Colonel
  • ****
  • Posts: 200
    • View Profile
Re: Modding master mods.
« Reply #12 on: March 29, 2023, 10:23:57 am »
Stateless calculation of some values. Can you make that some Lua script will have absolute not effects aside from return value?
And it will be not some gentleman agreement but enforced requirement.
Why this is needed? because if I remove call to some script hook or add it in another place nobody can detect it and relay on it.
How would you refactor big code base if any change of order of some function would break every mod?

State could be eliminated by removing the primitives that enable sharing of state.

I mentioned a Clausewitz II design trap in my previous post, which is essentially a reflection on the mis-use of state in fully scriptable engines.  I am aware of that problem.

It seems that the current approach of well thought out static configuration semantics plus lightweight scripting works out decently well.  It seems to be enough for me to make meaningful submods.  Now, if I wanted to approach making a master mod, I would actually consider remaking the engine.  Actually, the desired system could be concocted with the aid of Lua, but the engine that would result from it all would be a totally different one -- not a small interpreter compatible with original X-Com semantics.  It's a topic for a very much different project.  Quite a bit beyond the domain of pleasant sub-modding.
« Last Edit: March 29, 2023, 10:34:21 am by zee_ra »

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 8616
    • View Profile
Re: Modding master mods.
« Reply #13 on: March 29, 2023, 10:40:17 am »
Now, if I wanted to approach making a master mod, I would actually consider remaking the engine.  Actually, the desired system could be concocted with the aid of Lua, but the engine that would result from it all would be a totally different one -- not a small interpreter compatible with original X-Com semantics.  It's a topic for a very much different project.  Quite a bit beyond the domain of pleasant sub-modding.

I suggest to continue this discussion in Off-topic: https://openxcom.org/forum/index.php/board,1.0.html