ModPack Installer

[DoxaLogos (JG)]

So, one of my bad habits is I tend think.  While on a relapse, I thought about what would be a nice feature to OpenXcom that would make it even more accessible.  I thought that it's kind of hoky how mods are installed, so what does everyone think about a "mod" installer for OpenXcom?

I could possibly pull this off externally from OpenXcom in fairly short amount of time.  I'm quite handy with Python.  I was thinking that we could come up with a descriptor file inside the mod(yaml, json, ini... don't care), that the installer can use to know where to put stuff into OpenXcom.  Maybe also keep track of mods installed with another file located in OpenXcom.

Later on, it could be built into the executable and made into  a menu on one of the "Options" screens.

Useful?

[Solarius Scorch]

Possibly, but I'd wait until after the mod engine overhaul.

[kikimoristan]

would be cool if mods override but not overwrite data files. so make like 1 more folder called MODS and each mod gets a  directory with same name as the ruleset which acts as a fake data folder for that mod alone. any resources or STR in the mod that have same name as the ones in data folder get overriden by the mod. latest mod being the one that overrides the last. you can probably choose mod load order too.

for ex

data/Rulesets/badalien_mod.rul

data/MODS/badadlien_mod/Resources/badalien sprites/badalien.png  <=>  data/Resources/badalien sprites/badalien.png

<=> means interchangeable, equivalent, same as

data/MODS/modname/ takes precedence . if nothing in MODS then is pulled from data/ instead.

essentially making mods nondestructive

and you could use a zip file with the right folder structure/files as the modpack then have some little program that just takes zip files and unpacks them in data/  so you get data/MOD/modname/*stuff* and data/Rulesets/modname.rul so you can easily uninstall later without corrupting the data folder. The folder structure completely created by the modder . All the modpack installer does is unpack it in data. EZ

and this app would read current rul files and display a list..if you choose to delete a mod then it deletes rule file and it's corresponding folder in MODS...

you can still do it the old way by putting it in data if you want

[DoxaLogos (JG)]

Possibly, but I'd wait until after the mod engine overhaul.

So is there documentation about what the mod engine overhaul will be? Or do I just need to grok the source code?

I'm okay with waiting.  It would give me some time to get familiar with new system and test.

[DoxaLogos (JG)]

would be cool if mods override but not overwrite data files. so make like 1 more folder called MODS and each mod gets a  directory with same name as the ruleset which acts as a fake data folder for that mod alone. any resources or STR in the mod that have same name as the ones in data folder get overriden by the mod. latest mod being the one that overrides the last. you can probably choose mod load order too.

for ex

data/Rulesets/badalien_mod.rul

data/MODS/badadlien_mod/Resources/badalien sprites/badalien.png  <=>  data/Resources/badalien sprites/badalien.png

<=> means interchangeable, equivalent, same as

data/MODS/modname/ takes precedence . if nothing in MODS then is pulled from data/ instead.

essentially making mods nondestructive

and you could use a zip file with the right folder structure/files as the modpack then have some little program that just takes zip files and unpacks them in data/  so you get data/MOD/modname/*stuff* and data/Rulesets/modname.rul so you can easily uninstall later without corrupting the data folder. The folder structure completely created by the modder . All the modpack installer does is unpack it in data. EZ

and this app would read current rul files and display a list..if you choose to delete a mod then it deletes rule file and it's corresponding folder in MODS...

you can still do it the old way by putting it in data if you want

I do like the sound of what you're proposing.  It would be a lot easier to deploy into the data folder.  The advantage of having a "mod" manifest would be to give the mod developers possibly more control over how the mod is deployed if need be.  I guess I need to check into how the new system is being developed first though.

[Solarius Scorch]

So is there documentation about what the mod engine overhaul will be? Or do I just need to grok the source code?

I don't think there's anything concrete out there. I just heard a couple times something like this is planned, sorry.

[DoxaLogos (JG)]

So I've revisited the problem of making a script that can update your latest mod packs and update latest nightly with plans to pick mods to install later on down the road, and I've run into bit of a snag.

The most fundamental problem to updating to the latest mod right now is the lack of consistent naming between mods found in the saved rulesets and whats up on the mod website for the mod (both name of mod and filename of the mod).

Example:

Final Mod Pack on website has a file called "Final Mod Pack_1.2.1.zip" and the ruleset in save file and options.cfg has the name "FinalModPack".

Wait.. it gets worse...

Improved Hand Objects has a file called "ImprovedHandObsPack1.1.zip" with the following ruleset string "ImprovedHandOb"

Then to be even more inconsistent..

UFO REDUX has a file called "UFO REDUX v0.5.1" with a ruleset name of "UFO_Redux_Nightly". 

That's just three mods, and I shutter to think what more naming patterns ensue.  I'm not sure how to proceed.  I may have to whip up a partial matching algorithm to connect the installed ruleset mod to the correct mod on the website.  Sounds tricky and bug prone when I think about it.

Any ideas? or suggestions?

I know the ultimate solution long term is to develop a consistent format for mods that modders can adopt, but I have feeling that will take a long while to trickle through the modding community.

[Bloax]

It will definitely take a while to be adapted, but the sooner you start the sooner it will become reality.

[ivandogovich]

It appears as the a good first standard is for mod packages on the site should start with the exact name that the ruleset uses for selection.

I believe that CAML casing is a standard for developers in the code, and imo it makes sense to apply that here.

ImprovedHandObs  should be the rulename (or at lease the name of the ruleset file.  later metadata additions may allow for a Proper "Title" to display in game.
ImprovedHandObs_v1.1.zip could then be the file pack name.   What comes after the underscore could be less structured as modders could specify them however they like.

Your function would then only have to look for the most recent ImprovedHandObs* item on the site and DL it?

[DoxaLogos (JG)]

It appears as the a good first standard is for mod packages on the site should start with the exact name that the ruleset uses for selection.

I believe that CAML casing is a standard for developers in the code, and imo it makes sense to apply that here.

ImprovedHandObs  should be the rulename (or at lease the name of the ruleset file.  later metadata additions may allow for a Proper "Title" to display in game.
ImprovedHandObs_v1.1.zip could then be the file pack name.   What comes after the underscore could be less structured as modders could specify them however they like.

Your function would then only have to look for the most recent ImprovedHandObs* item on the site and DL it?

Yes, that would be a very good start if the ruleset name and the filename were the same up to the first "_".  That would solve a lot of problems in the future. 

UFO REDUX would have to become UfoRedux_*.zip for filename and UfoRedux in the ruleset. 

I think what concerns me is that there is no way enforce this.  Is the answer is to build the tool first that has value and maybe everyone will see that adopting a standard is worthwhile?

Overall, it would be a HUGE improvement to implement your simple suggestion over the anarchy that is here right now.   

I was looking for ideas on how to make the connections right now at the current state of things.  I'm looking into some fuzzy matching algorithms right now.  However, that might discourage some standardization.  I'll keep working on it. 


The other problem I face is determining if my current installation is already at the latest version of the mod that is up on the website.  I don't see any "version" information being dropped in the ruleset folders in consistent format (if any and that is up to date - it isn't!). 

Example:

FinalModPack  ruleset file has odd characters at the beginning then "Release v. 1.0.4" and it's at 1.2.1
Improved_HandOb has quite few lines then "v 1.1"  is at 1.1
UFO_Redux_Nightly has "version 0.4.8" and is at 0.5.1.

Versions are also not consistent on the file name

yeah, what quagmire I got myself into ...woohoo!

So what might be better would be FinalModPack_1.2.1.zip  with a comment string in the top of the ruleset file "version 1.2.1".  I can pull the version off the filename after the "_" and match it against the version string in the ruleset file.

That second step would help tremendously.  That would get my script started working more reliably.  I won't even get into "folder" structure to ease the installation process.

[ivandogovich]

These are all very sensible steps. 

Looking forward, there is a development effort underway by myk002 to standardize mod file structures and deployability. One of the fundamental aspects will be mod metadata that will handle alot of that.

Another thought is that some of this metadata is carried at the modsite.  you have to enter your mod version number when you upload, so that information is stored there.  If it could be retrieved during the download and annotated locally, then your system would know which version it has, and when an newer version is submitted.  (Conversely, the modsite could also read this from a metadata file when that is ready).

Its true that aside from YAML which is a must for modders, there are really no other standards in place, and its kind of Wild West.  Setting standards, and publishing those with the incentive of an installer that can handle the mods for the user, would, I think quickly bring the active modders on board.  Those that contribute regularly, I think would gladly get on board.

Another goal in the over all OpenXcom project is a mod builder tool that will help modders by using an interface to correctly format their mods.  This should greatly assist with standardization.

[jackstraw2323]

I assume loading mods via a folder would be a good first step for organization. One that I assume will happen with tftd if it's not already possible. After all the vanilla rules now load from a sub folder. There could be a mod.rul that declared a description of the folder contents and any other relevant info like version and compatibility numbers.

[hellrazor]

I assume loading mods via a folder would be a good first step for organization. One that I assume will happen with tftd if it's not already possible. After all the vanilla rules now load from a sub folder. There could be a mod.rul that declared a description of the folder contents and any other relevant info like version and compatibility numbers.

Well you already can load ruleset files from folders. Which should also fix issues with mods which use only one single file, e.g. the FMP which has like 32000+ lines in one file, which is as nightmare to look at ( i like small files, like the Xcom1Ruleset).

Versioning is something that is really up to the modder and not every new Version released contains changes to the ruleset, in case of Mods having Map's and Terrain's in it. Sometimes it is a collection of tine fixes to MCD or something like this.

In essence, we will have directories for mods. Each mod his own directory, with his own MAPS, TERRAIN, ROUTES, Resources etc..
And i can wait enough for this

[hellrazor]

A installer, raises a couple of questions.
First which platform (we have at least three, Windows, Linux, MacOS)
Each of those has different scripting languages etc..
Second different paths for userdirectories
Third Versioning.

The main point is updateing mods is relativly easy. As long as the Authors stick to the respective directory structure.
You download *.zip, extract it, copy contents to user directory.

So what's the big deal?

[DoxaLogos (JG)]

A installer, raises a couple of questions.
First which platform (we have at least three, Windows, Linux, MacOS)
Each of those has different scripting languages etc..

Yes, but Python runs on all 3 platforms.  That is the language I'm using.  So I don't see a problem there, only that I'm not a Mac user.  I can test on Windows and Linux, and I have experience working on both of those with Python.  This is the easiest part of the equation.

Second different paths for userdirectories

I agree, that is another problem I have running in the back of my mind.  Some of this is mitigated that most OS's have builtin environment variables like "HOME" to help me get started.  But, I'll have to write code that detects the platform it is running on and change paths accordingly.  Some path help may have to be provided by the user at runtime.  I don't see it being a huge problem... except on Mac again

Third Versioning.
The main point is updateing mods is relativly easy. As long as the Authors stick to the respective directory structure.
You download *.zip, extract it, copy contents to user directory.

So what's the big deal?

The mods are completely inconsistent right now with about everything that describes them, so building a programmatic interface (not just directory structure) is quite difficult.  Also mods like OXC extended and Soldier's Diary mod overwrite the OXC executable, so it extracts into a different directory than the data folder like most other mods.  Also, as mentioned previously, the name of the mod in the installed ruleset is different than what's up on the website (although Ivan's suggestion of saving file of what the installer installs will help).  So everything is at a "chicken and egg" problem right now.