OpenXcom Forum
OpenXcom => Suggestions => Topic started by: DoxaLogos (JG) on March 03, 2015, 05:55:08 pm
-
So, one of my bad habits is I tend think. While on a relapse, I thought about what would be a nice feature to OpenXcom that would make it even more accessible. I thought that it's kind of hoky how mods are installed, so what does everyone think about a "mod" installer for OpenXcom?
I could possibly pull this off externally from OpenXcom in fairly short amount of time. I'm quite handy with Python. I was thinking that we could come up with a descriptor file inside the mod(yaml, json, ini... don't care), that the installer can use to know where to put stuff into OpenXcom. Maybe also keep track of mods installed with another file located in OpenXcom.
Later on, it could be built into the executable and made into a menu on one of the "Options" screens.
Useful?
-
Possibly, but I'd wait until after the mod engine overhaul.
-
would be cool if mods override but not overwrite data files. so make like 1 more folder called MODS and each mod gets a directory with same name as the ruleset which acts as a fake data folder for that mod alone. any resources or STR in the mod that have same name as the ones in data folder get overriden by the mod. latest mod being the one that overrides the last. you can probably choose mod load order too.
for ex
data/Rulesets/badalien_mod.rul
data/MODS/badadlien_mod/Resources/badalien sprites/badalien.png <=> data/Resources/badalien sprites/badalien.png
<=> means interchangeable, equivalent, same as
data/MODS/modname/ takes precedence . if nothing in MODS then is pulled from data/ instead.
essentially making mods nondestructive
and you could use a zip file with the right folder structure/files as the modpack then have some little program that just takes zip files and unpacks them in data/ so you get data/MOD/modname/*stuff* and data/Rulesets/modname.rul so you can easily uninstall later without corrupting the data folder. The folder structure completely created by the modder . All the modpack installer does is unpack it in data. EZ
and this app would read current rul files and display a list..if you choose to delete a mod then it deletes rule file and it's corresponding folder in MODS...
you can still do it the old way by putting it in data if you want
-
Possibly, but I'd wait until after the mod engine overhaul.
So is there documentation about what the mod engine overhaul will be? Or do I just need to grok the source code?
I'm okay with waiting. It would give me some time to get familiar with new system and test.
-
would be cool if mods override but not overwrite data files. so make like 1 more folder called MODS and each mod gets a directory with same name as the ruleset which acts as a fake data folder for that mod alone. any resources or STR in the mod that have same name as the ones in data folder get overriden by the mod. latest mod being the one that overrides the last. you can probably choose mod load order too.
for ex
data/Rulesets/badalien_mod.rul
data/MODS/badadlien_mod/Resources/badalien sprites/badalien.png <=> data/Resources/badalien sprites/badalien.png
<=> means interchangeable, equivalent, same as
data/MODS/modname/ takes precedence . if nothing in MODS then is pulled from data/ instead.
essentially making mods nondestructive
and you could use a zip file with the right folder structure/files as the modpack then have some little program that just takes zip files and unpacks them in data/ so you get data/MOD/modname/*stuff* and data/Rulesets/modname.rul so you can easily uninstall later without corrupting the data folder. The folder structure completely created by the modder . All the modpack installer does is unpack it in data. EZ
and this app would read current rul files and display a list..if you choose to delete a mod then it deletes rule file and it's corresponding folder in MODS...
you can still do it the old way by putting it in data if you want
I do like the sound of what you're proposing. It would be a lot easier to deploy into the data folder. The advantage of having a "mod" manifest would be to give the mod developers possibly more control over how the mod is deployed if need be. I guess I need to check into how the new system is being developed first though.
-
So is there documentation about what the mod engine overhaul will be? Or do I just need to grok the source code?
I don't think there's anything concrete out there. I just heard a couple times something like this is planned, sorry.
-
So I've revisited the problem of making a script that can update your latest mod packs and update latest nightly with plans to pick mods to install later on down the road, and I've run into bit of a snag.
The most fundamental problem to updating to the latest mod right now is the lack of consistent naming between mods found in the saved rulesets and whats up on the mod website for the mod (both name of mod and filename of the mod).
Example:
Final Mod Pack on website has a file called "Final Mod Pack_1.2.1.zip" and the ruleset in save file and options.cfg has the name "FinalModPack".
Wait.. it gets worse...
Improved Hand Objects has a file called "ImprovedHandObsPack1.1.zip" with the following ruleset string "ImprovedHandOb"
Then to be even more inconsistent..
UFO REDUX has a file called "UFO REDUX v0.5.1" with a ruleset name of "UFO_Redux_Nightly".
That's just three mods, and I shutter to think what more naming patterns ensue. I'm not sure how to proceed. I may have to whip up a partial matching algorithm to connect the installed ruleset mod to the correct mod on the website. Sounds tricky and bug prone when I think about it.
Any ideas? or suggestions?
I know the ultimate solution long term is to develop a consistent format for mods that modders can adopt, but I have feeling that will take a long while to trickle through the modding community.
-
It will definitely take a while to be adapted, but the sooner you start the sooner it will become reality.
-
It appears as the a good first standard is for mod packages on the site should start with the exact name that the ruleset uses for selection.
I believe that CAML casing is a standard for developers in the code, and imo it makes sense to apply that here.
ImprovedHandObs should be the rulename (or at lease the name of the ruleset file. later metadata additions may allow for a Proper "Title" to display in game.
ImprovedHandObs_v1.1.zip could then be the file pack name. What comes after the underscore could be less structured as modders could specify them however they like.
Your function would then only have to look for the most recent ImprovedHandObs* item on the site and DL it?
-
It appears as the a good first standard is for mod packages on the site should start with the exact name that the ruleset uses for selection.
I believe that CAML casing is a standard for developers in the code, and imo it makes sense to apply that here.
ImprovedHandObs should be the rulename (or at lease the name of the ruleset file. later metadata additions may allow for a Proper "Title" to display in game.
ImprovedHandObs_v1.1.zip could then be the file pack name. What comes after the underscore could be less structured as modders could specify them however they like.
Your function would then only have to look for the most recent ImprovedHandObs* item on the site and DL it?
Yes, that would be a very good start if the ruleset name and the filename were the same up to the first "_". That would solve a lot of problems in the future.
UFO REDUX would have to become UfoRedux_*.zip for filename and UfoRedux in the ruleset.
I think what concerns me is that there is no way enforce this. Is the answer is to build the tool first that has value and maybe everyone will see that adopting a standard is worthwhile? :)
Overall, it would be a HUGE improvement to implement your simple suggestion over the anarchy that is here right now.
I was looking for ideas on how to make the connections right now at the current state of things. I'm looking into some fuzzy matching algorithms right now. However, that might discourage some standardization. I'll keep working on it.
The other problem I face is determining if my current installation is already at the latest version of the mod that is up on the website. I don't see any "version" information being dropped in the ruleset folders in consistent format (if any and that is up to date - it isn't!).
Example:
FinalModPack ruleset file has odd characters at the beginning then "Release v. 1.0.4" and it's at 1.2.1
Improved_HandOb has quite few lines then "v 1.1" is at 1.1
UFO_Redux_Nightly has "version 0.4.8" and is at 0.5.1.
Versions are also not consistent on the file name :)
yeah, what quagmire I got myself into ...woohoo!
So what might be better would be FinalModPack_1.2.1.zip with a comment string in the top of the ruleset file "version 1.2.1". I can pull the version off the filename after the "_" and match it against the version string in the ruleset file.
That second step would help tremendously. That would get my script started working more reliably. I won't even get into "folder" structure to ease the installation process.
-
These are all very sensible steps.
Looking forward, there is a development effort underway by myk002 to standardize mod file structures and deployability. One of the fundamental aspects will be mod metadata that will handle alot of that.
Another thought is that some of this metadata is carried at the modsite. you have to enter your mod version number when you upload, so that information is stored there. If it could be retrieved during the download and annotated locally, then your system would know which version it has, and when an newer version is submitted. (Conversely, the modsite could also read this from a metadata file when that is ready).
Its true that aside from YAML which is a must for modders, there are really no other standards in place, and its kind of Wild West. Setting standards, and publishing those with the incentive of an installer that can handle the mods for the user, would, I think quickly bring the active modders on board. Those that contribute regularly, I think would gladly get on board.
Another goal in the over all OpenXcom project is a mod builder tool that will help modders by using an interface to correctly format their mods. This should greatly assist with standardization.
-
I assume loading mods via a folder would be a good first step for organization. One that I assume will happen with tftd if it's not already possible. After all the vanilla rules now load from a sub folder. There could be a mod.rul that declared a description of the folder contents and any other relevant info like version and compatibility numbers.
-
I assume loading mods via a folder would be a good first step for organization. One that I assume will happen with tftd if it's not already possible. After all the vanilla rules now load from a sub folder. There could be a mod.rul that declared a description of the folder contents and any other relevant info like version and compatibility numbers.
Well you already can load ruleset files from folders. Which should also fix issues with mods which use only one single file, e.g. the FMP which has like 32000+ lines in one file, which is as nightmare to look at ( i like small files, like the Xcom1Ruleset).
Versioning is something that is really up to the modder and not every new Version released contains changes to the ruleset, in case of Mods having Map's and Terrain's in it. Sometimes it is a collection of tine fixes to MCD or something like this.
In essence, we will have directories for mods. Each mod his own directory, with his own MAPS, TERRAIN, ROUTES, Resources etc..
And i can wait enough for this :)
-
A installer, raises a couple of questions.
First which platform (we have at least three, Windows, Linux, MacOS)
Each of those has different scripting languages etc..
Second different paths for userdirectories
Third Versioning.
The main point is updateing mods is relativly easy. As long as the Authors stick to the respective directory structure.
You download *.zip, extract it, copy contents to user directory.
So what's the big deal?
-
A installer, raises a couple of questions.
First which platform (we have at least three, Windows, Linux, MacOS)
Each of those has different scripting languages etc..
Yes, but Python runs on all 3 platforms. That is the language I'm using. So I don't see a problem there, only that I'm not a Mac user. I can test on Windows and Linux, and I have experience working on both of those with Python. This is the easiest part of the equation.
Second different paths for userdirectories
I agree, that is another problem I have running in the back of my mind. Some of this is mitigated that most OS's have builtin environment variables like "HOME" to help me get started. But, I'll have to write code that detects the platform it is running on and change paths accordingly. Some path help may have to be provided by the user at runtime. I don't see it being a huge problem... except on Mac again :)
Third Versioning.
The main point is updateing mods is relativly easy. As long as the Authors stick to the respective directory structure.
You download *.zip, extract it, copy contents to user directory.
So what's the big deal?
The mods are completely inconsistent right now with about everything that describes them, so building a programmatic interface (not just directory structure) is quite difficult. Also mods like OXC extended and Soldier's Diary mod overwrite the OXC executable, so it extracts into a different directory than the data folder like most other mods. Also, as mentioned previously, the name of the mod in the installed ruleset is different than what's up on the website (although Ivan's suggestion of saving file of what the installer installs will help). So everything is at a "chicken and egg" problem right now.
-
just FYI, mod formats will be getting a makeover soon. the new package format includes a metadata file that will contain a url. we can use that to drive autoupdate.
-
just FYI, mod formats will be getting a makeover soon. the new package format includes a metadata file that will contain a url. we can use that to drive autoupdate.
That would be great! Did you have a rough ETA of the makeover?
Should I continue down my path of making an installer/updater?
I've gotten most of the web scraping done as the site stands now(in all it's flaws), and I starting to work on determining the mods and versions installed on a local system.
-
well, I'm not sure. adding autoupdate functionality to openxcom itself would require pulling in a bunch of libraries, which I'm not sure the devs want to do. the url metadata attached to the mod would certainly take the guesswork out of your code, though, so perhaps we can just work together a bit to make sure the info you need is available to your tool.
-
well, I'm not sure. adding autoupdate functionality to openxcom itself would require pulling in a bunch of libraries, which I'm not sure the devs want to do. the url metadata attached to the mod would certainly take the guesswork out of your code, though, so perhaps we can just work together a bit to make sure the info you need is available to your tool.
How is the metadata structured? Maybe as a XML file or something similar?
Which reminds me of something. Loading a Ruleset from a folder at the moment gives you kinda a problem if you receive YAML errors. Because it doesn't tell you the file the error was found in but only the overall line number (treating the directory as a single large file i guess) not so good for debugging mods.
-
well, I'm not sure. adding autoupdate functionality to openxcom itself would require pulling in a bunch of libraries, which I'm not sure the devs want to do. the url metadata attached to the mod would certainly take the guesswork out of your code, though, so perhaps we can just work together a bit to make sure the info you need is available to your tool.
Okay, I'll just plug away on what I have and change it when the new mod format appears.
I understand about the libraries things. Even though it's easier to pull in 3rd party libraries in python, I've had to make some decisions on what I depend upon for ease of installation (some 3rd party libs require a compiler installed for python). I bet OXC would need things like lxml, xsl, and a slew of networking libs to support the feature.
-
I think it should be separate app. This would be required by Windows because its tricky to replace exe that is now running.
-
What if open com could check the rule file for an alternate base directory for terrain, maps, etc. that way those resources could live under the /Resources/modpack/ directory. That would make more sense as you would then have 2 folders to install. One under rules and one under resources, and hopefully you wouldn't have any collision between mods provided they were namespaced correctly.
-
I think it should be separate app. This would be required by Windows because its tricky to replace exe that is now running.
Most excellent point indeed!
-
What if open com could check the rule file for an alternate base directory for terrain, maps, etc. that way those resources could live under the /Resources/modpack/ directory. That would make more sense as you would then have 2 folders to install. One under rules and one under resources, and hopefully you wouldn't have any collision between mods provided they were namespaced correctly.
Well, from what I'm learning, it looks like for the future, they have this worked out for when TFTD mod comes out.
-
How is the metadata structured? Maybe as a XML file or something similar?
Which reminds me of something. Loading a Ruleset from a folder at the moment gives you kinda a problem if you receive YAML errors. Because it doesn't tell you the file the error was found in but only the overall line number (treating the directory as a single large file i guess) not so good for debugging mods.
The metadata will be structured as a (very simple) yaml file. Separate from the ruleset(s). For loading errors, the log will show which file it was attempting to load when the yaml error happened. It doesn't appear in the popup message (that would actually take a bit of work to make happen), but it will be in the log.