Author Topic: Upgrading to the nightly  (Read 96401 times)

Offline Warboy1982

  • Administrator
  • Commander
  • *****
  • Posts: 2333
  • Developer
    • View Profile
Upgrading to the nightly
« on: May 07, 2015, 09:34:55 am »
as of today, the data folder structure will be drastically altered, and a new standard will be enforced for mods.

more details will appear in this thread, i just can't be bothered detailing them all yet.

full credit to myk002 for his work on this.
« Last Edit: May 07, 2015, 10:11:18 pm by Warboy1982 »

Offline myk002

  • Colonel
  • ****
  • Posts: 227
    • View Profile
Re: Upgrading to the nightly
« Reply #1 on: May 07, 2015, 09:46:25 am »
Overview:
In order to support both UFO and TFTD, we have to reorganize the OpenXcom directory structure a little bit.  Once this goes in, you'll have to reinstall OpenXcom from scratch, reinstall your UFO data files, reinstall your mods, and re-enable the mods in the Options/Mods screen.  And since we have to reinstall stuff anyway, we're taking the opportunity to change how mods are handled so it's easier to uninstall and upgrade them.

Details:
  • there is no folder named "data" anymore.  Now there are multiple folders where "data" used to be:
    • common: files used by OpenXcom that aren't specific to any one mod
    • standard: the built-in mods that come with OpenXcom
    • TFTD: copy your Terror From the Deep data into here in preparation for when OpenXcom enables TFTD
    • UFO: copy your UFO data into here
  • if you can't write to the TFTD or UFO directories in the data dir (for example if OpenXcom is installed in a system directory that you don't have write permissions for), you can create TFTD and UFO directories in your user dir (where your savegames are) and copy the files in there.
  • mod folders go in the "mods" folder in your user dir .  This directory gets created when you start OpenXcom for the first time.
  • mods are now installed in separate folders: one folder per mod, not all lumped together in the data directory like they used to be
  • to uninstall a mod, just delete the folder -- no more figuring out which files were overwritten!
  • to upgrade a mod, just delete the folder and replace it with the new version
  • enable mods in the options/mods screen, just like before, but there is some new stuff now:
    • the list of mods is actually a list of mods, not a list of rulesets like before.  if a mod has multiple rulesets, you only need to enable one item
    • hovering over a mod with the mouse will show some info about it in the bottom tooltip area: version, author, and description
    • you can use the up/down arrows to reorder the mods.  mods that load last override mods that load before them if they happen to modify the same records or provide the same files
    • there is a combobox at the top labeled 'Game type'.  This is where you'll choose whether to play UFO or TFTD (when it comes out).  Total conversion mods, like Piratez, will also appear here once they are updated to the new system.  The list of mods shown below is filtered by the game type you have selected.  When loading, the list of saves is also filtered by the selected active Game type.  Games saved before this change assume UFO as the game type, but it's up to the player to ensure that the same mods that were used to save the game are active now.

Screenshot: https://imgur.com/kqKpNsa


Step by Step Guide for Upgrading:
  • Copy your original UFO data files somewhere safe
  • Uninstall OpenXcom and remove any files left behind in the "data" directory
  • Install OpenXcom
  • Copy your original UFO data files into the UFO directory (either in the data dir or the user dir).  You can apply the universal patch files over the files here, but don't include any files from mods
  • Start OpenXcom to make sure it loads ok
  • Extract any mods you want to install into the "mods" directory, beneath where your savegames are located.  Ensure each mod is extracted into a separate directory under the mods directory.  See the example below.
  • Start OpenXcom, go to Options -> Mods, and enable mods you want to use.
  • Your old savegames will work just fine as long as the same mods you had loaded before are still loaded.

How to install a mod correctly:

Go to your mods directory, where your savegames are.  Extract the mod into a new subdirectory.  WinZip has an "Extract to" option that creates a directory whose name is based on the archive name.  It doesn't really matter what the directory name is as long as it is unique.  Some mods are packed with extra directories at the top, so you may need to move files around inside the new mod directory to get things straighted out.  For example, if you extract a mod to mods/LulzMod and you see something like:

    mods/LulzMod/data/TERRAIN/
    mods/LulzMod/data/Rulesets/

and so on, just move everything up a level so it looks like:

    mods/LulzMod/TERRAIN/
    mods/LulzMod/Rulesets/


Additional info for mod authors:
  • the only mod structure change is that ruleset files can now go in the top-level folder for your mod.  they can also appear in a "Rulesets" folder just like before, if that is more convenient for the mod.
  • Mods can have a file named metadata.yml that defines the following properties (default values shown in parens):
    • name (the directory name)
      the name of the mod.  this gets shown in the in-game mods list.  the names of the individual ruleset files no longer get shown anywhere.  when a user enables the mod by this name, all the mod's ruleset files will get loaded
    • version ("1.0")
    • author ("unknown author")
    • description ("No description.")
      these get shown in the tooltip when hovering over the mod name in the mods list
    • id (same as the name)
      the internal id that this mod is known by.  savegames, configuration files, and other mods' metadata files will refer to the mod by this string.  if you make the value of this attribute the same as the id you used when uploading to the mod portal (https://www.openxcom.com), then autoupdate tools may be able to use it to check for new versions.
    • master ("xcom1")
      the id of the master mod that this mod applies to (all current mods apply to the "xcom1" master).  if the mod applies to all masters (e.g. XcomUtil_Statstrings), use an asterisk (master: "*").
    • isMaster (false)
      set to true for total conversion mods that constitute their own "game type".  masters are listed in the combobox at the top of the mods page and all mods that do not declare them as their own masters will be filtered out.  masters can explicitly declare an upstream master to load, even though the upstream master will not be listed as active in the saved options.cfg.
    • loadResources ()
      ignored unless isMaster is set to true.  this can be a list of directory names in the data folder that contains resource files used by the mod.  this is how xcom1 loads the resources in UFO/ and how xcom2 loads the resources in TFTD/ (see their metadata.yml files for examples).  total conversion mods can use this to load xcom1 or xcom2 resources without declaring them as a master if they don't need the xcom1/xcom2 rulesets.
  • see metadata.yml files in https://github.com/SupSuper/OpenXcom/tree/master/bin/standard subdirectories for examples.
  • current mods work without changes, but must be reinstalled into their own separate directories in the mods folder.  their metadata is generated based on the defaults listed above
  • to have your mod work with both current openxcom and be ready for the upcoming changes, leave the rulesets in the Ruleset folder but add a metadata.yml file to your mod's root directory.  once everyone has an updated openxcom installation, you can migrate the ruleset files to your mod's root directory as well.
  • the sprite limit is now *per mod*, not *per ruleset*.  this means that you can split your ruleset into multiple files and sprites declared in one can be referred to in the others.  The sprite index limit is 1000 for regular mods, but is effectively infinite for base masters (that is, masters that do not declare a master of their own) .  Base masters can use as many sprite indices as they want.  Any mods that declare a master, though, must only use indices higher than what the master uses to ensure no collisions.  So, for example, if a base master uses ids for sprites up to 4350, maybe all mods for that master could start their ids at 6000 or so.  It is fine to have sprite id overlap between mods, as each mod loads sprites in a different namespace.  The engine will keep them straight and each mod will refer correctly to its own sprites.

more mod metadata can be added easily later.  for example, we could introduce mod dependency and incompatibility information so mod authors can require other specific mods to be loaded before theirs or prevent two known incompatible mods from being loaded at the same time.


Step by step guide for updating your mod to make it ready for the new order:

First, let's consider a simple mod that consists of just a ruleset and a resource file.  Alternate Smoke is a good example of this: https://www.openxcom.com/mod/alternative-smoke
Inside the download, it looks like this:

    $ unzip -l alternative_smoke_v1.0.zip
    Archive:  alternative_smoke_v1.0.zip
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  04-17-2015 10:04   data/
            0  04-17-2015 10:03   data/Resources/
            0  04-17-2015 10:04   data/Resources/AlternativeSmoke/
            0  04-17-2015 10:04   data/Resources/AlternativeSmoke/SMOKE/
         3750  03-24-2015 21:46   data/Resources/AlternativeSmoke/SMOKE/smoke_cloud.gif
            0  04-17-2015 10:04   data/Ruleset/
          227  04-17-2015 10:07   data/Ruleset/AlternativeSmoke.rul
    ---------                     -------
         3977                     7 files

let's condense that to:

    data/Resources/AlternativeSmoke/SMOKE/smoke_cloud.gif
    data/Ruleset/AlternativeSmoke.rul

Now, since mods no longer go in "data", but rather in their own directory under mods, let's chop off the top level so it looks like this:

    Resources/AlternativeSmoke/SMOKE/smoke_cloud.gif
    Ruleset/AlternativeSmoke.rul

Now users can extract the archive directly into the data directory for older OpenXcom installs or into a named subdirectory of mods for new OpenXcom installs.

Even though the new mod format allows rulesets to live in the top-level directory (i.e. AlternativeSmoke.rul instead of Ruleset/AlternativeSmoke.rul), let's leave it where it is for now in the Ruleset directory so it will work with older OpenXcom installs.  Later, once everyone is using a new OpenXcom build, mods can start just putting their ruleset files in the top level directory.

That is all that is minimally required, but if you want to see information about the mod when hovering over it on the Options/Mods page, you'll need to create a file named metadata.yml and add it to the root of your mod:

    metadata.yml
    Resources/AlternativeSmoke/SMOKE/smoke_cloud.gif
    Ruleset/AlternativeSmoke.rul

metadata.yml should have the following contents:

    name: "Alternative Smoke"
    version: 1.0
    description: "A less disturbing/distracting smoke animation."
    author: "Robin"
    master: "xcom1"


That's it!  For most mods, this is all that needs to be done.  The Final Mod Pack (https://www.openxcom.com/mod/final-mod-pack), for example, is already in the correct format -- that is, the MAPS, Resources, ROUTES, etc. directories are already at the top level.  It would just need a metadata.yml file like the one described above to make it ready.


Total conversion mods need a bit more work.  Let's take Piratez as an example (https://www.openxcom.com/mod/piratez).  It's metadata.yml file would look like this:

    name: "Piratez"
    version: 0.9h
    description: "The year is 2600. You run a gang of mutant pirates. Rob aliens and their human proxies for fun, profit and power."
    author: "Dioxine"
    id: piratez
    isMaster: true
    master: xcom1

This will cause Piratez to appear in the "Game type" list at the top of the mods page.  Other mods that modify Pirates can declare "pirates" as their master.  Note that Pirates would need a few other changes to be completely functional as a master, like unifying the sprite numbers across its rulesets.
« Last Edit: May 18, 2015, 01:40:16 am by myk002 »

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9145
    • View Profile
Re: Upgrading to the nightly
« Reply #2 on: May 07, 2015, 10:59:00 am »
SWEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEET!!!!1111

Offline Jo5hua

  • Captain
  • ***
  • Posts: 78
  • ModSite
    • View Profile
Re: Upgrading to the nightly
« Reply #3 on: May 07, 2015, 11:11:21 am »
as of today, the data folder structure will be drastically altered, and a new standard will be enforced for mods.

more details will appear in this thread, i just can't be bothered detailing them all yet.

How will this affect the modsite? Should we add something to distinguish the mods, or just utilize the "Game Version Number"... Thats the minimum game version number field that people enter when they add mods (currently not showing on the public mod page).

Offline kkmic

  • Commander
  • *****
  • Posts: 582
  • Undefined
    • View Profile
Re: Upgrading to the nightly
« Reply #4 on: May 07, 2015, 11:11:50 am »
This was soooooooo needed

LE:
@Jo5shua: let things settle for a while, before even starting to think about what should be changed in the mod portal
« Last Edit: May 07, 2015, 11:14:24 am by kkmic »

Offline 7Saturn

  • Colonel
  • ****
  • Posts: 457
    • View Profile
Re: Upgrading to the nightly
« Reply #5 on: May 07, 2015, 11:55:54 am »
Just one question: When will this be implemented? I just wrote a webpage on how to compile and install openxcom on ubuntu, but with changed data-structure, I will have to rewrite that and recompile it as well. Would be nice to know, when this will take effect...

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9145
    • View Profile
Re: Upgrading to the nightly
« Reply #6 on: May 07, 2015, 12:01:56 pm »
It says "as of today" in the first post, first sentence.
It is already implemented and ready to use.

Offline 7Saturn

  • Colonel
  • ****
  • Posts: 457
    • View Profile
Re: Upgrading to the nightly
« Reply #7 on: May 07, 2015, 12:08:49 pm »
So, »effective immediately«. OK, work's coming up...

Offline Warboy1982

  • Administrator
  • Commander
  • *****
  • Posts: 2333
  • Developer
    • View Profile
Re: Upgrading to the nightly
« Reply #8 on: May 07, 2015, 12:49:35 pm »
So, »effective immediately«. OK, work's coming up...

you might want to hold off on that, it's not completely finalized yet.

Offline 7Saturn

  • Colonel
  • ****
  • Posts: 457
    • View Profile
Re: Upgrading to the nightly
« Reply #9 on: May 07, 2015, 01:15:40 pm »
I trust, there will be a notification here, when that has happend.

Offline Ishmaeel

  • Captain
  • ***
  • Posts: 55
    • View Profile
Re: Upgrading to the nightly
« Reply #10 on: May 07, 2015, 02:01:35 pm »
Ah. Time to update my auto-download-'n-install script.

I've been dropping OXC directly into the Steam installation folder for client integration. I hope there will be provisions to specify the data folder location. May I report issues (if any) with that approach?

Offline Warboy1982

  • Administrator
  • Commander
  • *****
  • Posts: 2333
  • Developer
    • View Profile
Re: Upgrading to the nightly
« Reply #11 on: May 07, 2015, 02:23:10 pm »
sure, stress testing would be appreciated at this point

Offline hellrazor

  • Commander
  • *****
  • Posts: 2027
  • Deep Ruleset Digger & Bughunter
    • View Profile
    • Github Account
Re: Upgrading to the nightly
« Reply #12 on: May 07, 2015, 03:09:18 pm »
Just pulled this new awesome stuff from github.

I have already changed the directorystructure of my mods accordingly.
And they are loaded as intended.

But this metadata.yaml file doesn't seem to work.

Code: [Select]
name: "Hardmode Expansion Mod"
version: 0.7 Beta
description: "This mod intends to make the gameplay more challenging, featuring a couple of new races, new Weapons for Xcom and the Aliens. You will encounter bigger Alien Crews on all Missions."
author: "hellrazor"
master: "xcom1"

It exists in the Mods main directory.

PLEASE change your mky002!
The file is called metadata.yml not *.yaml. :)
« Last Edit: May 07, 2015, 03:15:11 pm by hellrazor »

Offline Meridian

  • Global Moderator
  • Commander
  • *****
  • Posts: 9145
    • View Profile
Re: Upgrading to the nightly
« Reply #13 on: May 07, 2015, 03:14:42 pm »
It was renamed to "metadata.yml" in the meantime.

Offline hellrazor

  • Commander
  • *****
  • Posts: 2027
  • Deep Ruleset Digger & Bughunter
    • View Profile
    • Github Account
Re: Upgrading to the nightly
« Reply #14 on: May 07, 2015, 03:15:47 pm »
It was renamed to "metadata.yml" in the meantime.

Well i just checked the defaults ruleset they have so it occured to me also.