Ruleset tools for the command line

[memmaker]

I will dump some tools I use for modding here.

They need Python3 and ruamel.yaml installed.

Install ruamel.yaml:

Code: [Select]

pip install ruamel.yaml

DISCLAIMER: These tools come as is, no warranty. If they wreck your rulesets, you have been warned. Try them on copies first. Have backups.

rul2csv - extract information from rulesets as CSV

Usage:

Code: [Select]

rul2csv "TOPLEVELKEY:FIELDNAME1(,...)" FILENAME_OR_DIRECTORY
Where the fieldnames support lists and key/value pairs with the following syntax:

Code: [Select]

SUB.FIELDNAME for key/value pairs
SUB-FIELDNAME for lists
You can override the automatic id-field detection by specifying it on the toplevel key like this (eg. for research):

Code: [Select]

TOPLEVELKEY-IDFIELD: FIELDNAME1,..

Example usage:

Code: [Select]

rul2csv "items:power,clipSize,confSnap.shots,confAimed.shots,damageAlter.ArmorEffectiveness,tags.GUN_TYPE,tags.CRIT_CHANCE" .

rul2csv "research-name:points,cost" . > csv_data/research.csv

rul2csv "armors:frontArmor,sideArmor,rearArmor,underArmor,damageModifier-none,damageModifier-projectile,damageModifier-incendiary" . > csv_data/armors.csv


Example Output

Code: [Select]

$ cat test1.rul
items:
  - type: STR_PISTOL
    power: 10
    accuracy: 20

$ rul2csv "items:power" test1.rul
type,power
STR_PISTOL,10

csv2rul - Creates rulesets from CSV data

Usage:

Code: [Select]

csv2rul items.csv

Example Usage:

Code: [Select]

$ csv2rul items test.csv
items:
  - type: STR_PISTOL
    power: 10

mergerules - Merges multiple rule files using one top-level key

Code: [Select]

mergerules items file1.rul file2.rul file3.rul (...)
mergerules items path

Example Usage:

Code: [Select]

$ cat test1.rul
items:
  - type: STR_PISTOL
    power: 10
    accuracy: 20
$ cat test2.rul
items:
  - type: STR_PISTOL
    power: 100
    tuCost: 15
$ mergerules items test1.rul test2.rul
items:
  - type: STR_PISTOL
    power: 100
    accuracy: 20
    tuCost: 15

splitrules - Split rule item lists in two, defined by some fields

Usage:

Code: [Select]

splitrules [OUTPUTMODE] "TOPLEVELKEY:FIELDNAME1(,...)" filename.rul

Example usage:

Code: [Select]

$ mergerules items test1.rul test2.rul > merged.rul
$ splitrules "items:power" merged.rul
items:
  - type: STR_PISTOL
    accuracy: 20
    tuCost: 15
------------------------------------
items:
  - type: STR_PISTOL
    power: 100
$ splitrules 1 "items:power" merged.rul
items:
  - type: STR_PISTOL
    accuracy: 20
    tuCost: 15
$ splitrules 2 "items:power" merged.rul
items:
  - type: STR_PISTOL
    power: 100

patchrules - Modify existing ruleset with delta from CSV

Usage:

Code: [Select]

patchrules TOPLEVELKEY patchData.csv targetRules.rul

Exmaple Usage:

Code: [Select]

$ cat merged.rul
items:
  - type: STR_PISTOL
    power: 100
    accuracy: 20
    tuCost: 15
$ cat test.csv
type,power
STR_PISTOL,66
$ patchrules items test.csv merged.rul
items:
  - type: STR_PISTOL
    power: 66
    accuracy: 20
    tuCost: 15


[memmaker]

Using the toolchain to create a day-time mod:

Code: [Select]

rul2csv "alienDeployments:shade,maxShade,minShade" ./XComFiles/ > temp/alienDeployments.csv

awk -F, '{
    threshold=7;
    shd=$2;
    mas=$3;
    mis=$4;
    if(shd>threshold)shd=threshold;
    if(mas>threshold)mas=threshold;
    if(mis>threshold)mis=threshold;
    OFS=",";
    if(NR==1){print $0;}
    else{print $1, shd, mas, mis;}
}' <temp/alienDeployments.csv > temp/daylight.csv

csv2rul temp/daylight.csv > temp/daylight.rul


[memmaker]

Added one more, this one needs Graphviz installed

graphfromrules - Create Graphviz files from rulesets

Usage:

Code: [Select]

graphfromrules "research:dependencies,unlocks" "cost,points" research.rul
Example usage:

Code: [Select]

$ graphfromrules "research:dependencies,unlocks" "cost,points" mergedResearch.rul
// Ruleset Graph
digraph {
STR_BASIC_DEPLOYMENT [label=STR_BASIC_DEPLOYMENT cost=180 points=20]
STR_BASIC_DEPLOYMENT -> STR_BASIC_DEPLOYMENT [label=dependencies]
STR_ADVANCED_DEPLOYMENT1 [label=STR_ADVANCED_DEPLOYMENT1 cost=480 points=20]
STR_ADVANCED_DEPLOYMENT1 -> STR_BASIC_DEPLOYMENT [label=dependencies]
...
STR_OBJECTIVES_FINAL [label=STR_OBJECTIVES_FINAL]
STR_OBJECTIVES_FINAL -> STR_OBJECTIVES_FINAL [label=dependencies]
}
Save this as a file with .GV extension (for GraphViz) and you can load it up in Gephi. From there one can layout the whole thing and export it as GEXF file.

Sample output (X-Comfiles Tech Tree):

https://memmaker.github.io/graphs/xcom-files-tech.html


Workflow for creating tech-trees:

Code: [Select]

# merging all research rules
mergerules research ./mods/XComFiles/ > dataSets/XfilesResearch.rul

# split the dependencies off
splitrules 2 "research-name:dependencies" ./dataSets/XfilesResearch.rul > ./dataSets/XfilesDepends.rul

# create the graph
graphfromrules "research-name:dependencies" "cost,points" ./dataSets/XfilesDepends.rul > ./dataSets/XfilesDepends.gv


[memmaker]

Python script for the extraction of Polygons from World.dat type files:

Code: [Select]

#!/usr/local/bin/python3

import sys
from pprint import pprint
#- load WORLD.DAT as binary
#- .DAT is a sequence of x1, y1, x2, y2, x3, y3, x4, y4, tex values
#- coordinates are 16-bit int, texture is 32-bit int
#- multiply coordinates by 0.125
#- output as - [x1, y1, x2, y2, x3, y3, x4, y4, tex] for yaml
#- if x4 == -1, don't output x4, y4

def readOneRecord(f):
    record = []
    for i in range(0, 8):
        readValue = f.read(2)
        if not readValue:
            return record
        intCoord = int.from_bytes(readValue, byteorder='little', signed=True)
        if i == 6 and intCoord == -1:
            f.read(2)
            break
        record.append(intCoord * 0.125)
    tex = int.from_bytes(f.read(4), byteorder='little', signed=True)
    record.insert(0,tex)
    return record

if len(sys.argv) > 1:
    path = sys.argv[1]
    with open(path,"rb") as f:
        nextRecord = readOneRecord(f)
        while len(nextRecord) > 0:
            pprint(nextRecord)
            nextRecord = readOneRecord(f)


[memmaker]

Python script for converting ruleset globe polygons into .DAT files:

Code: [Select]

#!/usr/local/bin/python3

import sys
from pprint import pprint
from ruamel.yaml import YAML
import warnings
from ruamel.yaml.error import ReusedAnchorWarning

def parseRuleFile(filename):
    with open(filename) as file:
        yaml = YAML()
        yaml.allow_duplicate_keys = True
        data = list(yaml.load_all(file))
        return data

def handleRuleFile(filename):
#    print("Trying to parse YAML in " + str(filename))
    data = parseRuleFile(filename)
    for docs in data:
        for key, value in docs.items():
            if key == "globe" and "polygons" in value:
                return value["polygons"]

def writePolygonsToDatFile(polygons):
    with open("NEWWORLD.DAT","wb") as f:
        for record in polygons:
            isFirst = True
            tex = -1
            for number in record:
                if isFirst:
                    isFirst = False
                    tex = number
                    continue
                number *= 8
                f.write(int(number).to_bytes(2, 'little', signed=True))
            if len(record) == 7: # short record
                f.write((-1).to_bytes(2, 'little', signed=True))
                f.write((0).to_bytes(2, 'little', signed=True))
            f.write(tex.to_bytes(4, 'little'))

if len(sys.argv) > 1:
    path = sys.argv[1]
    polygons = handleRuleFile(path)
    writePolygonsToDatFile(polygons)


[pedroterzero]

I made a bit of a more visual explanation how to use these on Windows: https://github.com/pedroterzero/oxce-yaml-helper/wiki/Memmaker-ruleset-tools-python

[Finnik]

Nice, ty)