OpenXcom Forum
Modding => Tools => Topic started by: memmaker on March 17, 2020, 11:08:38 am
-
I will dump some tools I use for modding here.
They need Python3 and ruamel.yaml installed.
Install ruamel.yaml:
pip install ruamel.yaml
DISCLAIMER: These tools come as is, no warranty. If they wreck your rulesets, you have been warned. Try them on copies first. Have backups.
rul2csv - extract information from rulesets as CSV
Usage:
rul2csv "TOPLEVELKEY:FIELDNAME1(,...)" FILENAME_OR_DIRECTORY
Where the fieldnames support lists and key/value pairs with the following syntax:
SUB.FIELDNAME for key/value pairs
SUB-FIELDNAME for lists
You can override the automatic id-field detection by specifying it on the toplevel key like this (eg. for research):
TOPLEVELKEY-IDFIELD: FIELDNAME1,..
Example usage:
rul2csv "items:power,clipSize,confSnap.shots,confAimed.shots,damageAlter.ArmorEffectiveness,tags.GUN_TYPE,tags.CRIT_CHANCE" .
rul2csv "research-name:points,cost" . > csv_data/research.csv
rul2csv "armors:frontArmor,sideArmor,rearArmor,underArmor,damageModifier-none,damageModifier-projectile,damageModifier-incendiary" . > csv_data/armors.csv
Example Output
$ cat test1.rul
items:
- type: STR_PISTOL
power: 10
accuracy: 20
$ rul2csv "items:power" test1.rul
type,power
STR_PISTOL,10
csv2rul - Creates rulesets from CSV data
Usage:
csv2rul items.csv
Example Usage:
$ csv2rul items test.csv
items:
- type: STR_PISTOL
power: 10
mergerules - Merges multiple rule files using one top-level key
mergerules items file1.rul file2.rul file3.rul (...)
mergerules items path
Example Usage:
$ cat test1.rul
items:
- type: STR_PISTOL
power: 10
accuracy: 20
$ cat test2.rul
items:
- type: STR_PISTOL
power: 100
tuCost: 15
$ mergerules items test1.rul test2.rul
items:
- type: STR_PISTOL
power: 100
accuracy: 20
tuCost: 15
splitrules - Split rule item lists in two, defined by some fields
Usage:
splitrules [OUTPUTMODE] "TOPLEVELKEY:FIELDNAME1(,...)" filename.rul
Example usage:
$ mergerules items test1.rul test2.rul > merged.rul
$ splitrules "items:power" merged.rul
items:
- type: STR_PISTOL
accuracy: 20
tuCost: 15
------------------------------------
items:
- type: STR_PISTOL
power: 100
$ splitrules 1 "items:power" merged.rul
items:
- type: STR_PISTOL
accuracy: 20
tuCost: 15
$ splitrules 2 "items:power" merged.rul
items:
- type: STR_PISTOL
power: 100
patchrules - Modify existing ruleset with delta from CSV
Usage:
patchrules TOPLEVELKEY patchData.csv targetRules.rul
Exmaple Usage:
$ cat merged.rul
items:
- type: STR_PISTOL
power: 100
accuracy: 20
tuCost: 15
$ cat test.csv
type,power
STR_PISTOL,66
$ patchrules items test.csv merged.rul
items:
- type: STR_PISTOL
power: 66
accuracy: 20
tuCost: 15
-
Using the toolchain to create a day-time mod:
rul2csv "alienDeployments:shade,maxShade,minShade" ./XComFiles/ > temp/alienDeployments.csv
awk -F, '{
threshold=7;
shd=$2;
mas=$3;
mis=$4;
if(shd>threshold)shd=threshold;
if(mas>threshold)mas=threshold;
if(mis>threshold)mis=threshold;
OFS=",";
if(NR==1){print $0;}
else{print $1, shd, mas, mis;}
}' <temp/alienDeployments.csv > temp/daylight.csv
csv2rul temp/daylight.csv > temp/daylight.rul
-
Added one more, this one needs Graphviz installed
graphfromrules - Create Graphviz files from rulesets
Usage:
graphfromrules "research:dependencies,unlocks" "cost,points" research.rul
Example usage:
$ graphfromrules "research:dependencies,unlocks" "cost,points" mergedResearch.rul
// Ruleset Graph
digraph {
STR_BASIC_DEPLOYMENT [label=STR_BASIC_DEPLOYMENT cost=180 points=20]
STR_BASIC_DEPLOYMENT -> STR_BASIC_DEPLOYMENT [label=dependencies]
STR_ADVANCED_DEPLOYMENT1 [label=STR_ADVANCED_DEPLOYMENT1 cost=480 points=20]
STR_ADVANCED_DEPLOYMENT1 -> STR_BASIC_DEPLOYMENT [label=dependencies]
...
STR_OBJECTIVES_FINAL [label=STR_OBJECTIVES_FINAL]
STR_OBJECTIVES_FINAL -> STR_OBJECTIVES_FINAL [label=dependencies]
}
Save this as a file with .GV extension (for GraphViz) and you can load it up in Gephi. From there one can layout the whole thing and export it as GEXF file.
Sample output (X-Comfiles Tech Tree):
https://memmaker.github.io/graphs/xcom-files-tech.html
Workflow for creating tech-trees:
# merging all research rules
mergerules research ./mods/XComFiles/ > dataSets/XfilesResearch.rul
# split the dependencies off
splitrules 2 "research-name:dependencies" ./dataSets/XfilesResearch.rul > ./dataSets/XfilesDepends.rul
# create the graph
graphfromrules "research-name:dependencies" "cost,points" ./dataSets/XfilesDepends.rul > ./dataSets/XfilesDepends.gv
-
Python script for the extraction of Polygons from World.dat type files:
#!/usr/local/bin/python3
import sys
from pprint import pprint
#- load WORLD.DAT as binary
#- .DAT is a sequence of x1, y1, x2, y2, x3, y3, x4, y4, tex values
#- coordinates are 16-bit int, texture is 32-bit int
#- multiply coordinates by 0.125
#- output as - [x1, y1, x2, y2, x3, y3, x4, y4, tex] for yaml
#- if x4 == -1, don't output x4, y4
def readOneRecord(f):
record = []
for i in range(0, 8):
readValue = f.read(2)
if not readValue:
return record
intCoord = int.from_bytes(readValue, byteorder='little', signed=True)
if i == 6 and intCoord == -1:
f.read(2)
break
record.append(intCoord * 0.125)
tex = int.from_bytes(f.read(4), byteorder='little', signed=True)
record.insert(0,tex)
return record
if len(sys.argv) > 1:
path = sys.argv[1]
with open(path,"rb") as f:
nextRecord = readOneRecord(f)
while len(nextRecord) > 0:
pprint(nextRecord)
nextRecord = readOneRecord(f)
-
Python script for converting ruleset globe polygons into .DAT files:
#!/usr/local/bin/python3
import sys
from pprint import pprint
from ruamel.yaml import YAML
import warnings
from ruamel.yaml.error import ReusedAnchorWarning
def parseRuleFile(filename):
with open(filename) as file:
yaml = YAML()
yaml.allow_duplicate_keys = True
data = list(yaml.load_all(file))
return data
def handleRuleFile(filename):
# print("Trying to parse YAML in " + str(filename))
data = parseRuleFile(filename)
for docs in data:
for key, value in docs.items():
if key == "globe" and "polygons" in value:
return value["polygons"]
def writePolygonsToDatFile(polygons):
with open("NEWWORLD.DAT","wb") as f:
for record in polygons:
isFirst = True
tex = -1
for number in record:
if isFirst:
isFirst = False
tex = number
continue
number *= 8
f.write(int(number).to_bytes(2, 'little', signed=True))
if len(record) == 7: # short record
f.write((-1).to_bytes(2, 'little', signed=True))
f.write((0).to_bytes(2, 'little', signed=True))
f.write(tex.to_bytes(4, 'little'))
if len(sys.argv) > 1:
path = sys.argv[1]
polygons = handleRuleFile(path)
writePolygonsToDatFile(polygons)
-
I made a bit of a more visual explanation how to use these on Windows: https://github.com/pedroterzero/oxce-yaml-helper/wiki/Memmaker-ruleset-tools-python
-
Nice, ty)