OpenXcom Forum
Modding => Tools => Topic started by: pedroterzero on December 14, 2020, 03:17:24 pm
-
Hi there!
I recently started work on a visual studio code extension that works in addition to the existing excellent Ruleset Validator (https://openxcom.org/forum/index.php/topic,6552.0.html) by SupSuper.
My aim is to make a bit more of modding IDE experience for OpenXcom (Extended) rulesets by adding some IDE like features to the ruleset editing itself. The goal is to increase the quality of life during modding even more by solving a few of the problems that I saw myself while doing some work on mods. It should also raise the general quality of the rulesets themselves by pointing out issues that are easily overlooked if they're only inspected manually.
Some of the more interesting features are:
- Jump from a string ID or sprite ID to its definition. This saves a lot switching back and forth manually between files
- Y-script syntax highlighting/colouring
- Spreadsheet/CSV editor. This allows you to view and edit a ruleset as a spreadsheet
- See translations for string IDs by hovering on them
- Reference checking. This means that it checks automatically whether a string ID you've entered actually exists and whether it is of the right type (so if something expects a research entry, it checks that it is)
- Documentation hover. The tool integrates the documentation from the ruleset wiki page (https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)) into the tool itself, so you can hover over a rule to see what it means and what the options are.
- Context aware autocomplete across files: offer autocomplete on items that are in other files, and also of the correct type (so if a research entry is required, only offer those and not manufacture rules for example)
- Preview extraSprites by hovering over them in the ruleset file itself
- Finds duplicate string ids and sprite ids. That way if you inadvertently defined something twice, you can easily see it in an overview (this feature is disabled by default)
Some features I hope to add in the future:
- Show missing translations
- Detect common problems that may lead to crashes or segmentation faults and prevent them before they happen
- and more
Where to get it and more information
The extension and instructions how to install it, and a lot more information can be found on the visual studio marketplace: https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper (https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper)
Some demos of what this looks like:
Jump to definition
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/go-to-definition.gif)
Syntax highlighting/colouring
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/syntax-highlighting.png)
CSV/Spreadsheet editor
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/csv-editor.gif)
Reference checking:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/reference-checking.gif)
Documentation hover:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/documentation-hover.gif)
-
Making a second post so I can link (sorry :()
-
This a great tool I was using for a while (as I was honored to be an alpha tester), it is invaluable for making big mods, so I refuse to do modding without it, and this thing is in my must-have list (together with ruleset tools and map editor), so I recommend to use it to everyone!
-
Making a second post so I can link (sorry :()
how could you! :>
y-scripts syntax highlighting
I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.
-
I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.
I think it would be great!
And also auto-suggest of it, like how IDE does, I'm casting `itemRule.get` and it suggests what can I get based on what hook I am scripting. Highlighting variables, tag names, remembering variables to warn it is not defined in the beginning of the script or there is no such tag.
I am sure that would really help to populate y-scripting to people if there would be a good helper around.
-
This a great tool I was using for a while (as I was honored to be an alpha tester), it is invaluable for making big mods, so I refuse to do modding without it, and this thing is in my must-have list (together with ruleset tools and map editor), so I recommend to use it to everyone!
Thank you for your kind words!
I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.
For now it would probably mostly about colouring the code in VSCode. I was hoping to base it an another language and add some specific stuff. What language would be a good pick for this?
I was hoping to use this as a basis: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide -- VSCode uses TextMate internally. I have never done it before so it'll take a bit of experimentation ;)
It would help to have all symbols, that way I can possibly even add some semantic highlighting later, if the syntax highlighting works.
I think it would be great!
And also auto-suggest of it, like how IDE does, I'm casting `itemRule.get` and it suggests what can I get based on what hook I am scripting. Highlighting variables, tag names, remembering variables to warn it is not defined in the beginning of the script or there is no such tag.
I am sure that would really help to populate y-scripting to people if there would be a good helper around.
This is a more advanced topic I think. This would require me to actually parse the code probably, but we'll see. First I'll start with the colouring at some point, that's probably complicated enough ;)
-
y-scripts are most close to assembler, one operation per line (more accurately per `;` as it could be in one line if split by `;`). There are some exceptions like `if` blocks that can combine multiple comparison and create scope.
Variable for now can be defined only on top of script, `var TYPE NAME VALUE;` in far future I plan allow variables defined in each scope.
Each line look like
TypeAName.funcName valueOfTypeA arg1 arg2;or
valueOfTypeA.funcName arg1 arg2;Both mean same.
-
y-scripts are most close to assembler, one operation per line (more accurately per `;` as it could be in one line if split by `;`). There are some exceptions like `if` blocks that can combine multiple comparison and create scope.
Thanks. That helps. I'll try to get it in a test version sometime (I am currently working on some other features, improved logic checking and context aware autocomplete). If I get anywhere I'll let you know!
-
I just released v0.6.0 on marketplace (https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper). This is quite a big release, although it maybe doesn't seem like it.
Most importantly I added a mechanism to do specific logic checking on rulesets. This allows for checking and preventing some common (and uncommon) crashes and segmentation faults that might occur. Some occur when starting OXCE, others happen only when the specific rule triggers.
Having them show up as problems in vscode should be big time-saver and QA improvement. Many thanks to Filip H and Finnik for providing me with input on these. I hope to add more detection as I become aware of the possible problems that could occur. Feel free to let me know when there's something the extension isn't picking up.
I am currently working on finishing a few other features too, namely scripting highlighting, context aware autocomplete (only completes the correct types, and works across files) and scripting parsing (so you can tell if you've made mistakes in the script in vscode itself).
A preview of the highlighting in action:
(https://i.postimg.cc/JndpwyJp/highlighting.png)
-
Praise colors!
(https://steamuserimages-a.akamaihd.net/ugc/440605165249728813/4A0EAB3A009688E19D70680256247FB17F708B20/)
-
Can we pin this post?
-
I just released version v0.7.0 with the syntax highlighting to the marketplace, so it's now ready for use!
-
Over the weekend I released 0.8.0 with a new feature and a quite some fixes. It now has context aware autocomplete, across files. From the readme:
Go to where you would like to insert a reference to another rule, then type CTRL+space. This will you show you approppriate suggestions; for example when adding an item to requiresBuy in items, it will only show research rules. It will also work across files, making life easier.
This is what it looks like:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/context-aware-autocomplete.gif)
-
Sounds useful. Many thanks for your work!
-
Over the weekend I released 0.8.0 with a new feature and a quite some fixes. It now has context aware autocomplete, across files. From the readme:
(https://cdn.wallpapersafari.com/24/98/frgM4X.jpg)
-
Sounds useful. Many thanks for your work!
Image
Cheers for the support guys ;) I hope you try it out sometime, or if you already have, you find it useful!
In other news, there's now a discord channel courtesy of efrenespartano: https://discord.gg/2WjVTvJcbQ
-
this is by the far one of the most useful modding tools I've ever had. great work!!
p.s - that font is sleek. what's it called?
-
this is by the far one of the most useful modding tools I've ever had. great work!!
Thank you for the kind words, it is appreciated! If you have any suggestions about how I could improve it even more, let me know!
p.s - that font is sleek. what's it called?
If you mean the font in VSCode the demo videos, I use Courier New. It's a pretty old default Microsoft font.
-
Thank you for the kind words, it is appreciated! If you have any suggestions about how I could improve it even more, let me know!
I guess the ability to ignore some of the warnings would be great. Sometimes the ruleset works as intended but linker sees that as error. maybe adding #ignoreLinker to end of line to ignore certain properties for checking?
+ also highlight syntax for loop
for example word 'loop' and 'var' not highlighted on
loop var i 10;
debug_log i;
end;
-
I guess the ability to ignore some of the warnings would be great. Sometimes the ruleset works as intended but linker sees that as error. maybe adding #ignoreLinker to end of line to ignore certain properties for checking?
I'd prefer to fix them, as in that case the tool is reporting false positives. Can I check out your rulesets somewhere so I can see what's going wrong?
+ also highlight syntax for loop
for example word 'loop' and 'var' not highlighted on
loop var i 10;
debug_log i;
end;
I'll look into updating the syntax highlighting! I based the highlighting on a version of OXCE where loops weren't a thing yet ;)
-
I'd prefer to fix them, as in that case the tool is reporting false positives. Can I check out your rulesets somewhere so I can see what's going wrong?
these all are the false positives despite the code working as intended; linker will tell on attachment...
'cannons-conventional.rul' (STR_CANNON is pre-defined, not sure about STR_NONE)
"STR_NONE" does not exist (craftWeapons.clip) for STR_CANNON_UC
'STR_CANNON_UC' launcher 'STR_CANNON': item does not exist. This will cause a crash on loading OpenXcom!'terror-unit-weapons.rul' (bulletSprite 10 should be vanilla blaster launcher's bulletsprite)
"10" does not exist (items.bulletSprite) for STR_SECTOPOD_AUX_WEAPON
Hint: this needs a sprite in Projectiles with spriteID 350 (10 * 35)
(see documentation by hovering over "bulletSprite:")
'coverall.rul'
"MAN_0" does not exist (armors.spriteInv) for STR_COVERALL_UC
'awacs.rul' (on extrasprites INTICON.PCK is made of six sprites starting from 26;)
"26" (also "27", "28") does not exist (crafts.sprite) for STR_HAWKEYE (also STR_AWACS, STR_DARKSTAR)
Hint: this needs THREE extraSprites (see documentation by hovering over "sprite:"):
- One in INTICON.PCK with spriteID 26
- One in INTICON.PCK with spriteID 37 (26 + 11)
- And one in BASEBITS.PCK with spriteID 59 (26 + 33)
'mapScripts_CMPV.rul' (I'm not sure about this one, but I've tested the maps that was supposed to get segfault and they were okay, looks like groups can be arbitary (https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)#Map_Blocks))
'Group '4' does not exist in terrain for RICE_FARM_SCRIPT. This will cause a segmentation fault when loading the map!
+ ADDED
Units-Alien.zip contains alien deployment data, which was split into:
- data (where alienRank is stored)
- width, length, height
- civilians, alert (and background), briefing, markername, duration and despawnpenalty
since the only part where splitting is not allowed is data, files above works fine and without bug (afaik), however they're spewing out like, 40 fatal errors.
-
stuff...
Thanks for reporting all of these, I have fixed them and will push them to marketplace soon as 0.8.2. There was only one issue (below) that was not actually a false positive but the reported problem did not match with the reality.
'mapScripts_CMPV.rul' (I'm not sure about this one, but I've tested the maps that was supposed to get segfault and they were okay, looks like groups can be arbitary (https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)#Map_Blocks))
'Group '4' does not exist in terrain for RICE_FARM_SCRIPT. This will cause a segmentation fault when loading the map!
This one does not actually cause a segfault. It does happen with addLine, but for addBlock it just gets ignored. I have updated the error message in this case to reflect that. For addLine it will still report a segfault.
-
This is an awesome extension, thank you!
I have a suggestion re performance.
I loaded X-COM Files with this extension and it took, I think, over 15 minutes to load all rulesets, and my laptop fan was going bonkers during this entire time. X-COM Files has some massive rulesets, e.g. research_XCOMFILES.rul is over 24000 lines long, and the loading progress was stuck on it for the majority of the time it took to load.
Is there a way to speed up this loading process? E.g. cache it, and later diff and update incrementally? If I reopen VS Code I believe it will try to load everything anew.
-
Is there a way to speed up this loading process? E.g. cache it, and later diff and update incrementally? If I reopen VS Code I believe it will try to load everything anew.
Thanks for the kind words! There is already cache on a file basis, once a .rul file has been parsed, it should not have to be parsed again unless it changes (or the extension gets an update). However one change means that entire file needs to be parsed again, due to my current implementation.
I need to take a look at why it's taking exponentially long to deal with larger files like in XCF, I am not sure. But I would probably need to do a pretty big overhaul on the main parsing bit of the code. It's one of the first things I wrote for this extension, and knowing what I know now I would have done it differently. It's not the easiest bit for me to work in though.
I'll try to have a look when I can if there's any quick band-aid fixes I can apply, I'll let you know here if I do.
-
Thanks for the kind words! There is already cache on a file basis, once a .rul file has been parsed, it should not have to be parsed again unless it changes (or the extension gets an update). However one change means that entire file needs to be parsed again, due to my current implementation.
I need to take a look at why it's taking exponentially long to deal with larger files like in XCF, I am not sure. But I would probably need to do a pretty big overhaul on the main parsing bit of the code. It's one of the first things I wrote for this extension, and knowing what I know now I would have done it differently. It's not the easiest bit for me to work in though.
I'll try to have a look when I can if there's any quick band-aid fixes I can apply, I'll let you know here if I do.
Indeed, upon next open the rules were loaded in about 10 seconds. Awesome! I underestimated your implementation, sorry! ;)
I totally see how this can be a highly nontrivial issue. As the saying goes, there are three hard problems in software engineering: cache invalidation and off by one errors ;)
Nevertheless, I think that with fast reload without edits this works quite well already. For example, if I need to make edits to one of these humongous files, I could batch them and don't reopen VS Code until I am done, then reload. Also, it is an argument for splitting these files into smaller ones (I do hope OXCE supports reading rules of one type from multiple files).
I think your tool adoption would benefit from being more explicit about this caching. I think some people might give up before the 15 minutes for initial load are up. If the tool would somehow report how long it will take, and clarify it will be fast without edits, and reload only specific files upon edit, that would motivate people to give it a proper try.
-
Indeed, upon next open the rules were loaded in about 10 seconds. Awesome! I underestimated your implementation, sorry! ;)
I totally see how this can be a highly nontrivial issue. As the saying goes, there are three hard problems in software engineering: cache invalidation and off by one errors ;)
15 minutes, and then 10 seconds cached is still too long, obviously ;D I still hope to be able to do something about it sometime.
Nevertheless, I think that with fast reload without edits this works quite well already. For example, if I need to make edits to one of these humongous files, I could batch them and don't reopen VS Code until I am done, then reload.
Good to hear. You could try to see how long saving a single file works. The moment you save is the moment it parses the file again. Differential parsing could help in this matter, but I'm not sure how to approach that (yet). The yaml library I use offers no such functionalities, I'd have to find something generic, or set it up myself.
Also, it is an argument for splitting these files into smaller ones (I do hope OXCE supports reading rules of one type from multiple files).
Yes, that is very possible. You can basically split things anyway you like (as far as I'm aware), and that would certainly help the problem. It's a workaround and it obviously doesn't directly address the core problem, but it will help.
I think your tool adoption would benefit from being more explicit about this caching. I think some people might give up before the 15 minutes for initial load are up. If the tool would somehow report how long it will take, and clarify it will be fast without edits, and reload only specific files upon edit, that would motivate people to give it a proper try.
I think the larger problem is that not too many people are aware of it, and secondly that people are quite content just using notepad++ for their modding. That said, I could drop it in the README somewhere. For most mods that I've tried it with, it's not really a problem though. It's the really large ones (there's only XCF, FMP and X-Piratez that are like that I think?) that are causing the long load times. There is something in my parser that does not scale, clearly ;D
-
I just published 0.8.2 to vscode marketplace. Not a very exciting release with many new features, but a great number of fixes/tweaks.
On the roadmap currently:
- Ruleset spreadsheet editor (allows two-way editing of .rul files as spreadsheets either in vscode itself or using Excel)
- Y-script syntax validation
- Missing translation detection
- Missing files detection (sprites/sounds)
- other stuff/li]
This is the changelog for 0.8.2:
- Added missing vanilla armor sprites
- Fix sprite index checking with subX/subY for more sprite types (INTICON)
- Do not retrieve keys from aliases (they only need checking once), fixes problems
- Store logic data for vanilla assets (but don't check them), prevents false positives when checking them
- Improved message in case of mapblock that does not cause segfault but will just cause block to be ignored
- Fix autocomplete for line that is also start of an array entry
- Improve duplicate error message (use related information, #35 (https://github.com/pedroterzero/oxce-yaml-helper/issues/35))
- Fix relatedlogic fields overwriting existing typeLinks
- Added missing type link for alienMissions.waves[].ufo
- Added missing lt & le in y-script highlighting
- manufacture.requiredItems add crafts as valid type
- Added missing stringType (alienDeployments.objectiveFailed)
- Combine alienMissions data from multiple files
- Ufopaedia image no longer required for type 3, apparently
- Parse comments in extraSprites/extraSounds (so duplicates can be ignored)
- Enable subX/subY for BIGOBS and FLOOROBS
- Add hitMissSound vanilla ids
- No longer check ufopaedia.weapon (it's a translatable string)
- Properly handle refNodes in logic checkers may need to add in other positions)
- Reload Language/*.yml on changes, so translations get picked up #26 (https://github.com/pedroterzero/oxce-yaml-helper/issues/26)
-
Just to say that I've tried pedroterzero's tool recently and I'm completely sold to it just for parsing the rulesets and fixing bugs I had no idea they were there.
For large mods, this is a must have if just to periodically check your rulesets and increase mod stability. And it will save you a lot of pain and time lost from bugs not to easy to find.
-
Just to say that I've tried pedroterzero's tool recently and I'm completely sold to it just for parsing the rulesets and fixing bugs I had no idea they were there.
For large mods, this is a must have if just to periodically check your rulesets and increase mod stability. And it will save you a lot of pain and time lost from bugs not to easy to find.
Thank you! That is very high praise indeed coming from one of the most experienced modders out there. I appreciate it very much.
-
I have just released the v1.0.0 (first "stable") version of my extension has been released on the marketplace. It includes the new CSV/spreadsheet editor and many fixes/corrections. Many thanks to all the testers/users of this exntesion that helped me get it over the line. Visual studio code should auto update you to this latest version.
Here is a demo of the new spreadsheet editor:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/csv-editor.gif)
-
I am working with rulesets a lot and I am using this extension from the very beginning. This extension is ABSOLUTELY MUST HAVE for any modder, even if you are making a tiny weapon mod! It saves so much time debugging. With that csv editing, balancing things became a much more pleasurable experience. Not to mention y-script highlighting, it's just great!
I saw this extension evolving, features being implemented, and I am very happy for you @pedroterzero!