OpenXcom Forum

Modding => Tools => Topic started by: pedroterzero on December 14, 2020, 03:17:24 pm

Title: VSCode OpenXcom Modding Tools
Post by: pedroterzero on December 14, 2020, 03:17:24 pm
Hi there!

I recently started work on a visual studio code extension that works in addition to the existing excellent Ruleset Validator (https://openxcom.org/forum/index.php/topic,6552.0.html) by SupSuper.

My aim is to make a bit more of modding IDE experience for OpenXcom (Extended) rulesets by adding some IDE like features to the ruleset editing itself. The goal is to increase the quality of life during modding even more by solving a few of the problems that I saw myself while doing some work on mods. It should also raise the general quality of the rulesets themselves by pointing out issues that are easily overlooked if they're only inspected manually.

Some of the more interesting features are:


Some features I hope to add in the future:


Where to get it and more information
The extension and instructions how to install it, and a lot more information can be found on the visual studio marketplace: https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper (https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper)

Some demos of what this looks like:

Jump to definition
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/go-to-definition.gif)

Syntax highlighting/colouring
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/syntax-highlighting.png)

Reference checking:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/reference-checking.gif)

Documentation hover:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/documentation-hover.gif)
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on December 14, 2020, 04:28:43 pm
Making a second post so I can link (sorry :()
Title: Re: VSCode OpenXcom Modding Tools
Post by: Finnik on December 14, 2020, 05:18:23 pm
This a great tool I was using for a while (as I was honored to be an alpha tester), it is invaluable for making big mods, so I refuse to do modding without it, and this thing is in my must-have list (together with ruleset tools and map editor), so I recommend to use it to everyone!
Title: Re: VSCode OpenXcom Modding Tools
Post by: Yankes on December 14, 2020, 05:47:37 pm
Making a second post so I can link (sorry :()
how could you! :>

y-scripts syntax highlighting
I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.
Title: Re: VSCode OpenXcom Modding Tools
Post by: Finnik on December 14, 2020, 08:35:57 pm
I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.

I think it would be great!
And also auto-suggest of it, like how IDE does, I'm casting `itemRule.get` and it suggests what can I get based on what hook I am scripting. Highlighting variables, tag names, remembering variables to warn it is not defined in the beginning of the script or there is no such tag.
I am sure that would really help to populate y-scripting to people if there would be a good helper around.
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on December 14, 2020, 09:00:48 pm
This a great tool I was using for a while (as I was honored to be an alpha tester), it is invaluable for making big mods, so I refuse to do modding without it, and this thing is in my must-have list (together with ruleset tools and map editor), so I recommend to use it to everyone!

Thank you for your kind words!

I would be very grateful, how much syntax could you highlighting? Line know function names? I make in my script code option to dump all know symbols.
Right now format is not very parser friendly, but I could alter it to return some format that could be used by human and machines.

For now it would probably mostly about colouring the code in VSCode. I was hoping to base it an another language and add some specific stuff. What language would be a good pick for this?

I was hoping to use this as a basis: https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide -- VSCode uses TextMate internally. I have never done it before so it'll take a bit of experimentation ;)
It would help to have all symbols, that way I can possibly even add some semantic highlighting later, if the syntax highlighting works.

I think it would be great!
And also auto-suggest of it, like how IDE does, I'm casting `itemRule.get` and it suggests what can I get based on what hook I am scripting. Highlighting variables, tag names, remembering variables to warn it is not defined in the beginning of the script or there is no such tag.
I am sure that would really help to populate y-scripting to people if there would be a good helper around.

This is a more advanced topic I think. This would require me to actually parse the code probably, but we'll see. First I'll start with the colouring at some point, that's probably complicated enough ;)
Title: Re: VSCode OpenXcom Modding Tools
Post by: Yankes on December 14, 2020, 09:19:37 pm
y-scripts are most close to assembler, one operation per line (more accurately per `;` as it could be in one line if split by `;`). There are some exceptions like `if` blocks that can combine multiple comparison and create scope.
Variable for now can be defined only on top of script, `var TYPE NAME VALUE;` in far future I plan allow variables defined in each scope.
Each line look like
Code: [Select]
TypeAName.funcName valueOfTypeA arg1 arg2;or
Code: [Select]
valueOfTypeA.funcName  arg1 arg2;Both mean same.
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on December 14, 2020, 09:57:45 pm
y-scripts are most close to assembler, one operation per line (more accurately per `;` as it could be in one line if split by `;`). There are some exceptions like `if` blocks that can combine multiple comparison and create scope.

Thanks. That helps. I'll try to get it in a test version sometime (I am currently working on some other features, improved logic checking and context aware autocomplete). If I get anywhere I'll let you know!
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on January 25, 2021, 03:25:51 pm
I just released v0.6.0 on marketplace (https://marketplace.visualstudio.com/items?itemName=pedroterzero.oxc-yaml-helper). This is quite a big release, although it maybe doesn't seem like it.

Most importantly I added a mechanism to do specific logic checking on rulesets. This allows for checking and preventing some common (and uncommon) crashes and segmentation faults that might occur. Some occur when starting OXCE, others happen only when the specific rule triggers.

Having them show up as problems in vscode should be big time-saver and QA improvement. Many thanks to Filip H and Finnik for providing me with input on these. I hope to add more detection as I become aware of the possible problems that could occur. Feel free to let me know when there's something the extension isn't picking up.

I am currently working on finishing a few other features too, namely scripting highlighting, context aware autocomplete (only completes the correct types, and works across files) and scripting parsing (so you can tell if you've made mistakes in the script in vscode itself).

A preview of the highlighting in action:
(https://i.postimg.cc/JndpwyJp/highlighting.png)
Title: Re: VSCode OpenXcom Modding Tools
Post by: Yankes on January 25, 2021, 07:15:27 pm
Praise colors!


(https://steamuserimages-a.akamaihd.net/ugc/440605165249728813/4A0EAB3A009688E19D70680256247FB17F708B20/)
Title: Re: VSCode OpenXcom Modding Tools
Post by: Finnik on February 23, 2021, 05:06:03 pm
Can we pin this post?
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on February 28, 2021, 12:25:54 am
I just released version v0.7.0 with the syntax highlighting to the marketplace, so it's now ready for use!
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on April 26, 2021, 11:53:27 am
Over the weekend I released 0.8.0 with a new feature and a quite some fixes. It now has context aware autocomplete, across files. From the readme:

Quote
Go to where you would like to insert a reference to another rule, then type CTRL+space. This will you show you approppriate suggestions; for example when adding an item to requiresBuy in items, it will only show research rules. It will also work across files, making life easier.

This is what it looks like:
(https://raw.githubusercontent.com/pedroterzero/oxce-yaml-helper/main/docs/context-aware-autocomplete.gif)

Title: Re: VSCode OpenXcom Modding Tools
Post by: Solarius Scorch on April 26, 2021, 04:42:42 pm
Sounds useful. Many thanks for your work!
Title: Re: VSCode OpenXcom Modding Tools
Post by: efrenespartano on April 26, 2021, 04:48:48 pm
Over the weekend I released 0.8.0 with a new feature and a quite some fixes. It now has context aware autocomplete, across files. From the readme:


(https://cdn.wallpapersafari.com/24/98/frgM4X.jpg)
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on April 26, 2021, 11:50:01 pm
Sounds useful. Many thanks for your work!
Image

Cheers for the support guys ;) I hope you try it out sometime, or if you already have, you find it useful!

In other news, there's now a discord channel courtesy of efrenespartano: https://discord.gg/2WjVTvJcbQ
Title: Re: VSCode OpenXcom Modding Tools
Post by: greenscarf on April 27, 2021, 03:46:56 am
this is by the far one of the most useful modding tools I've ever had. great work!!


p.s - that font is sleek. what's it called?
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on April 28, 2021, 02:49:03 pm
this is by the far one of the most useful modding tools I've ever had. great work!!

Thank you for the kind words, it is appreciated! If you have any suggestions about how I could improve it even more, let me know!

p.s - that font is sleek. what's it called?

If you mean the font in VSCode the demo videos, I use Courier New. It's a pretty old default Microsoft font.
Title: Re: VSCode OpenXcom Modding Tools
Post by: greenscarf on April 29, 2021, 01:21:10 am
Thank you for the kind words, it is appreciated! If you have any suggestions about how I could improve it even more, let me know!


I guess the ability to ignore some of the warnings would be great. Sometimes the ruleset works as intended but linker sees that as error. maybe adding #ignoreLinker  to end of line to ignore certain properties for checking?


+ also highlight syntax for loop
for example word 'loop' and 'var' not highlighted on
Code: [Select]
loop var i 10;
    debug_log i;
end;
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on April 29, 2021, 10:50:54 am
I guess the ability to ignore some of the warnings would be great. Sometimes the ruleset works as intended but linker sees that as error. maybe adding #ignoreLinker  to end of line to ignore certain properties for checking?

I'd prefer to fix them, as in that case the tool is reporting false positives. Can I check out your rulesets somewhere so I can see what's going wrong?

+ also highlight syntax for loop
for example word 'loop' and 'var' not highlighted on
Code: [Select]
loop var i 10;
    debug_log i;
end;
I'll look into updating the syntax highlighting! I based the highlighting on a version of OXCE where loops weren't a thing yet ;)
Title: Re: VSCode OpenXcom Modding Tools
Post by: greenscarf on April 29, 2021, 12:31:52 pm
I'd prefer to fix them, as in that case the tool is reporting false positives. Can I check out your rulesets somewhere so I can see what's going wrong?


these all are the false positives despite the code working as intended; linker will tell on attachment...
'cannons-conventional.rul' (STR_CANNON is pre-defined, not sure about STR_NONE)
Code: [Select]
"STR_NONE" does not exist (craftWeapons.clip) for STR_CANNON_UC
'STR_CANNON_UC' launcher 'STR_CANNON': item does not exist. This will cause a crash on loading OpenXcom!
'terror-unit-weapons.rul' (bulletSprite 10 should be vanilla blaster launcher's bulletsprite)
Code: [Select]
"10" does not exist (items.bulletSprite) for STR_SECTOPOD_AUX_WEAPON
Hint: this needs a sprite in Projectiles with spriteID 350 (10 * 35)
        (see documentation by hovering over "bulletSprite:")

'coverall.rul'
Code: [Select]
"MAN_0" does not exist (armors.spriteInv) for STR_COVERALL_UC


'awacs.rul' (on extrasprites INTICON.PCK is made of six sprites starting from 26;)
Code: [Select]
"26" (also "27", "28") does not exist (crafts.sprite) for STR_HAWKEYE (also STR_AWACS, STR_DARKSTAR)
Hint: this needs THREE extraSprites (see documentation by hovering over "sprite:"):
        - One in INTICON.PCK with spriteID 26
        - One in INTICON.PCK with spriteID 37 (26 + 11)
        - And one in BASEBITS.PCK with spriteID 59 (26 + 33)

'mapScripts_CMPV.rul' (I'm not sure about this one, but I've tested the maps that was supposed to get segfault and they were okay, looks like groups can be arbitary (https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)#Map_Blocks))
Code: [Select]
'Group '4' does not exist in terrain for RICE_FARM_SCRIPT. This will cause a segmentation fault when loading the map!

+ ADDED


Units-Alien.zip contains alien deployment data, which was split into:
since the only part where splitting is not allowed is data, files above works fine and without bug (afaik), however they're spewing out like, 40 fatal errors.
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on May 02, 2021, 03:16:30 pm
stuff...

Thanks for reporting all of these, I have fixed them and will push them to marketplace soon as 0.8.2. There was only one issue (below) that was not actually a false positive but the reported problem did not match with the reality.

'mapScripts_CMPV.rul' (I'm not sure about this one, but I've tested the maps that was supposed to get segfault and they were okay, looks like groups can be arbitary (https://www.ufopaedia.org/index.php/Ruleset_Reference_Nightly_(OpenXcom)#Map_Blocks))
Code: [Select]
'Group '4' does not exist in terrain for RICE_FARM_SCRIPT. This will cause a segmentation fault when loading the map!

This one does not actually cause a segfault. It does happen with addLine, but for addBlock it just gets ignored. I have updated the error message in this case to reflect that. For addLine it will still report a segfault.
Title: Re: VSCode OpenXcom Modding Tools
Post by: spawarotti on May 16, 2021, 11:45:18 pm
This is an awesome extension, thank you!

I have a suggestion re performance.

I loaded X-COM Files with this extension and it took, I think, over 15 minutes to load all rulesets, and my laptop fan was going bonkers during this entire time. X-COM Files has some massive rulesets, e.g. research_XCOMFILES.rul is over 24000 lines long, and the loading progress was stuck on it for the majority of the time it took to load.

Is there a way to speed up this loading process? E.g. cache it, and later diff and update incrementally? If I reopen VS Code I believe it will try to load everything anew.
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on May 17, 2021, 11:58:31 am
Is there a way to speed up this loading process? E.g. cache it, and later diff and update incrementally? If I reopen VS Code I believe it will try to load everything anew.

Thanks for the kind words! There is already cache on a file basis, once a .rul file has been parsed, it should not have to be parsed again unless it changes (or the extension gets an update). However one change means that entire file needs to be parsed again, due to my current implementation.

I need to take a look at why it's taking exponentially long to deal with larger files like in XCF, I am not sure. But I would probably need to do a pretty big overhaul on the main parsing bit of the code. It's one of the first things I wrote for this extension, and knowing what I know now I would have done it differently. It's not the easiest bit for me to work in though.

I'll try to have a look when I can if there's any quick band-aid fixes I can apply, I'll let you know here if I do.

Title: Re: VSCode OpenXcom Modding Tools
Post by: spawarotti on May 18, 2021, 02:19:36 am
Thanks for the kind words! There is already cache on a file basis, once a .rul file has been parsed, it should not have to be parsed again unless it changes (or the extension gets an update). However one change means that entire file needs to be parsed again, due to my current implementation.

I need to take a look at why it's taking exponentially long to deal with larger files like in XCF, I am not sure. But I would probably need to do a pretty big overhaul on the main parsing bit of the code. It's one of the first things I wrote for this extension, and knowing what I know now I would have done it differently. It's not the easiest bit for me to work in though.

I'll try to have a look when I can if there's any quick band-aid fixes I can apply, I'll let you know here if I do.

Indeed, upon next open the rules were loaded in about 10 seconds. Awesome! I underestimated your implementation, sorry! ;)

I totally see how this can be a highly nontrivial issue. As the saying goes, there are three hard problems in software engineering: cache invalidation and off by one errors ;)

Nevertheless, I think that with fast reload without edits this works quite well already. For example, if I need to make edits to one of these humongous files, I could batch them and don't reopen VS Code until I am done, then reload. Also, it is an argument for splitting these files into smaller ones (I do hope OXCE supports reading rules of one type from multiple files).

I think your tool adoption would benefit from being more explicit about this caching. I think some people might give up before the 15 minutes for initial load are up. If the tool would somehow report how long it will take, and clarify it will be fast without edits, and reload only specific files upon edit, that would motivate people to give it a proper try.
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on May 18, 2021, 10:45:49 am
Indeed, upon next open the rules were loaded in about 10 seconds. Awesome! I underestimated your implementation, sorry! ;)

I totally see how this can be a highly nontrivial issue. As the saying goes, there are three hard problems in software engineering: cache invalidation and off by one errors ;)

15 minutes, and then 10 seconds cached is still too long, obviously ;D I still hope to be able to do something about it sometime.

Nevertheless, I think that with fast reload without edits this works quite well already. For example, if I need to make edits to one of these humongous files, I could batch them and don't reopen VS Code until I am done, then reload.

Good to hear. You could try to see how long saving a single file works. The moment you save is the moment it parses the file again. Differential parsing could help in this matter, but I'm not sure how to approach that (yet). The yaml library I use offers no such functionalities, I'd have to find something generic, or set it up myself.

Also, it is an argument for splitting these files into smaller ones (I do hope OXCE supports reading rules of one type from multiple files).

Yes, that is very possible. You can basically split things anyway you like (as far as I'm aware), and that would certainly help the problem. It's a workaround and it obviously doesn't directly address the core problem, but it will help.

I think your tool adoption would benefit from being more explicit about this caching. I think some people might give up before the 15 minutes for initial load are up. If the tool would somehow report how long it will take, and clarify it will be fast without edits, and reload only specific files upon edit, that would motivate people to give it a proper try.

I think the larger problem is that not too many people are aware of it, and secondly that people are quite content just using notepad++ for their modding. That said, I could drop it in the README somewhere. For most mods that I've tried it with, it's not really a problem though. It's the really large ones (there's only XCF, FMP and X-Piratez that are like that I think?) that are causing the long load times. There is something in my parser that does not scale, clearly  ;D
Title: Re: VSCode OpenXcom Modding Tools
Post by: pedroterzero on June 04, 2021, 03:10:25 pm
I just published 0.8.2 to vscode marketplace. Not a very exciting release with many new features, but a great number of fixes/tweaks.

On the roadmap currently:
This is the changelog for 0.8.2: