OXCE Lua

[Whispers]

Hey,

That's right, I've started it.

https://github.com/ken-noland/OpenXcom

How it works

I've modified Game and Mod so that they take in a ModFile class which contains a list of all the mods active and are in the correct loading order. Once Mod has finished loading, then I load up LuaMod which goes through each metadata.yml and checks to see if there is a tag for a lua file in there. The metadata should look like this:

Code: [Select]

#
# metadata.yml for Lua Test

name: "Lua_Test"
version: 1.0
description: "Verifying the callback mechanism works"
author: Whispers
master: xcom2
lua: bootstrap.lua
Once it sees the "lua: bootstrap.lua", then it knows this is a mod which uses the new Lua stuff. Once that is detected, it uses the path to the mod, then appends the lua filename, then loads the Lua file. A couple of things to note here is that I don't think this will work for zipped up files in its current state, but that may come later. In fact, I'll post a bug on the repo so I know I will catch it at some point. The other thing to note is that I've overriden the Lua print and redirected it to log, so anything you print, or any Lua errors, will get output to the game log.

Once loaded, it is executed once, and only once. It's up to the mod to register events.

Code: [Select]

function onLoad(doc)
print("the above 'doc' contains the entire save state converted from YAML to Lua Tables")))
end

game.on_load_game:register_callback(onLoad)
There will be more hooks, but I had to start somewhere. Once the game is loaded, then things become more interesting. "game.base" becomes available, which I'm currently working on. On the C++ side, I've been working on how to simplify the registration of events and callbacks, as well as how best to expose the data. I've come up with a mechanism which allows a programmer to register a container and the LuaApi does the rest, but it's still very early at the moment.

And that brings me to the current status. All the Lua stuff does at the moment is likely to change. For instance, 2 days ago I was designing the api to use getters and setters, but that's not necessary Lua. Want to change a soldiers name? Just set it in Lua and then you're done. Err, at least that's the idea anyway. Want to send a message to another mod? Sure, I'll get to that eventually. Cross mod communication is definitely on my agenda. My first priority is setting up the API and exposing as much as I can.

So, there are two types of callbacks, at the moment. Events and Accumulators. Events fire off just like you would expect them to. Things like onLoad and onSave or, when I get around to it, onTick are all events. They fire off with parameters that be be read by the Lua state. Most events fire off for every mod that registers a callback, but a handful of events can be consumed. When an event is consumed by the callback, it does not fire off for the next mod and it could also stop the default behavior in C++. Accumulators are a bit different. The first parameter for an accumulator callback is the result of the previous accumulator(or the default value). The accumulator must return a value which is passed to the next one or sent to the game.

The framework for Lua is mostly there. The Api has started to take shape. I'm very hopeful that this time around it will be an official release.

Big changes

This does introduce some changes that I feel are necessary.

1) YAML is no longer required in the binary dependencies. I would also like to get SDL removed, therefore removing all binary dependencies altogether, but... time...
2) I've dropped support for the old MSVC Project, Makefile and I will not be supporting any other build system other than CMake. Reason why is because in order to grab the YAML and (in the future) SDL stuff and build it, I use CMake for that. There's also a strong argument that if you are developing on something that is shared, then you should all be using the same process that is used to trigger the nightly builds.
3) I've squashed over 800 warnings from build the x64 version, but more are coming. I basically squash them as I'm building.
4) I've been slowly but steadily removing the "auto" from the variable declarations. In the process, I've found a TON of potential bugs and issues and fixed them as I went along. Far too many to list here.
5) I've exposed a getGame function callable from anywhere. I hate singletons, but in order to get the hooks where I needed them to be I need to get the Game*. Now, singletons are evil, but there is an alternative called Thread Local Storage. Not that it matters, due to how many other globals are floating around, but theoretically, you could fire off different Game objects on different threads. This keeps things nice and tidy for the future.
6) You need to have a C++20 compiler now

Backstory: A long time ago I started on working with integrating Lua into OpenXcom. At the time, I was basically told the change was not wanted by the authors of OXCE, so I abandoned it. This time around, *I* wanted the change so I could implement some custom logic without having to touch the source. It's taken me the last two weeks to get it up and running again, but this time it's far more powerful than before.

Next Steps

Continue implementing more events, exposing more data and fleshing out the API. Specifically, the next API I am working on is for UI.

I haven't even defined what a release looks like. I'd love to get your feedback and thoughts.

[Meridian]

This does introduce some changes that I feel are necessary.

1) YAML is no longer required in the binary dependencies. I would also like to get SDL removed, therefore removing all binary dependencies altogether, but... time...
2) I've dropped support for the old MSVC Project, Makefile and I will not be supporting any other build system other than CMake. Reason why is because in order to grab the YAML and (in the future) SDL stuff and build it, I use CMake for that. There's also a strong argument that if you are developing on something that is shared, then you should all be using the same process that is used to trigger the nightly builds.
3) I've squashed over 800 warnings from build the x64 version, but more are coming. I basically squash them as I'm building.
4) I've been slowly but steadily removing the "auto" from the variable declarations. I mean, seriously, that's just lazy. In the process, I've found a TON of potential bugs and issues and fixed them as I went along. Far too many to list here.

Just a few quick technical comments from OXCE:

1. We intentionally didn't target YAML-CPP repository, because we (both OXC and OXCE) had a lot of negative experience with that. Many YAML-CPP versions (for example v0.5.2 or v0.7.0) simply don't work with OpenXcom. It was broken for months, sometimes years long at times. (Situation with SDL is even more complicated. For some builds, e.g. Android, we've even forked SDL repo.)

2. FYI, OXCE doesn't use OXC github CI builds (yet). Developers (incl. OXC developers) use MSVC project. Our build servers use also other build methods beside cmake. Not to mention Android/iOS builds, which use yet more methods.

4. Not sure why you talk about laziness. "Auto" was added intentionally, before everything was without auto, now auto is used where we think it's appropriate. I've also made some cleanups myself (e.g. 9a90fab4372841b387319900993d0267251db18b) and right now both Yankes and I are happy with the current state. If you have a different preference, it's fine, but calling us lazy is a bit rude. If you've found any potential bugs, please report them.

[Whispers]

1. We intentionally didn't target YAML-CPP repository, because we (both OXC and OXCE) had a lot of negative experience with that. Many YAML-CPP versions (for example v0.5.2 or v0.7.0) simply don't work with OpenXcom. It was broken for months, sometimes years long at times. (Situation with SDL is even more complicated. For some builds, e.g. Android, we've even forked SDL repo.)

That's the beauty of FetchContent within CMake. We can specify the exact commit or tag that needs to be synced. I wanted to go this route as it cleans up a lot of potential compilation issues.

2. FYI, OXCE doesn't use OXC github CI builds (yet). Developers (incl. OXC developers) use MSVC project. Our build servers use also other build methods beside cmake. Not to mention Android/iOS builds, which use yet more methods.

That's fine. I've given up on this ever going upstream. If I am going to need to maintain a forked repo for the long haul, then I need to set things up in a way that makes it easier for me. OXC and OXCE both have stated missions of keeping the gameplay as core to the original as possible(with quite a few exceptions), and I completely understand and agree. My goal is different. I would like to see all sorts of crazy modifications that don't necessarily align with the goals or limitations of the current systems.

Packaging for Android should actually be easier now, as CMake as native support for that, and it has integration in Visual Studio.

It should be noted that CMake isn't REALLY a build system, but a build system generator. It can output VS Projects, Makefiles and more. I did go through the effort of getting the FetchContent stuff to work with VS Projects, but it took an entire afternoon of hacking together batch files and other means that just didn't make the code any better.

4. Not sure why you talk about laziness. "Auto" was added intentionally, before everything was without auto, now auto is used where we think it's appropriate. I've also made some cleanups myself (e.g. 9a90fab4372841b387319900993d0267251db18b) and right now both Yankes and I are happy with the current state. If you have a different preference, it's fine, but calling us lazy is a bit rude. If you've found any potential bugs, please report them.

Oh my. I completely apologize! I've been running on very little sleep and that slipped out while I was typing up all the notes. My sincere apologies! I've removed it from the post.

Personally, I really dislike the auto keyword, and there was one situation where the code was dereferencing a pointer to a vector and assigning it to auto, which copies the vector. Then, immediately underneath, it was iterating through the elements and deleting them, then clearing the newly created array, meanwhile leaving the parent array full of pointers that led to deleted memory. That example comes to mind, but I'd have to dig to find it again. I've literally changed a huge chunk of the codebase by now, so finding it will take me more than a cursory glance at the commits.

Ah man, now I feel bad about the "lazy" comment. This isn't my first time on these forums, but still, it's been long enough that this is basically a first meeting, and I completely screwed it up. Sorry about that Meridian.

[Yankes]

And where was this memory leak? I recall couple of places where someone copy vector but in any of this places was no memory leak as objects from vector lived lot longer. And even more, original vector should not be changed in that case too.

[Meridian]

Ah man, now I feel bad about the "lazy" comment. This isn't my first time on these forums, but still, it's been long enough that this is basically a first meeting, and I completely screwed it up. Sorry about that Meridian.

No worries, all is well.

Looking forward to seeing how this project evolves.

[Whispers]

And where was this memory leak? I recall couple of places where someone copy vector but in any of this places was no memory leak as objects from vector lived lot longer. And even more, original vector should not be changed in that case too.

My bad, upon further review it appears as though the array copy was erroneous, but it wasn't deleting from the copied array. It was roughly line 2511 of BattlescapeGame.cpp. Mine looks a lot different then yours will, but something like this:

Code: [Select]

bu->removeSpecialWeapons(_save);
std::vector<BattleItem*> invCopy = bu->getInventory();
for (BattleItem* bi : invCopy)
{
_save->removeItem(bi);
}

bu->setTile(nullptr, _save);
_save->clearUnitSelection(bu);
delete bu;

[Yankes]

No copy there was erroneous, did this variable was named `invCopy`, right? this copy is critical for code correctness.

[Whispers]

Update:

Right, there's been some big modifications to the original source. Maxmahem has graciously offered to help move towards a more Entity Component System compliant code base. The reason why is that we need handles to objects so we don't have to manage the object lifecycle as much on the Lua side. I've started working on the UI side, wrapping all the interface objects and converting those over to composable elements(components) to make things easier when translating to and from Lua.

Overall, this is shaping up to be a huge refactor.

I have a branch that I've been toying with where I take the Lua bindings and I redirect it to a json serializer. I open up a plain old TCP socket and I accept basic HTTP GET and POST commands which basically means I can now edit anything that is exposed to Lua through a web page. It's actually something I've done for a lot of projects I've worked on, and it greatly helps test and diagnose issues. I'll probably merge that in to the Lua branch soon, although it has very little to do with Lua.

Would love to get your thoughts on these changes. Feel free to leave feedback here, or on the repo, any time!