OpenXcom Forum
OpenXcom Forks => More Forks => Topic started by: Whispers on June 07, 2024, 03:23:35 pm
-
Hey,
That's right, I've started it.
https://github.com/ken-noland/OpenXcom
I’m excited to introduce you to my ongoing OpenXcom fork, which started as a fork of Brutal (BOXCE) but has now evolved into a distinct project with its own vision and goals. While this project won’t merge with OpenXcom Extended (OXCE), the aim is to open new doors for modders by providing a foundation that’s as flexible and accessible as possible. You can learn more about OXCE here (https://openxcom.org/forum/index.php?topic=5251.0) to see where this project diverges.
Project Goals
This project’s primary goal is to enable a level of scripting that hasn’t been possible before, using Lua—a scripting language widely recognized and commonly used in the modding community. Lua will allow modders to script almost everything in the game, from region behaviors and globe mechanics to detailed particle effects, providing an unprecedented level of customization. My hope is that Lua scripting will make OpenXcom modding accessible to a broader audience and allow for rapid prototyping and development without diving into the complexities of the underlying codebase.
As part of this overhaul, we are also upgrading the underlying systems to modern APIs by replacing SDL with Vulkan for graphics and OpenAL for audio. This update ensures that OpenXcom remains compatible with modern platforms and continues to be maintainable long-term.
Building Toward New Possibilities
With this new scripting system, there are some powerful modding possibilities within reach:
- Complete UI Overhauls: The scripting system allows modders to redefine the game’s UI, enabling complete overhauls if desired. Modders can customize surfaces, blocks, and interactive elements, creating tailored interfaces.
- Procedural Terrain Generation: Modders can define their own terrain elements and structures, potentially enabling procedural terrain for dynamic mission environments.
- Custom Attributes: Lua scripting supports defining custom attributes in any object, allowing you to add new stats or attributes without the need for complicated workarounds
- Dynamic Mission Types and Objectives: With scripting, modders could introduce custom mission types, adapting objectives based on player actions or randomized conditions.
- Custom AI Behaviors: Lua makes it possible to script custom AI tactics, from enemy behaviors to ally actions, offering more reactive and intelligent opponents.
- Environmental Effects and Weather Systems: Modders could add dynamic environmental conditions, such as weather or hazardous zones, that impact gameplay in real time.
- Campaign Progression and Branching Paths: The system could support dynamic campaigns with branching storylines, influenced by player choices and mission outcomes.
- Advanced Resource Management: Beyond custom attributes, modders could introduce complex resource systems, alternate economies, or even faction loyalty mechanics.
- Enhanced Mod Compatibility and Interactions: The Lua foundation allows for dynamic interaction between mods, creating new possibilities for layered, collaborative mod features.
Why Lua and Vulkan?
The choice to use Lua for scripting was driven by the desire to make OpenXcom modding as accessible and flexible as possible. Lua is a widely-used language with a low learning curve, making it easy for modders to jump in and create complex, game-changing modifications without needing to delve into C++ code. Its simplicity is paired with powerful functionality, allowing modders to define game behaviors, create custom attributes, and introduce unique gameplay mechanics. Lua also integrates smoothly with C++, making it an ideal choice for a project where scripts need to interact closely with the game’s underlying systems.
The switch from SDL to Vulkan was equally intentional. SDL has served the project well for years, but Vulkan offers greater control over graphics rendering and better performance on modern hardware. This control is critical in allowing OpenXcom to support advanced visual features and to run efficiently across platforms, including potential future extensions to Mac and Linux. Vulkan also supports multi-threading better than SDL, which is a huge advantage for performance as the game grows more complex. This shift paves the way for visual enhancements, optimized performance, and cross-platform support, ensuring OpenXcom is equipped to handle both the current goals and future expansions.
Runtime Type Generation and Serialization
A key feature in this project is the use of runtime type generation to handle serialization between the engine, JSON objects on disk, and Lua functions and tables. Using runtime type information, the system automatically generates code to serialize and deserialize data to and from the engine. This allows modders to define custom attributes directly in Lua, which can then be accessed within the engine or stored in JSON files for persistence. This approach reduces the need for extensive C++ modifications for each new attribute or entity, simplifying the integration of modder-defined data.
This setup relies on type reflection to dynamically retrieve type information and automate serialization code generation. Custom attributes or entities defined in Lua are accessible within the engine, with JSON handling save states and configurations without requiring a unique serializer for each type. By generating Lua and JSON serializers automatically, components—such as gameplay attributes, UI elements, or terrain features—are consistently accessible across scripting and storage.
To maintain backward compatibility with older mods, YAML serialization is handled through hand-coded serializers. This approach ensures that legacy mods developed with YAML remain functional without modification. The combination of runtime type generation, automatic Lua and JSON serialization, and manual YAML support provides a flexible system that accommodates both legacy and new modding capabilities.
Future Vision
Looking beyond the initial goals, this project opens up some ambitious possibilities for the future of OpenXcom modding. Here are a few of the ideas that could become feasible with the new foundation:
- Playing X-COM 1 and 2 in a Seamless Campaign: Imagine starting with soldiers from X-COM 1 and transitioning directly into X-COM 2, carrying over your personnel, resources, and accomplishments to fight new threats with the same team.
- Complex Geopolitical Dynamics: The new scripting system could allow for deeper political interactions, where countries and factions have evolving allegiances, conflicts, and responses to player actions. Modders could build complex systems of geopolitical intrigue, making each decision impact the broader world in meaningful ways.
- Fog of War: Enabling an ongoing fog of war across the global map would add a layer of mystery and tension to strategic decision-making, concealing alien bases, research installations, and potentially hostile factions until they’re scouted or discovered.
- Multiple Globes for Interplanetary Operations: This new foundation makes it feasible to support multiple globe views, allowing interplanetary missions where players can explore, engage, and defend across different planets, each with its own conditions and alien threats. An interplanetary scope would expand the tactical and strategic gameplay far beyond Earth.
With these possibilities, OpenXcom could evolve into a platform that accommodates far more complex, immersive, and expansive gameplay. For anyone interested in following or contributing to this vision, I’ve outlined the project roadmap (https://github.com/ken-noland/OpenXcom/wiki/Roadmap) on the project’s GitHub Wiki. While we’re not yet at a public release stage, nightly builds are planned as we make further progress.
Thank you to everyone following and supporting the project. This is still a work in progress, but I’m excited to see what new experiences this foundation will enable in the OpenXcom universe. Stay tuned for more updates as we continue expanding OpenXcom’s possibilities!
-
Update:
Right, there's been some big modifications to the original source. Maxmahem has graciously offered to help move towards a more Entity Component System compliant code base. The reason why is that we need handles to objects so we don't have to manage the object lifecycle as much on the Lua side. I've started working on the UI side, wrapping all the interface objects and converting those over to composable elements(components) to make things easier when translating to and from Lua.
Overall, this is shaping up to be a huge refactor.
I have a branch that I've been toying with where I take the Lua bindings and I redirect it to a json serializer. I open up a plain old TCP socket and I accept basic HTTP GET and POST commands which basically means I can now edit anything that is exposed to Lua through a web page. It's actually something I've done for a lot of projects I've worked on, and it greatly helps test and diagnose issues. I'll probably merge that in to the Lua branch soon, although it has very little to do with Lua.
Would love to get your thoughts on these changes. Feel free to leave feedback here, or on the repo, any time!
-
Yet another update.
I decided to use run time type reflection to make the Lua binding easier, and in the process, I created the entity inspector. Early work on the entity inspector had it running in the same process as the game and using wxWidgets to manage the UI elements. Basically, anything that is an entity can have all it's components and their properties viewed and possibly edited by the inspector. However, I ran into a significant snag. It turns out that SDL hogs all the input for the process and there is no way for SDL and wxWidgets to live happily side by side in the same process. So, I decided to ditch SDL and rewrite the audio/video/input systems.
I've got it to the point now where most of the UI is rendering using Vulkan, and audio is through OpenAL, but that can be swapped out later if needed. I have a lot of work to do on the video playback, but most of the surface level stuff is functional.
I've ripped out the file handling and options logic and replaced it with a virtual filesystem and new options class because, well, to be honest, it was bugging me having things stored as global variables.
Everything I've done so far has been compiled on Windows and Linux. I don't have a Mac, although I did spend some time to see if I could spin up a Mac VMWare instance. That does appear to be working, but I haven't had a chance to get it all set up with homebrew and whatever else is needed. I did ditch all build systems other than CMake, so I'm not going to maintain the visual studio solution files or makefiles or any of that anymore. This is not a reflection on those build systems, but CMake can target almost every major IDE and handle the prerequisite gathering and all of that stuff without having to try to locate the right libraries for your system and place them in the right folder.
So, what I thought would be a few weeks of "just wrap things in Lua" turned out to be a major rewrite on all the core systems. I even changed up the log system so I could, in theory, have an in game console outside of the entity inspector. I've probably broken a few things along the way, but the general direction is good and seeing the front end menu pop up after weeks of uncertainty, well, really gave me a huge boost of morale.
-
I've posted an update:
https://github.com/ken-noland/OpenXcom/wiki/Update-%E2%80%90-November-9th-2024
In short...
What started as a simple Lua integration for OpenXcom turned into a major overhaul as each step uncovered more work. Moving to an Entity Component System (ECS) seemed like the best way to manage object lifecycles and interactions between Lua and C++, but it meant converting the game’s architecture piece by piece. This led to developing an entity inspector, which then revealed limitations with SDL’s event handling, pushing a switch to Vulkan. With SDL out of the picture, it made sense to revisit other areas—Options, Filesystem, Mods—that had grown cluttered over years of patching. Each fix led to another realization, eventually bringing about the need for a centralized Engine class to handle all the stuff SDL used to do, but outside of the Game class.
Switching to Vulkan and the ECS framework means resources are now managed through handles. With new Options class, a cleaner filesystem, and a CMake-based build system, the project is more cross-platform ready. The end goal is still to provide a more open foundation for modders, allowing Lua to prototype features that could later be optimized in C++. It has definitely taken longer than planned, but is still progressing.
I'm currently working on the oxce-lua-vulkan branch. I'll try to post updates more regularly now that the core is almost done.
-
Also, I'm not sure BOXCE is the best place for this project. I mean, it really has nothing to do with Brutal at all, other than I forked from there.
I mean, if OXCE is "A younger brother of OpenXcom focused on modding support and QoL." then this would be like the step son from a previous marriage after OXCE has grown up :P
-
where should I move it then?
-
I have no idea to be honest. It doesn't quite fit with "WIP", as that's for mods AFAIK. It doesn't quite fit in "Build and Ports" either. Perhaps just in the root "OpenXcom Extended" board? I just worry that by being here, people will either expect Brutal to get this at some point, or that the branch will support Brutal in the future, which we don't really have plans for, and I've already been told that Ximli wants to keep things simple and not pull in branches that contain such drastic changes as this one introduces(unless it's proven solid and there's enough support for it)
-
Perhaps just in the root "OpenXcom Extended" board?
Moved.
EDIT: I've created a separate forum category for OpenXcom Forks and moved all forks out from the Modding category
I just worry that by being here, people will either expect Brutal to get this at some point, or that the branch will support Brutal in the future, which we don't really have plans for, and I've already been told that Ximli wants to keep things simple and not pull in branches that contain such drastic changes as this one introduces(unless it's proven solid and there's enough support for it)
I worry that by being here, people will expect OXCE to get this at some point.
-
Moved.
I worry that by being here, people will expect OXCE to get this at some point.
Yeah, I can completely understand that. Things have drastically changed since I first posted, so I'll edit the main post to reflect that this is a fork, and will remain a fork unless that decision changes in the future.
-
I've updated the main post and highlighted that this will not be going upstream. I'll keep posting updates here and leave the main post alone, but the old post felt like it needed to go.
-
Thanks for the update.
I have also deleted a few early comments, which were now out of context and irrelevant.
PS: also please update the typo in the first post: OXCE stands for OpenXcom Extended, not OpenXcom Community Edition
-
Procedural Terrain Generation: Modders can define their own terrain elements and structures, potentially enabling procedural terrain for dynamic mission environments.
How?
-
How?
So, the way Lua bindings are supposed to work is that each event in game is mapped to a corresponding event in Lua. Mods can then subscribe to those events, allowing them to alter or change things exposed through the event, but they can also modify or alter things that exist in the game itself, and that includes loading or creating assets and resources on-the-fly.
The way things are exposed is through Run Time Type Reflection.
So, if we are talking about just Battlescape Terrain, then you can see they are composed of Map Blocks. Each block could be created/loaded independently either through loading MCD and PCK files, or you could define them through Lua. Theoretically, because resources can be created and destroyed with Lua, then you can write your own handler when the terrain is generated which defines the map blocks per each terrain and that would allow you to create dynamic terrain, otherwise known as Procedural Terrain Generation.
By being able to intercept or override events in game and redirect them to scripts, and giving scripts the ability to create/load/unload resources on the fly, we hope to allow for much greater control for the mods.
-
That's a theory, but how as a modder (read - game designer) that would help me? There should be super smart logic for procedural generation, if we do not use map blocks to generate battlescape map. I can't imagine any modder develop such a model, no matter if it would be Lua, C++, Pascal or whatever.
The only way I can imagine mapbgeneration can be fundamentally improved is what was done for XCOM 2 map generation with plots and parcels. I imagine, some sort of map block relation could be potentially implemented. Say, currently setting up something like road with parking attached to it is not trivial - you have to make complicated mapscrit with check, remove and then add commands to change original road tile to connect it with parking. I imagine there could be a more handy map generation process, but I can't see how Lua can help here, sorry. Am I wrong?
-
That's a theory, but how as a modder (read - game designer) that would help me? There should be super smart logic for procedural generation, if we do not use map blocks to generate battlescape map. I can't imagine any modder develop such a model, no matter if it would be Lua, C++, Pascal or whatever.
The only way I can imagine mapbgeneration can be fundamentally improved is what was done for XCOM 2 map generation with plots and parcels. I imagine, some sort of map block relation could be potentially implemented. Say, currently setting up something like road with parking attached to it is not trivial - you have to make complicated mapscrit with check, remove and then add commands to change original road tile to connect it with parking. I imagine there could be a more handy map generation process, but I can't see how Lua can help here, sorry. Am I wrong?
Good point! I totally get that serious procedural generation is more than just plugging in Lua—it’s about having the right tools and logic in place. Right now, the main goal with Lua is to make the current map block system as flexible as possible for modders, letting them tweak terrain and inject their own elements easily. It’s about getting up and running with what we already have, exposing what we can, and all hopefully without reinventing the wheel(too much).
But looking ahead, Lua could actually help with more rule-based or conditional generation if we go in that direction. Think of things like:
- Rule-based Layouts: Lua could allow modders to define placement rules, so you could script things like “place a road block next to open terrain” or “cluster buildings together.” This would let us build up a library of reusable conditions, similar to the parcels in XCOM 2.
- Dynamic Block Connections: Lua could enable blocks that adapt to their neighbors, making it easier to set up things like connected roads or parking lots without extensive manual scripting. It’s a way to add some fluidity to the current system without requiring everything to be hard-coded.
- Scriptable Templates: Imagine being able to design mini-layouts (like a courtyard, plaza, or forest cluster) that the generator pulls in based on the surrounding blocks or mission type. Lua could make it easy to slot these in dynamically.
So, while Lua isn’t the magic key to procedural generation alone, it gives us a framework to build on. And if we get the tools and logic right, it wouldn’t matter if the scripting was in Lua, C++, or even Pascal—the goal is really to expand the possibilities for map creation, whatever the language happens to be.
There are some tools we could look at in the future that are engine agnostic. An old colleague of mine has a tool for Unreal(and a version for Unity) called Apparance(https://www.apparance.uk/). It's a rule based parameterized procedural generator. Perhaps I'll pull him in to this conversation.
-
I thought I'd give a bit more of an update now that I've returned from a short "break". Unfortunately, I've had to pay some bills these last few weeks so I wasn't able to make much progress, but while I was working on other things, I've been thinking about how to handle the Vulkan renderer moving forward.
When working on something like this, the main goal is just to get it to work and then iterate on it. Well, I managed to get it to work. Sortof...
It was mimicking the surface blitting from SDL with some fairly basic shaders. I even went so far as to make the texture coordinates as integers instead of floats, using a fixed sampler, and then I was able to pass in the same coordinates from before and it would render the text, windows, and so on. Palettes were handled by uniform buffers and the palette ID was passed in as part of the rendering pipeline. I even went so far as to fix the aspect ratio in the game surface to windows surface shader.
However, the structure of it all bothered me. It wasn't very flexible and it felt a little like I was cramming functions into objects just for one single thing. That's not good design!
So, I've started cutting it up into smaller and smaller bits. While I'm at it, I want to make it API agnostic as well. If you want to render using DirectX12, that is something someone else can handle, but it should be possible.
Why go through all this effort? Easy! In the process of creating the new renderer, I've created a handful of "managers". Under normal circumstances, I don't like calling things "Managers". The term itself is ubiquitous since technically everything you write in code manages something, but it actually makes things super easy to manage resources. I'll hold my disdain for calling things Managers for now, and I'll just tell myself it sounds better than calling it "System". However, I digress. So, with that out of the way, practically every resource gets it's own data type and manager. All the Windows, Buttons, Text Boxes, Images, Palettes, Fonts and more! When I get to Battlescape, I'll be repeating the same pattern.
For instance, the base game only had 3 fonts that I know of. There's the DOS font that you see at the startup of the game while Mod.cpp loads, then there's the two in-game fonts(Large and Small). Now, I've got a FontManager. You can register your own font, complete with multiple pages and unicode support if you need it. Using the Large or Small font is still there, but now they are explicitly used by their handles. Also, I've kept the "turn big font into small font if it doesn't fit onscreen" for now, but I'm seriously debating if I should keep it in. Anyway, I digress. With the FontManager, I can now tie that into the entity inspector and scripting to allow me to register and use my own font if I want to.
However, one thing that struck me while I was doing the Window Manager is that I can now allow modders to create their own custom frames. The old way of drawing the frames was to use FillRect with preset values. With Vulkan, that doesn't work so well. I mean, you CAN do it, but I didn't want to. Vulkan works best if you create resources and then upload them to the GPU once, and only once. Draing the frame using FillRect is possible, but since the frame itself is animated when popping up, I didn't want to have to push up images every frame when I could just pass up coordinates. I can just upload a texture, specify the 8 cardinal and diagonal directions, then render those at the window coordinates. Instead of hardcoding that 5x5 corners, and the 5x1 sides, and 1x5 top and bottom, I can just let the WindowManager create the frame and the WindowFrame can be driven by the data. Data, as you can probably guess, can be modified through mods.
Why this long-winded explanation? Well, it's because in the process of redoing all the Window stuff with the new Vulkan images, I realized that a dedicated modder could use this framework to simply replace the border texture by specifying a file and the coordinates, then they can have ornate frames around in-game windows. Cool, eh? (I know, not the "killer feature" for this branch, but I thought it was nifty)
I'm going to spend a little while longer on Vulkan, mainly because I still have a week or two of real work to pay the power bills, but also because I'm having a bit of fun with it. I really enjoy trying to make the interfaces as elegant and clear as possible, and being distracted by real work has given me the opportunity to really just slowly refine what I have into something much cooler!
-
Hi Whispers,
I've starred the repo, and just wanted to register my interest in attempting to build your fork on macOS.
-
Hi Whispers,
I've starred the repo, and just wanted to register my interest in attempting to build your fork on macOS.
Thanks!
I have a "hackintosh" that I've created that I use to check compatibility. Specifically, I check to make sure it works with homebrew. I don't spend as much time on it as I do Linux/Android/Windows though, so I do expect things to break there from time to time.
-
Occasional breakage is expected ;D
-
Just a quick update. I've finally managed to get most of the Vulkan stuff to a place I like. There's still some things I want to improve there, but I've got to make progress on the Lua side as well, so starting today I'm shifting focus to getting the Mod.cpp cut up and exposed through Lua.
The way things used to work has drastically changed. Before, you used to be able to write directly to surfaces, either by blitting or drawing directly on them. With Vulkan, you don't have that ability anymore. Everything is handled through primitives. When you create a primitive, a device local(GPU only) buffer is created and updated. The hardest primitive to make, by far, was the TextPrimitive(although an argument could be made for the line primitive since I did spend a week trying to find out why it was missing a single pixel).
Text wrapping, alignment, and placement is... complicated. Because of various localization needs, I ended up just using an established library called HarfBuzz for the text shaping, and yet another library called libunibreak for the word wrapping. The two combined should allow for languages like Hebrew and Arabic if the needs ever arises. Kerning is now a feature, which allows two character to adjust their overlap if that's something that is ever needed. However, something bothered me about the existing text rendering. Turns out, by using a special marker, you can change the font mid text, and fonts have multiple images associated with them. This messes up the render state which needs to track these changes and update the resource bindings. This meant I needed to implement Text Sections. While I was doing so, I decided that, well, since I have to support text sections, might as well see what else I can change about the text. That's when I decided to implement ANSI escape codes.
So, ANSI escape codes are a pain. I know. It would be nice if someone would just implement a CSS style processor that keeps track of a stack of states. However, the one really nice thing about ANSI escape codes is that I don't have to do that. I don't have to track a stack of states and manage all the details with the state changes. Instead, I can cut up the text into sections based on the escape code. If someone wants to write a stack based state management for text, feel free to do so. I had to make some choices as to where I wanted to spend my time. So, the cool thing is, now we have in-line text colors and styles. In a single line, you can have up to 16 colors. You can even change the background if you'd like(which will come in handy when it comes to text input selection). Cool, eh?
Here's the unit test running RGB colors through a single line of text:
(https://raw.githubusercontent.com/ken-noland/OpenXcom/refs/heads/oxce-lua-vulkan/src/Tests/Data/Data/Test/Graphics/Text/102_multi_color.png)
And the code:
https://github.com/ken-noland/OpenXcom/blob/bd779c0f38a2d2eae8e056119e998f669cfa5bc3/src/Tests/Engine/Graphics/Primitives/TestText.cpp#L475
So, next up is Mod.cpp. Sorry, but that file is WAY too large and needs to be cut down. I'll be doing that with the use of Factories. I'm probably not going to do the whole thing, but pick and choose some of the rules that would get me the biggest results. After Mod.cpp, then I've got the input system to handle. That one gets a little complicated since I do want to have both Android and Windows/Mac/Linux all go through the same input interface.