Messages

16

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 18, 2018, 08:43:29 pm »

So far all events I have planned are global events. I can see circumstances where you want to "only run this function on this particular instance of an object" , but that would be too intrusive for a first pass API. I'd have to go in and modify every spot where an instance of an object is created and either do multi-inheritance to a common base class which holds the necessary functions to allow binding functions to that instance, or I'd have to do some other method of allowing local instance callbacks which would touch more files than I would like to at the moment.

I think for the first version I will only have global events. Things like xcom.geoscape.register5SecCallback, xcom.battlescape.registerTurnBegin and even xcom.geoscape.registerUpdateCallback for those needing frame to frame updates. For other things, like xcom.soldier.get(soldier).registerDamageCallback(), I don't think the initial API will handle it like that. Instead, you could have something like xcom.soldier.registerDamageCallback() which takes a function that has a parameter for which soldier was damaged. It covers the same use case, but perhaps not as cleanly because you'd get that callback for EVERY soldier that receives damage instead of one specific soldier.

has<x>Callback and clear<x>Callback appear to be okay, so I'll move forward with that idea for now.

I'm still toying with ideas on how to architect the API that is less intrusive on the engine, yet allows controlling as much of the underlying data as possible.

17

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 18, 2018, 04:49:49 pm »

and Yeah, I know you were just giving an example, just pointing out that in this instance, its already there.

The point was not to showcase new features, but how I go about building the API functions.

18

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 18, 2018, 03:18:56 pm »

Ugh, I'm not really happy with the "register callbacks" mechanism. It lacks a certain grace that I usually like my APIs to have, but I simply can't think of a better way to deal with it.

Scenario:

As a mod developer, I want a show, every day in game time, my current funds

Solution:

Using the register callback mechanism, you can specify a lua function to be called. Something like this:

Code: [Select]

function on1Hour(gametime)
  print("Your current funds are " .. tostring(xcom.game.getFunds()))
end

xcom.geoscape.registerOn1HourCallback(on1Hour)

Now, every hour, on the hour, the function will be called. If you try to register more than one function, then it will simply overwrite whatever callback you had previously. Perhaps I should return the previous function? Maybe set it up so there is a has<x>Callback and clear<X>Callback? Hmmm, I'm not sure. I need to get this right because the callbacks are the key to the whole API.

19

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 18, 2018, 12:22:00 am »

Quick question to @Yankes and @Meridian.

What are your thoughts on std::shared_ptr and std::unique_ptr, etc?

Reason I ask is because it would make a large portion of the Lua code a lot easier. Instead of having to check for invalid inputs, I can guarantee a pointer will never be null simply by increasing a reference count.

It's not necessary, but thought I would ask.

20

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 11:06:07 pm »

`std::tuple` for rescue ;P

Where's my hammer? I've got to knock some sense into...

But for more serious answer, pair or tuple are good as implementation detail, not as public interface. Adding name give greater meaning to code that some random pack of values.

Oh, okay, I'll put my hammer away now.

21

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 10:12:46 pm »

One solution is create two contexts, if Lua job is only glue code then it will work fine, but if you want put lot of custom state in it then it will not work.

Contexts store all the globals for a lua script, and I imagine there will be a lot of people who want to access their globals of the ordinary script execution.

Instead, this means the concept of using the registerHook is more applicable. When the code executes a registered hook, it simply sets a flag in ModLuaScript as to what is calling the hook, then when the script calls into a C function, its up to the bound C function to get the current ModLuaScript to check the calling context and can then selectively throw an error. I'm not a fan of doing that, but it fills the requirement.

Another is add two different bindings for same type, it will be `BattleUnit` and `BattleUnitConst`, depending on details it could end up as big hack.

Eeeeks... My eyes, my eyes!

All refactors are welcomed, if you refactor half of code base and it simplify and improve code quality and then it fine, I only do not want brutal force solution that make code ugly (like function with 20 arguments, 100 copies of same code, e.t.c.). Change of `FileModInfo`  is good example because I can throw away Lua changes and still include it to OXCE.

Heh, so you're snooping on my changes already

Yeah, not a fan of using std::pair the way it was being used, so I threw that in there quickly because I needed to get the path to the mod, and refactoring felt better than hacking in another std::pair

22

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 09:22:18 pm »

Overall it could be simplify to two global states, one is allowed to modify thins other is not.

Unfortunately it's not that easy. Once a Lua context is created, the API is injected into the context. Once the API is injected, I can't easily turn functions on or off depending on the context that they are run in, unless I block out functions based on the calling event.

If your changes are similarly localized and do not leak every other place then is fine.

Yeah, I go one step further. I have to insert source in events in the logic object that, well, triggers the event, but otherwise all functions and script bindings are done in completely separate source files and only access public member functions of existing objects. There's never any concern about "leakage" since I never allocate anything.

I did have to tweak how mods are loaded in order to get the path of the mod that is trying to execute the script, but otherwise it's just a matter of putting in hooks and creating get/set functions

And after year you get bored and stop updating it or opposite I get bored and stop contributing to OXCE. In both cases will stay big blob of code that nobody want to touch. Right now I not interested in changing script engine or maintain another one but if you fulify my condition then I will accept it and take responsibility of maintaining it further.

Heh, not likely to happen, but I get your point. Being honest, I'm a bit disappointed in myself for not finding OpenXcom earlier. I could have done this years ago

23

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 07:40:46 pm »

Updated OP with more updates

24

OpenXcom Extended / Re: OpenXcom Extended (OXCE) main thread

« on: September 17, 2018, 07:26:32 pm »

Nice one!

25

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 02:33:31 pm »

Hmm, trying to think of a way to do terrain generated pressure plates. Something that is only executed once a soldier is standing on it. That could be another mod that I write to showcase the API.

26

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 17, 2018, 01:35:28 pm »

but Lua shroud obey game objects life cycle, if some object right now is created by function A then Lua should use this function too to create object. Same with deleting. If something is not changeable by game engine scripts usually should not change this too, this is important in case when some game logic really on that value is not changed.

Best if you could control per script call, some scripts could be allowed to change unit in any way other can only read.

Hmm, not really easily done with the way I have started implementing the API. I do have a container class I quickly wrote which maps user objects to their respective class pointers, so maybe in that container class I could write some access rights management, but there will be HUNDREDS of hooks. On<Whatever Event I Put A Hook For> would then set the access rights to object classes, but that seams extreme. Can you point to a situation where one thing can be read, but not written during a single event, yet can be read/written on other events?

You can compare it with current scripts, if your changes will be deleting my function parameter and replacing by your then its 100% fine. But if you need change every game logic class then is not fine.

The hooks will need to be implemented in the logic classes so I do imagine it touching most logic classes

Whole point is that I'm not interesting in maintaining two script engines in OXCE (and probably Meridian too) this is way I set high-bar that new script engine is objective better in every aspect compare to my engine. Otherwise I will stick to my creation This mean you will need do more work but I would be unwise to give up couple of years of work for free :>

I get that. I'm actually saying I will maintain the Lua stuff and you could continue to maintain the existing script stuff.

Instead of wiki I would be more interested in WiP commits that show how it will work.

Problem is that I'm not allowed to touch a source code editor over the weekends. My wife is very strict about that

Writing the wiki also helps me clear my thoughts. It helps make sure what I'm working on will actually work.

I'll push up something later today to show progress.

Figured I'd throw my two cents in, because I've always been curious about integrating Lua scripting (I've played with it in other games), but have no idea how. Usually when people talk about scripting they think about modders doing anything, limitless possibilities! I don't think that's the point. OpenXcom is already open-source, the possibilities are already limitless, so clearly it's not that simple. At the end of the day, modders are modders and programmers are programmers and I don't think scripting is what will convert them. But in my eyes it has the potential to help solve one of OpenXcom's biggest problems: maintainability. Now that vanilla is basically done, all that's left is adding modding features, forever and ever...

Everytime a mod needs a feature, a modder has to convince a programmer to add that feature, and that feature is then available to every single mod. This poses multiple problems. Every feature has to be compatible with every other feature, and as mods grow, so will the engine. Features are never removed, only added, and the weight is on every programmer that follows. Every feature, no matter how specific and particular, has to be generalized to work for everyone. Etc. Does it make sense for the weight of every mod to be on every other mod? On every programmer? Probably not. It's a maintainability and scalability nightmare, and in the end programmers have to be choosy and decide what is worth supporting or not, fragmenting the modders and codebases.

So, instead of the engine having to support every feature for every mod, what if mods could carry their own code? Their own features? Instead of Items needing 100+ properties, what if they could instead have scriptable functions for onFire, onUse, onHit, etc. This would both give mods more power and make code lighter and easier to maintain for programmers. Now I don't think this means exposing everything and creating a completely generic scriptable engine, that's its own nightmare. Just exposing the most common "game events" from which modders can manipulate the simulation would suffice, letting them add their own associated state and UI if needed. A happy middleground between modder potential and programmer scope.

So why Lua vs Yankes scripts? Well it's pretty clear adoption has not been great, modders would rather bug programmers than deal with it, and the only programmers that deal with it are... Yankes. And I think a big factor is because modders aren't programmers. You can give them all the code and potential in the world, if they don't know how to use it, it's useless. So will they code Lua? Probably not. The draw of scripting isn't to the modders who can't code, but to the programmers who can't code C++ but can deal with something simpler like Lua, which in turn help the modders. I doubt they're itching to learn a whole new bespoke scripting language.

As for performance... honestly, you could do a lot worse than Lua. It's the most common lightweight scripting language around, designed for embedded, runs on every platform on the planet. It'll never beat native, it's a bytecode VM, but for anything outside a tight game loop it should be fine. After all, OpenXcom struggles on anything under a 2Ghz and does all its graphical operations on the CPU, so I don't think we're winning any awards for performance either.

Yup

27

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 16, 2018, 03:29:37 pm »

For lack of a better place to put it, I'm going to start putting the lua API documentation in github.

https://github.com/xxWhispers/OpenXcom/wiki

Right now I've only just started putting together the list of available functions, but I suppose I should start somewhere

I'll add a "getting started using lua" once the project is in more of a stable state.

28

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 16, 2018, 02:54:31 pm »

Ah, yes.. forgot my audience for a minute there

By "full access" I mean basically anything you can see in the save file, and also what is loaded in the rules, with some exceptions to anything that may totally break the game(for example, you are allowed to see what mods are loaded, but you can't add new mods to that list on demand)

Cheers,
-Ken

29

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 16, 2018, 01:59:14 pm »

I had a longer post typed up, but due to power loss I guess I'm being forced to be brief

We compare not "script<->not scripts" but "2 scripts engine <-> 1 script engine". I add script exactly because of this reason you mentioned, but this can't be justification for another engine if one exists.

One of the biggest design tenants is that I do not want to replace or port any existing functionality so I do see yankesScript continuing for now. I'll maintain and expand the Lua API, and you can continue to focus on expanding yankesScript if you would like.

Another thing is demand for scripts in OXCE, right now I only one who push for new scripts (as my pet feature). Current interest is moderate (could be caused by limited scope of my scripts).

There's a huge demand from the modders I've spoken to on Discord.

Probably best benchmark for me would be custom bliting (altering pixel values on drawing surfaces).
Simply replace `ScriptWorkerBlit::executeBlit` with LUA call that will do same work without any slowdown.
Some thing like this: https://www.youtube.com/watch?v=FKzqCOg2DrQ (only units graphic, around 24 at once on screen).

Sounds like a good benchmark! I'll sneak that in as a "psychedelic" mod

And there is big difference, because my scripts do not allocate any memory only get buffer from C++ stack where they place variables. I simply traded speed for some limitations. If I would need allocate something then I would ask game code to create if for me.

The Lua context(which holds the memory for the Lua scripts) will initialise only with function pointers to allow the user to CRUD(create, read, update, delete) game objects. This means we can limited things to just "read only" if we want, but also means that no game state information will be held in the Lua context itself.

This object that script edit is pointer to `BattleUnit`. you can't have copy of unit because this will miss point of using game objects.
Another thing my scripts are static typed. This mean if you try modify const object then script will refuse to parse.

As mentioned above, we can limit access to read only since nothing will be stored in the Lua context.

If this is from current github repo then its need C++11, if it from some else it probably still use C++98 and boost.

Thanks! I'll pull the latest and compile from there instead of using Boost(that should cut compile time down a bit)

a) Have better capabilities that my scripts (easy to do).

Done

b) It do not have big overhead (probably harder).

Lua doesn't have a big overhead.

c) Can replace all current usage of my scripts.

Why is this a requirement? I'm not trying to replace yankesScript, at least, not initially. I'm only trying to provide a mod(der) friendly interface for other people to make what they want.

d) Do not disturb code base to much (e.g. you do not change multiple of functions signatures to pass some state).

Might need to define "too much"

I'll have to change the interface for loading/saving objects(almost everything in SaveGame, and most things that already take YAML node as a parameter) to include the lua context so that I can pass data to/from the game state.

e) If LUA have state it should be able to restore it after game save and load.

Yeah, still thinking on the best way to handle persistence. My current thoughts are to create a yaml node for game objects that stores the data per each mod, but some saves are already getting too large, so I might need to think about this some more.

f)  Meridian is fine too to change script engine.

Cool!

g) mxe work without any hacks

I don't think this'll be an issue

h) Have "killer app" for people like Dioxine,  Solarius Scorch, Hobbes or other big modders

"Hello World" isn't enough

I'm compiling a list of "example mods" as a way of specifying a Minimal Viable Product(MVP) milestone. So far, I have the following:

* add new field to the soldier for mana, and show how that can be used to implement new gameplay mechanics
* modify world zones and restrict missions in the zones to allow for diplomacy as a new mechanic
* implement "traitors" which are enemies that flip sides during battle and join your team
* open for more suggestions!

With all this I will happy kill my scripts and replace it with LUA

Like I said, the goal is not to kill yankesScript, but to supplement it initially. If there comes a point where the two collide, then yeah, we may want to revisit this strategy, but otherwise I don't see a problem with them co-existing

btw If your mod feature align with my or Meridian goals then is possible to include it to OXCE+ directly without need for script engine.

Yeah, but that simply continue the current process for writing custom logic in mods, which is to modify the engine in order to make something new. I would like to see what else the modders come up with given nearly full access to the internals of the engine

30

OXCE Builds & Ports / Re: [OXCE+] Lua Integration

« on: September 15, 2018, 10:37:23 pm »

My concerns is that each script engine is burden on source code (for me my script is lesser burden but for other programmers this could be opposite) to maintain to incompatible engines.

Ah, okay. Well, having modders make demands/hacks into code I see as a larger burden, and one that leads to more issues as it progresses. Having a script API that you agree upon at the start and then slowly adding to it is more flexible, IMO.

Speed of my script engine rely on it simplicity (implementation is complex but it all boils down to switch in loop) and using same memory model as C++. LUA have different model and need some conversions to transfer data (I could be wrong there, best if you would show code in action).

If you implemented Lua using default stack operations only, then yes, you have to convert values to/from script. However, there are user variables that you can define which use the memory address you give it and type you specify, however, I wouldn't go that route. Lua is FAST. I mean, seriously fast. I suppose we could do a side by side comparison using some algorithm(s), like md5, or perhaps something more memory intensive like huffman encoding, but I totally see your point about it being too slow for graphics purposes. For that, you would need something a little closer to the metal.

For comparison, the setting of 1 variable actually sets 3 points in contiguous memory. The first point is the ref counter, the second is the data type, and the third is the data itself. There isn't a single conditional in that code, which  means that it runs faster then your average if statement. Getting a value has one conditional, which is merely a conditional to ensure the value in memory is in the correct format - otherwise convert it, or throw an exception the old school longjmp way, not the fancy and expensve stack unwinding C++ way. Since the logic is run by token static single assignment, it's just as fast, and in a lot of cases faster than a logic tree switch statement.

I didn't build Lua, but if I were to build a scripting language, I would do something very similar to what they have done.

Another thing is LuaJit have cost too (on start up) and probably is big as whole OXC exe (again I did not see your implementation to be 100% sure).
In some UI parts it not matter but it should replace my script that need this.

Yes, and that is one of the biggest drawbacks of LuaJit. I'm not a fan of using it unless we do run into performance issues at runtime. I don't see that being an initial problem though, unless someone writes a script for every tile tha tgoes through every other tile of the level and every pixel at the same time. (I could totally see someone attempting to do that). In those cases, there's Lua's interrupt ability, which allows you to pause currently executing scripts and report errors/warnings to developers basically telling them to stop hogging all the resources!

You completely misunderstood meaning of `const` especially in context of my script (where I do not allow removing it). This allow me send some object to script and do not worry that someone modify it without my permission. This is great tool to reason on what happens in code and script that are called.

Oh, that's handy! Out of curiosity, instead of creating a whole system that then needs to be guarded by conditional if(s) which cause pipeline execution slowdowns(branch prediction is the largest CPU performance killer after cache misses because it causes pipeline flushes)... why not just copy it and let them modify the copy of the const so it doesn't have any effect on the original const variable?

OXCE do not use boost. This was yaml dependency that was drop by them when they switch to C++11.

Yeah, I've never used YAML in any projects before. This is my first time playing with it and I've already got mixed feelings about it. I did, however, quickly pull the latest luaYaml to allow me to get the config stuff exposed to Lua. So far, no problems there

Also, do you know what version they switched to C++11? I'm running 0.5.1 at the moment, but have considered updating that.

And more importantly it will work with VS? this is windows that do not have bash. Another important toolchain to consider is mxe (gcc based but have predefined libs available). This is why I ask ablaut dependencies.

Ah, yes, of course Lua works on MSVC. For MXE, I may need a bit of help on that, but I don't see that as a major issue.

I suppose the question I have is what would it take to get you to accept this? I'm not going to start up another branch just because of adding in Lua. That seams excessive and would further fragment the modding community. My biggest reason for doing this is because I wanted to do a quick mod and someone told me I'd have to muck about the engine code and release my own executable version if I wanted to do it, but that doesn't feel like a mod anymore. Simply exposing a bunch of stuff from the engine to a scripting interface is sufficient for what I wanted to do, so I started looking in to it. If you're saying you won't take it, then there's no point in doing it.