[WIP][SOURCEMOD][OXCE] Brutal-AI

[Xilmi]

Test with Brutal AI OXCE 7.8.4, but Brutal AI turned on :  1st turn "hidden movements" takes 258 seconds before it's my turn to play.

I'll get this Mod and your savegame and test for myself what makes it take so long.

[Xilmi]

Used the new version from github, i noticed there are a bunch of dll in there, while for my test i moved them into my openxcom folder just in case they're required, are they really needed (i ask because by default there's no dll in Meridian's OXCE directory usually) ?

Yes, they are needed. Meridian uses some other way to compile in which the dll's are put into the exe-file. That also explains the difference in the size of the exe-files.

I used the same save i posted there previously :
https://openxcom.org/forum/index.php/topic,10854.msg150565.html#msg150565
(using the 40K mod + its Rosigma Expansion) standard 50x50 , there were 28 enemy units that all have flying ability and a medium sized UFO of 3 levels high

For me the map was 60x60x9 compared to the standard 50x50x4, so that's a total of 32,400 tiles instead of the standard 10,000. The height of the UFO itself isn't important, the height of the entire map is. But the catch really is that all of the enemies can fly. With walking units, the amount of potential tiles to analyze is much lower. Usually between 1 and 2 times of what the result of multiplying the x and y dimensions are.

For me it took roughly 60 seconds to resolve the 1st turn. But I have a pretty new computer, so I can see it taking 4 times longer on weaker hardware. Maybe some changes between 1.0.1 and 1.1.0 also helped speeding it up a little.

But yes, it is too slow to be bearable, even with the improvements I have made. I have to think about what else I can do to make it faster without compromising the AI too much.

Btw.: This Mod looks really impressive art-wise! Big props to it's authors.

[Rangerh]

just tried with BrutalAI version 1.10 i still get over 250 seconds for a turn with my stress test save in 40k+Rosigma.
Though there may be difference with how the terrain is generated, as on some test it was jungle-like  that seems be be more busy with obstacles than other terrain types.
Will have to save exactly before pressing the End Turn so those tests always happen in the exact same condition (AI default spawn location, same terrain, etc..)

[Yankes]

Why game need to check whole map? Could be better filter tiles that have no impact on final result? Some times making more operation make code lot faster if it permitting skipping other operations.
I had performance problem with light propagation, on some very big maps you can notice 1s lag when you open doors, solution was:
https://github.com/Yankes/OpenXcom/commit/e6dcfc177222517cecb89d780dc2cb78fff27699
Loot of new complex code and lot more at first glace work done, but with this code lag is barley noticeable.

This mean one solution could be process only xcom units and mark every place where any xcom units are visible, then when your process aliens you consider only places where have any visibility or at list visibility is possible. It do not matter even if half of tiles will be wrongly recognized as "have visible xcom unit" if you process only 10% of current tiles. This is still 10x speed up.

[Xilmi]

just tried with BrutalAI version 1.10 i still get over 250 seconds for a turn with my stress test save in 40k+Rosigma.
Though there may be difference with how the terrain is generated, as on some test it was jungle-like  that seems be be more busy with obstacles than other terrain types.
Will have to save exactly before pressing the End Turn so those tests always happen in the exact same condition (AI default spawn location, same terrain, etc..)

Yeah, I wasn't really expecting an improvement in this regard in 1.1.0 over 1.0.1.

I've now looked into some pruning and in my test-save got from 58 seconds to 24. Not an order of magnitude but about 60% reducting isn't bad either.
The units in the mod also have a lot of TUs. So they can still reach almost 7000 different tiles. I also figured out what takes how long.

Trying to attack stuff and the calculations for move-order-manipulation: 5 seconds (total for all units)
Pathfinding: 3 seconds (total for all units)
The remaining 50 seconds was analysis of the tiles where to go to. So we are actually looking of an improvement of 50 to 16 seconds for that part.

Behavior is impacted minimally as many of the options looked at before would get a bad score anyways. I have seen a few scenarios though where it does act differently from before.

I'll look at Yankes' example of whether there is some inspiration for what I can use. Theorethically what I need could even be similar to what he did.

Edit: I actually tried to play this mission. It is insanely hard. The enemies only die with 3 or 4 of my people focus firing them for several turns. I killed 2 opponents in total during my attempt.

[Xilmi]

This mean one solution could be process only xcom units and mark every place where any xcom units are visible, then when your process aliens you consider only places where have any visibility or at list visibility is possible.

Is the "visibility is possible" check faster than _save->getTileEngine()->calculateLineVoxel()?
I want to find Positions from where I could attack a target, regardless of my own vision-range.
_save->getTileEngine()->canTargetUnit() was too slow for me since it tries all sorts of voxels within the target-tile. For simplification I only looked form tile-center to tile-center, which already was a great improvement.

If I could get a similar result to what calculateLineVoxel does faster, that would be very helpful.

[Yankes]

Check if tile is visible first, is not exactly same but lot of cheaper to do. There is multiple places where you could cut corners in algorithm to skip majority of tiles in check (like check tiles that have closet path to your alien, have max range cut off from human, etc).

[Xilmi]

Check if tile is visible first, is not exactly same but lot of cheaper to do. There is multiple places where you could cut corners in algorithm to skip majority of tiles in check (like check tiles that have closet path to your alien, have max range cut off from human, etc).

Isn't the visible-check only for what X-Com currently sees? I thought about that but there's two issues:
I wouldn't be able to find tiles that have Line-of-sight but are outside of visible range and I wouldn't be able to find tiles that they are not looking at. (Have turned their back to)
I'd need something like: "What would they see if they had infinite vision range and would look at all directions?" But something like that should be possible to create and then put into a buffer. It would then be the same for every alien. And I could exclude all tiles that are not in that from checking. But I guess on maps with very little obstacles it wouldn't make much of a difference. Maybe if I don't just use it as a filter but instead of what I use now. I'd just need to figure out a way to avoid aliens to place themselves in ways that increases the chance of shooting into each other's back.

The change I made to get from 50 to 16 seconds was to only consider what is in range to walk to, which is kinda similar to what vanilla-AI does in findFirePoint. The difference is only that if I can't find a fire-point I now simply walk closer instead and hope I'll find one in the next turn. This of course means It'll never walk into a building from where it would have a line of fire too, if it is the other way. But it likely wouldn't have done so anyways because that would have gotten a lower score unless in very specific constructed scenarios.

[Xilmi]

Here's a new version:

https://github.com/Xilmi/OpenXcom/releases/tag/v_1_1_1

Not much has changed actually. But turn times should be better. (Roughly 40% of before)

[Xilmi]

1.1.2 is availble:

https://github.com/Xilmi/OpenXcom/releases

It fixes an endless loop when playing with regular-AI without traceAI enabled. (Incorrect nesting of conditions)

Also as a reminder:

Please make sure to actually enable it to avoid disappointment for not making any difference!

It will now be also enabled by default if you don't have an options.cfg where it is disabled already.

[Rangerh]

I think my previous save was bad in fact, because i did it before reaching the landed UFO, meaning the map would never be generated the same once you intercepted the UFO, and maps can get generated more or less complicated and so influencing the AI turn time.

So this time , always using the same landed UFO, i made the save from BEFORE you click on next turn, so with a same basis it makes it easier to check the difference of impact between Brutal AI versions.

I attached the 2 saves (each save use a different map but still the same landed UFO)

SAVE : testBrutalAIv6
BrutalAI 1.1.2 disabled : 20 seconds (tested with traceAI on true)
BrutalAI 1.1.2 enabled : 53 seconds (tested with traceAI on true)

SAVE : testBrutalAIv7
BrutalAI 1.1.2 disabled : 23 seconds (tested with traceAI on true)
BrutalAI 1.1.2 enabled : 59 seconds (tested with traceAI on true)

That's quite an impressive improvement in comparison to my previous tests (that was at +/- 250 seconds !) , congratulation on that great optimisation !

[Xilmi]

Would you say it is playable like that?

Another question: As I said, I tried and failed to beat this mission. Is this mission supposed to be winnable with your squad or is it something you normally wouldn't have done either?

I got other missions from your original save-game that were way, way easier. With enemies that would die in one hit and only a handful of them. Not almost 30 super-tanky enemies.

[Rangerh]

If it can still go down in time, the better but it is playable that way.
The 1st turn is always the longest, as from the 2nd turn there are some units that have been killed already, lowering the turn time.

This is a very difficult mission in normal condition : you're outnumbered and outgunned and enemies are flyiers so they can see your troops mostly everywhere and flyiers usually have higher dodge capacity so you will mostly miss a lot of your shots while they will not. Some enemies have also access to some shielding that your units don't.
But the feeling of enemies being near bullet proof while your troops die super fast comes from the level of difficulty:

Something to note is that 40K (and Rosigma) replace the original game difficulty selection by factions (so you can play with Space Marines, Sister of Battle, Inquisition, Imperial Guards, Arbites, etc... that have their own troops, facilities, weapons and vehicles, it's a very cool way to add a lot to the replay value)
And then the mod will assign a level of difficulty to your current game somewhere in the ruleset.
By example i am playing with the Space Marine faction, and in the ruleset it is forced to use the difficulty that is equal to Veteran in the normal game (that's the 3rd difficulty level, so if you're not an expert in the game mechanics, that's indeed very hard)

Additionally there's a sniper/spotter functionality that is described there and is running in 40k+Rosigma :
https://openxcom.org/forum/index.php/topic,5679.0.html
that basically make sure the AI has some kind of maphack so it can shoot at you from beyond your view (this functionality is not used in default OXCE UFO and TFTD, it's for modder and nearly every mod use it and i guess with BrutalAI enabled it must probably make it hell for the player).

But for that 40k+Rosigma mod, there's a way to lessen the difficulty by using a minimod so you can lower or increase (there are some masochists i guess ) the difficulty if it's too hard or too easy for you:
https://openxcom.org/forum/index.php/topic,10802.0.html

On lower difficulty you'll notice the enemy is much less bullet proof and will die as much as your units die, it actually feel more balanced.
To me it's much more enjoyable by slightly lowering the difficulty that way, but to each their own. By example that hard mission is winnable on a lower difficulty (i did) .

There are some more minimods i posted in the 40K board to remove some features (the removal of that RNG dodging, the removal of those enemy transforming into new enemies on death that transform again into another enemy on death...) and can help to bring down the difficulty of the mod if it's still too hard.

[Xilmi]

"But the feeling of enemies being near bullet proof while your troops die super fast"
I didn't have the impression that my troops die super fast. Infact they also seemed quite tanky. But the sheer numerical advantage of the enemy eventually overpowered them.

"Additionally there's a sniper/spotter functionality that is described there and is running in 40k+Rosigma :
https://openxcom.org/forum/index.php/topic,5679.0.html
that basically make sure the AI has some kind of maphack so it can shoot at you from beyond your view (this functionality is not used in default OXCE UFO and TFTD, it's for modder and nearly every mod use it and i guess with BrutalAI enabled it must probably make it hell for the player)."
That part actually doesn't work like that with BrutalAI enabled. Or let me put it that way: Brutal AI has it's own sniper/spotter-mechanic that is actually more fair.

Normal Sniper-Spotter mechanic uses a flag called "TurnsLeftSpottedForSnipers" tied to the unit. The unit can get revealed like that by attacking itself or being spotted. The problem with that is that it will stick to the unit for several turns, regardless of where it goes in the meantime.
I introduced a new flag that only gets triggered by actual visual contact and only remains for the rest of the turn.
However, every unit can now target the revealed unit for the remainder of the turn.

With high sight-radius, high TUs, few obstacles on the map and sniper/spotter enabled the behavior of the normal AI actually will not differ that much from that of Brutal-AI.
The main difference is that they are pathfinding over several turns and can walk all the way from the command room of a battleship to a position from where they can camp the skyranger-exit.
Also if there is a line of fire within their within their movement-radius towards a unit that is currently spotted, they will find it and go there, regardless of how far the unit is. So they will not only snipe well if they are already in position but also if they have to move to get to position. And that's actually the part that makes it slow. That they analyze every reachable tile for whether they can get a line of fire (before it was every tile).
The third big difference is that they can issue a wait-command to wait for other units to act first. This makes sniper+spotter way more efficient because the Spotter doesn't wase their time-units on killing the target he just spotted. He leaves that to the snipers in the background and will only attack himself if all snipers failed (or there are none). This means he can move in further and reveal even more targets.
That's basically how my AI counters smoke grenades. The Spotter doesn't get "depleted" by my own spotter for killing him and then being stranded. He spots, has someone else kill my spotter and then continues to walk forward and reveal more targets. This required a rewrite of big parts of the AI-turn-processing logic as "waiting and then continuing" wasn't supported at all.

[Rangerh]

in that mission your troops have terminator armor, that's  very strong as an armor. But before reaching that point (a soldier needs to get 30kills to be able to wear such armor, and then you need to manufacture the armor as it can't be purchased) your troops die very fast in the normal conditions (Rosigma lessened that fortunately by improving the armor system, but in base 40K i got lots of armored units one shotted by anything).

Great news that BrutalAI is replacing the sniper/spotter system by its own, i was a bit worried it would get things simply impossible if both were running at the same time.