Thanks for the save. I'll look at it once I'm back home.
I think I can explain the reason why this happens without looking at it. Maybe while explaining I can come up with a solution.
When the AI has not spotted anyone of your team, they basically have two steps. The first step is the peaking step. Which has several priorities:
If they think they will end up in smoke, they can spend up to 50% of their TUs to move forward.
If they think they can establish visual contact, thy can also spend up to 50% of their TUs to look for your units.
If neither of this is true, they will spend as little TUs as required to establish a line of sight to the first tile towards your unit that they currently can't see. Also up to 50% of TUs but usually less.
That phase works fine and isn't the one concerned here.
The second phase is the one that is supposed to "improve the position". For that there's 2 important parameters: Closeness to the suspected position enemy and safety from being discovered by the enemy. If there's a lot of cover around this works really good. If there's no or only very bad cover around, then the "safety"-part falls short to contribute much and closeness to the enemy becomes very predominant.
What's lacking completely in these considerations is "preserving TUs".
I guess they still hid behind the ship in the scenario you described, so it's cover also contributed to improving their position.
If your soldiers were doing something else and weren't aware of their presence, that could even be considered a smart move. Taking a little risk to be in striking-distance for the next turn.
But since you knew they were there, you obviously would look for them on the next turn and catch them with their pants (TUs) down.
So here's my idea:
The location where and when a unit was last spotted is tracked for each of the three teams. The AI uses this information about your units to have an idea where they could be without having to cheat and just "know". Their own information like that also exists so it can be used in autoplay. I as a player, have a good idea when my units were spotted by the AI and can take it into account in my decision-making. For example when I peak around a corner and look in the face of an alien and only have very little TUs to hide again, I know the alien knows that my unit must be close and likely will look for it right where I hid it. So I deliberately try to avoid doing things that would reveal my units position in order to sneak up on the AI.
This means the aliens could also "look up themselves" for whether they were recently spotted by me and take that into account for their decision-making.
In this example they'd know they were seen by you and instead of valuing "getting closer" could value "preserving TUs" or even "getting further away".
Now I'm really eager to experiment in this direction.
So again, many thanks for your feedback. Feedback like this is the lifeblood for ongoing AI-improvement!
Edit: The approach kinda works. But considering themselves as spotted after opening a door is a bit problematic in that regard.
Edit2: While I agree that in this particular situation holding the position would be better than trying to get closer, my attempts of realizing it lead to regression in my benchmark-scenarios.