I know even better way for "improvement". It is called "don't ever move, just shoot, act like a turret". Yeah, if you just want to make game harder, paradoxically you should abolish AI completely, since AI actions right now are mostly suicidal.
I disagree on "not moving" providing the most challenge. At least you'd have to go to better locations to camp from while also avoiding clumping. Getting into better positions to shoot from without exposing yourself in the process can be highly advantageous.
Also staying in your position while somewhere else on the map your friends are engaged in a fire-fight sounds quite sub-optimal.
In an essence, most of XCOM missions ideologically are castle siege and storming. Bad guys are living in an area for a while, then Skyranger flies in and starts massacring: first those without cover, then those who are still hiding inside buildings. Right now an AI has a behavior of a lemmings. Player just adopts to that and a castle siege game becomes a tower defense game.
I agree with your assessment of the state of the default-AI. It's exploitable. A big reason for the tower-defense-exploit is that they are kinda hard-coded to be hiding until turn 20 and starting from turn 21 they get the idea that maybe they should start exploring. (Exception are x-com-base-defense-missions, where they take a more active role right away). So they give you all the time you need to set up a trap and then walk into it one by one. Simply removing that switch-flip would help them big time in regards of playing more castle-defensy.
Could also make them all gather up in the "lobby" of their respective vessel, spread them around and aim at the door. I agree that this would make it hard.
However, I have a vastly different vision of how the solution should look like. I don't want the AI to play a passive and defensive-camping-game with potentially even more hide&seek.
I kinda want to reverse the roles here. Basically the player shall feel like their units are being hunted by the aliens and not the other way around. I want the aliens to be proactive but not in a lemming-like way.
What I'm thinking of is slowly moving towards where they think they can take good aim, reserving TUs for reaction-fire while avoiding exposure to enemy vision. There already are good methods for evaluating terrain and the path-finding, with active "Sneaky AI", does the "avoid exposure"-part. The important part is to not act like independent individuals who do that one by one but instead in a coordinated and decisive way.
I kinda already achieved that with some caveats. The main one being processing-time caused by "brute-forcing" the search for the ideal position to be at.
Realizing what you described would be possible. An algorithm would need to analyze the map for a ratio of "how many tiles do I have vision to" vs. "from how many tiles am I visible".
This could be helped and made cheaper with something that I'd call "semi"-cheating. Semi-cheating is when you use information you normally wouldn't have in a way to emulate experience and intuition.
In this case: You don't need to consider vision to and from all other tiles but only the tiles that there are enemies at. Using the information where enemies are to snipe them despite not seeing them obviously oversteps the boundary to actual cheating. Using it to save processing-time, emulate experience and prune useless behavior is more of a grey-area, which I think might be okay in this game.
I mean obviously the Sky-Ranger is where the X-Com-sodliers are coming from. And when we assume the Aliens are highly intelligent and communicate back to Cydonia, having them act ignorant to that circumstance seems more immersion-breaking then them knowing it's full of enemy-units.
Then there's also the circumstance that the player gets acoustic clues of the wereabouts of the aliens. So some sort of compensation for the aliens seems feasible.
In the end it is about the user-experience anyways. And it's very subjective what different people consider fun. I guess I first try to make it fun for myself.
