Melee tactics are all being discussed, so if you have a suggestion of optimal behavior in this particular case - it's most welcome. Right now no other than hide and ambush was proposed.
Personally, I think one single optimal model for BAI tactics is not good, so boosting melee strategy in one condition (closequarters) will lead to decrease of melee efficiency in different (open field).
BTW which version of BAI did you use?
I'm not sure I'm that experienced with XCom to give ideas on how it should work, but there are a few things I want to mention. I always try to use the latest version. My settings are in a post above.
I had a monster mission fighting Chryssalids and it was really dumb. Chryssalids can easily kill my units in 1 hit, but what do they do? They where absolutely terrified of my units for some reason and during the entire battle, 25 something turns, and they did not come close once during this (I use synthsuits with black ops weapons). All they did was walk back and forth trying to keep distance, sometimes taking cover. All they attacked where the civilians and once they they where turned to zombies they really didn't know what to do. Even the zombies behaved like this while in zombie missions they just try to swarm you.
Compare this to my previous post about ranged units forming lines to melee me and this becomes something that's not quite right.
I think that units, no matter how unmatched they are, should fight with their ranged weapons, unless they are melee units, and only consider melee if they think they can survive the next turn (hit and run) or can't run away.
But I don't want to enable BAI for brutes either. Having monsters hide behind obstacles indefinitely and then rush out and unleash 500 hits when you take one wrong step is no fun either. It's especially bad when you face monsters so tough you can't deal with them at the start of the game, that they just rush you and kill you.
But I also kinda want brutes to have some BAI features, cause right now they just walk around in circles and don't know what to do. If you have a flying unit you can just herd them and that feels off with such powerful units as Chryssalids.
An other problem with inconsistency I have is with Chasers. From the first turn they just rush you with no regards to survival. And if one of them has the mighty egg launcher it's really bad. Add to this the absurd amounts of units in XCF. I'm not sure how to deal with this other the just blindly run out from the spawn and try to hide, or if possible try to spread bombs, but running around with pre primed bombs is bad. But if you can survive 2 or 3 turns, then they tend to become really cowardly.
In one ship mission only a few spawned outside the ship and got killed. The rest inside the ship, 10+, just decided to cover in fear or something, and camp the door. Not very Chaser like behavior and they got a rocket for it.
On the other hand, this behavior works very well for Hybrid bases. The assaults come rushing forth while the rest do a more careful approach. It feels really nice and the only problem again is the huge amount of units at the start.
One more thing to mention is the option to tweak aggressiveness. I don't like this, just have an option for setting aggressiveness mode and then just do the tweaking under the hood from testing.
One more thing I noticed playing campaign. It is near impossible to acquire live specimen as they all carry preprimed grenades. Kamikazes.
Looks like the only way to do it is to jump on somebody with few soldiers with stun rods and pick the primed grenade once the body falls. Didn't try it yet but it seems to be quite difficult.
Long range stunning becomes quite useless for this purpose.
You said you are playing campaign with this mod too. How do you capture them?
I had this problem too and if they had a key item it was really bad so I turned it off. I used dogs to pick up bombs but a big problem is also that you can't tell if it's a mine or a bomb.
I think they should be able to pre prime but then use it the next turn or un prime it, or something like that. That might be a bit convoluted though.