What would you do to limit players from spamming Terror Missions/Alien Infiltration missions?
Probably the same as many limitations in games: missions require resources, and resources are scarce. You only have a puny Martian base at your disposal, nothing impressive really. You must at least maintain your continuous presence on Earth - otherwise your meticulously woven web of intrigue (agents, lobbyists, Men in Black, half-human hybrids etc.) will collapse, and your invasion will be over... Or at least stalled by a few decades.
How would you handle the enemy AI? Would it play fair?
Of course! Every AI should play fair!
Would the Council actually be intelligent, reasonable human beings who don't underfund XCOM?
I would prefer to make things appear exactly like in the original game, though I can't explain much at this point. Anyway, the truth is that you're not as much as an invader as you would like; you must gain power by diplomacy. Humans may be underdeveloped in comparison, but you're completely outnumbered. You basically have to play Hernan Cortez.
Would XCOM be the only force capable of killing your forces, and not local militia or the armed forces?
I would like to see the armies of the world to terrori- ehm, to fight. Or just small platoons really, since you wouldn't want to fight real tanks and high-altitude bombers anyway. These soldiers you fight would be around Squaddie level, but their weaponry would be of lower quality (X-Com gets the best the earth has to offer, while normal guys get normal stuff - Kalashnikovs, M-16s and the like). The difference between then and X-Com would at first be HWPs and a bit better stats on average, though over time they would get much better weapons and armour (not to mention psionics; most of your species are very weak to psionics).
Would the enemy AI be able to handle having inferior weaponry?
Yes, if it's sufficiently numerous.
Would it be able to do tactics like throwing a smoke grenade on the skyranger ramp, camping a ufo untill turn 20, or using rookies as disposable meatshields?
That's hardly for me to say, but I can't see why not, provided it's done seriously.
What would keep it from being too EASY, other than making the AI cheat? How would the AI start out weak and puny, but eventually defeat you(and have it believable?) How would the AI controlled XCOM deal with your PSI Spamming in the later months?
These questions are crucial in any game design, and are resolved over long periods of time, spent on coming up with ideas, heated discussions and throwing objects at each other.
In other words, I have no idea right now; we'd have to think on this hard. But I think the current progress mechanics (for aliens) isn't terrible.
Chryssalids? Would it always fight to the death, or sometimes try to abort?
Terror units probably wouldn't be able to abort a mission, nor start it without at least one individual from their master race. Same as X-Com tanks really.
What about base layouts for if you attack XCOM's base?
Probably random, but must feature all buildings that should be there.
This is fun!