In vanilla X-Com, alien missions have no fail state and can only be delayed. Eventually, all nations except Russia will quit the project. Hypothetically you could lose the game if the aliens ever infiltrated Russia, but it's in Siberia which is considered to be barren land, so the aliens won't do life-oriented missions there (harvest, abduction, infiltration). Mods can change all of these things.
There's very little to do in the end game. I'd recommend playing a mod that gives you more to do. I know that Precentor Appollyon's Hybrid Mod allows you to go to Cydonia and the game doesn't end when that mission is over.