I tried the flamethrower mod, but haven't been all that impressed by it. This is a brain stormy post and kind of a "call to the community".
It is possible to add new explosion sprites for the nightly. I've already done it in the ExtraExplosions mod. If we use arcing shot with rapid bullet speed and a "long" bullet sprite, it could look like someone is spraying burst of flamey stuff that spreads/splashes upon hitting the floor. Add "fwoosh" firing and exploding sounds and we might get something better than the current flamethrower.
The only thing left to balance would be the damage. Currently, incendiary hits are 10hp, ignoring armor. That makes an accurate autoshot (8 shots) deal 80 damage to any alien in the game regardless of armor (except snakemen, ethereals, silacoids and chryssalids who have some resistance). Feels a bit OP and it does not really work since armor should protect you somewhat. I guess the answer to that is to redefine the armor of certain aliens (sectopods for sure) to be more resistant to incendiary damage.
So all of that sounds pretty doable, I'd say. Except that I have no clue where to get sounds and the only program I use for graphics is paint.. If there's anyone willing to contribute sprites and sounds, I would be happy to put things together in a ruleset and publishing it (with all due credits of course).