I did a fairly extensive test with incendiary ammo in both GOG's X-Com (presumably the same as the original) and OpenXcom.
Incendiary in OpenXcom:
A shot (any power) deals around 5-10 damage to the unit, seemingly (very) slightly reduced by under armor. 2x2 units take double damage. I'm not sure how the sustained damage works, didn't test that much.
Reapers appear to take more damage exactly based on their incendiary weakness (1.7).
Incendiary in GOG X-Com:
The initial incendiary shot deals 1 damage to units, with a low chance of dealing a bit more (possibly due to direct hits). Ending the turn or having any of your soldiers fire a weapon will trigger a damage cycle for all units standing in fire. I did not test if alien weapons' fire triggers a damage cycle. The first damage cycle does around 5-10 damage and they ramp up, killing a soldier in around 3-6 turns.
I was unable to test incendiary against reapers.
I want to know what the rest of you know about incendiary. It's a difficult subject so tell me all you know! I'll try to update the OP as info comes in.
P.S.: If you have experimental data, make sure you mention which version of X-Com it comes from! (TFTD info welcome too!)
edit 7/25/2017: Assessment: The original game had incendiary deal either 1 or 5-10 damage per incendiary hit, then if they were lit or standing in fire, they took fire damage each fire damage tick. Setting off any incendiary weapon would cause every unit on fire or in fire to take one fire damage tick. (This is the Funky Fire bug.) Openxcom has incendiary deal 1 or 5-10 damage per hit but the chance of 1 is said in the UFOPaedia article to be equal to the chance of any other number (1/7 by default). The Funky Fire bug is removed. Incendiary ignores armor completely and 2x2 units can have 4x damage dealt to them. It's still not a lot of damage even to reapers, but it can be a way to kill sectopods or cyberdiscs if your other weapons cannot penetrate their armor.